Hi Dmitry, Sorry for the late reply. For CFI specific code generation, pcc is a better person to answer. But on the issue of global variables being optimized, that hasn't happened yet. That would be great if you wanted to pick that up! In your original email example, it seems like the file static i=53 could be constant propagated since there are no other defs, and the code in get_fptr simplified during the compile step, but I assume this is part of a more complex example where it is not possible to do this? Also note that with r327254 we started importing global variables. Do you know why we don't import in your case? I wonder if it has to do with it being CFI inserted code? Teresa On Tue, Apr 17, 2018 at 9:17 AM <dmitry.mikulin at sony.com> wrote:> I watched Teresa’s talk on ThinLTO from last year’s CppCon, and it > sounded like adding global variable information to the summaries was in the > works, or at least in planning. Can someone (Teresa?) please share the > current status? If it’s part of future plans, are there any specific > proposals that can be picked up and worked on? > > Thanks! > > > > On Apr 9, 2018, at 6:51 PM, via llvm-dev <llvm-dev at lists.llvm.org> > wrote: > > > > Hi, > > > > I’m working on setting up ThinLTO+CFI for a C application which uses a > lot of function pointers. While functionally it appears stable, it’s > performance is significantly degraded, to the tune of double digit > percentage points compared to regular LTO+CFI. > > > > Looking into possible causes I see that under ThinLTO+CFI iCall type > checks almost always generate jump table entries for indirect calls, which > creates another level of indirection for every such call. On top of that it > breaks the link order layout because real function names point to jump > table entries. It appears that I’m hitting a limitation in ThinLTO on how > much information it can propagate across modules, particularly information > about constants. In the example below, the fact that “i” is effectively a > constant, is lost under ThinLTO, and the inlined copy of b.c:get_fptr() in > a.c does not eliminate the conditional, which, for CFI purposes requires to > generate a type check/jump table. > > > > I was wondering if there was a way to mitigate this limitation. > > > > a.c > > ============================> > typedef int (*fptr_t) (void); > > fptr_t get_fptr(); > > int main(int argc, char *argv[]) > > { > > fptr_t fp = get_fptr(); > > return fp(); > > } > > > > > > b.c > > ============================> > typedef int (*fptr_t) (void); > > int foo(void) { return 11; } > > int bar(void) { return 22; } > > > > static fptr_t fptr = bar; > > static int i = 53; > > > > fptr_t get_fptr(void) > > { > > if (i >= 0) > > fptr = foo; > > else > > fptr = bar; > > > > return fptr; > > } > > > > _______________________________________________ > > LLVM Developers mailing list > > llvm-dev at lists.llvm.org > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >-- Teresa Johnson | Software Engineer | tejohnson at google.com | 408-460-2413 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180417/e6e25f35/attachment.html>
Hi Teresa, Thanks for the info! This example is my attempt to reduce FreeBSD kernel to something more manageable :) I will take a look at why globals are not being imported in this case. What’s the best tool to look into ThinLTO objects and their summaries? Most dumping tools don’t seem to like ThinLTO bitcode files… Hopefully Peter can chime in regarding CFI related issues. Thanks. Dmitry. On Apr 17, 2018, at 9:37 AM, Teresa Johnson <tejohnson at google.com<mailto:tejohnson at google.com>> wrote: Hi Dmitry, Sorry for the late reply. For CFI specific code generation, pcc is a better person to answer. But on the issue of global variables being optimized, that hasn't happened yet. That would be great if you wanted to pick that up! In your original email example, it seems like the file static i=53 could be constant propagated since there are no other defs, and the code in get_fptr simplified during the compile step, but I assume this is part of a more complex example where it is not possible to do this? Also note that with r327254 we started importing global variables. Do you know why we don't import in your case? I wonder if it has to do with it being CFI inserted code? Teresa On Tue, Apr 17, 2018 at 9:17 AM <dmitry.mikulin at sony.com<mailto:dmitry.mikulin at sony.com>> wrote: I watched Teresa’s talk on ThinLTO from last year’s CppCon, and it sounded like adding global variable information to the summaries was in the works, or at least in planning. Can someone (Teresa?) please share the current status? If it’s part of future plans, are there any specific proposals that can be picked up and worked on? Thanks!> On Apr 9, 2018, at 6:51 PM, via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote: > > Hi, > > I’m working on setting up ThinLTO+CFI for a C application which uses a lot of function pointers. While functionally it appears stable, it’s performance is significantly degraded, to the tune of double digit percentage points compared to regular LTO+CFI. > > Looking into possible causes I see that under ThinLTO+CFI iCall type checks almost always generate jump table entries for indirect calls, which creates another level of indirection for every such call. On top of that it breaks the link order layout because real function names point to jump table entries. It appears that I’m hitting a limitation in ThinLTO on how much information it can propagate across modules, particularly information about constants. In the example below, the fact that “i” is effectively a constant, is lost under ThinLTO, and the inlined copy of b.c:get_fptr() in a.c does not eliminate the conditional, which, for CFI purposes requires to generate a type check/jump table. > > I was wondering if there was a way to mitigate this limitation. > > a.c > ============================> typedef int (*fptr_t) (void); > fptr_t get_fptr(); > int main(int argc, char *argv[]) > { > fptr_t fp = get_fptr(); > return fp(); > } > > > b.c > ============================> typedef int (*fptr_t) (void); > int foo(void) { return 11; } > int bar(void) { return 22; } > > static fptr_t fptr = bar; > static int i = 53; > > fptr_t get_fptr(void) > { > if (i >= 0) > fptr = foo; > else > fptr = bar; > > return fptr; > } > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-- Teresa Johnson | Software Engineer | tejohnson at google.com<mailto:tejohnson at google.com> | 408-460-2413 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180418/9862a945/attachment.html>
Note that the renaming of functions happens with full LTO as well, but I think we might end up doing it more often with ThinLTO because with ThinLTO we need to decide which functions need jump tables (and therefore need to be renamed) before cross-module importing/inlining, as opposed to after inlining with full LTO. Regarding the issue with constants, once the importing issue is solved I think the direct call would still go through the jump table. If that ends up being a problem for you I guess we can consider introducing an optimization pass to replace direct calls to jump tables with direct calls to real functions. Regarding the link order issue, can you work around it by repeating each entry in your orderfile with ".cfi" appended? Something like: fun1 fun1.cfi fun2 fun2.cfi Regarding bitcode files, the files we create for ThinLTO contain multiple modules, and you can use the llvm-modextract tool to extract the individual modules and inspect them with standard tools such as llvm-dis. Peter On Wed, Apr 18, 2018 at 4:49 PM, via llvm-dev <llvm-dev at lists.llvm.org> wrote:> Hi Teresa, > > Thanks for the info! > This example is my attempt to reduce FreeBSD kernel to something more > manageable :) > > I will take a look at why globals are not being imported in this case. > What’s the best tool to look into ThinLTO objects and their summaries? Most > dumping tools don’t seem to like ThinLTO bitcode files… > > Hopefully Peter can chime in regarding CFI related issues. > > Thanks. > Dmitry. > > > On Apr 17, 2018, at 9:37 AM, Teresa Johnson <tejohnson at google.com> wrote: > > Hi Dmitry, > > Sorry for the late reply. For CFI specific code generation, pcc is a > better person to answer. But on the issue of global variables being > optimized, that hasn't happened yet. That would be great if you wanted to > pick that up! > > In your original email example, it seems like the file static i=53 could > be constant propagated since there are no other defs, and the code in > get_fptr simplified during the compile step, but I assume this is part of a > more complex example where it is not possible to do this? Also note that > with r327254 we started importing global variables. Do you know why we > don't import in your case? I wonder if it has to do with it being CFI > inserted code? > > Teresa > > On Tue, Apr 17, 2018 at 9:17 AM <dmitry.mikulin at sony.com> wrote: > >> I watched Teresa’s talk on ThinLTO from last year’s CppCon, and it >> sounded like adding global variable information to the summaries was in the >> works, or at least in planning. Can someone (Teresa?) please share the >> current status? If it’s part of future plans, are there any specific >> proposals that can be picked up and worked on? >> >> Thanks! >> >> >> > On Apr 9, 2018, at 6:51 PM, via llvm-dev <llvm-dev at lists.llvm.org> >> wrote: >> > >> > Hi, >> > >> > I’m working on setting up ThinLTO+CFI for a C application which uses a >> lot of function pointers. While functionally it appears stable, it’s >> performance is significantly degraded, to the tune of double digit >> percentage points compared to regular LTO+CFI. >> > >> > Looking into possible causes I see that under ThinLTO+CFI iCall type >> checks almost always generate jump table entries for indirect calls, which >> creates another level of indirection for every such call. On top of that it >> breaks the link order layout because real function names point to jump >> table entries. It appears that I’m hitting a limitation in ThinLTO on how >> much information it can propagate across modules, particularly information >> about constants. In the example below, the fact that “i” is effectively a >> constant, is lost under ThinLTO, and the inlined copy of b.c:get_fptr() in >> a.c does not eliminate the conditional, which, for CFI purposes requires to >> generate a type check/jump table. >> > >> > I was wondering if there was a way to mitigate this limitation. >> > >> > a.c >> > ============================>> > typedef int (*fptr_t) (void); >> > fptr_t get_fptr(); >> > int main(int argc, char *argv[]) >> > { >> > fptr_t fp = get_fptr(); >> > return fp(); >> > } >> > >> > >> > b.c >> > ============================>> > typedef int (*fptr_t) (void); >> > int foo(void) { return 11; } >> > int bar(void) { return 22; } >> > >> > static fptr_t fptr = bar; >> > static int i = 53; >> > >> > fptr_t get_fptr(void) >> > { >> > if (i >= 0) >> > fptr = foo; >> > else >> > fptr = bar; >> > >> > return fptr; >> > } >> > >> > _______________________________________________ >> > LLVM Developers mailing list >> > llvm-dev at lists.llvm.org >> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >> > > -- > Teresa Johnson | Software Engineer | tejohnson at google.com | > 408-460-2413 > > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >-- -- Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180418/52ff030e/attachment-0001.html>
On Wed, Apr 18, 2018 at 4:49 PM, <dmitry.mikulin at sony.com> wrote:> Hi Teresa, > > Thanks for the info! > This example is my attempt to reduce FreeBSD kernel to something more > manageable :) > > I will take a look at why globals are not being imported in this case. > What’s the best tool to look into ThinLTO objects and their summaries? Most > dumping tools don’t seem to like ThinLTO bitcode files… >Sadly there isn't a really great way to dump the summaries. =( There was a patch awhile back by a GSOC student to dump in YAML format, but there was resistance from some who preferred dumping to llvm assembly via llvm-dis and support reading in the summary from llvm assembly. It's been on my list of things to do, hasn't yet risen high enough in priority to work on that. For now, you have to use llvm-bcanalyzer -dump and look at the raw format. Teresa> Hopefully Peter can chime in regarding CFI related issues. > > Thanks. > Dmitry. > > > On Apr 17, 2018, at 9:37 AM, Teresa Johnson <tejohnson at google.com> wrote: > > Hi Dmitry, > > Sorry for the late reply. For CFI specific code generation, pcc is a > better person to answer. But on the issue of global variables being > optimized, that hasn't happened yet. That would be great if you wanted to > pick that up! > > In your original email example, it seems like the file static i=53 could > be constant propagated since there are no other defs, and the code in > get_fptr simplified during the compile step, but I assume this is part of a > more complex example where it is not possible to do this? Also note that > with r327254 we started importing global variables. Do you know why we > don't import in your case? I wonder if it has to do with it being CFI > inserted code? > > Teresa > > On Tue, Apr 17, 2018 at 9:17 AM <dmitry.mikulin at sony.com> wrote: > >> I watched Teresa’s talk on ThinLTO from last year’s CppCon, and it >> sounded like adding global variable information to the summaries was in the >> works, or at least in planning. Can someone (Teresa?) please share the >> current status? If it’s part of future plans, are there any specific >> proposals that can be picked up and worked on? >> >> Thanks! >> >> >> > On Apr 9, 2018, at 6:51 PM, via llvm-dev <llvm-dev at lists.llvm.org> >> wrote: >> > >> > Hi, >> > >> > I’m working on setting up ThinLTO+CFI for a C application which uses a >> lot of function pointers. While functionally it appears stable, it’s >> performance is significantly degraded, to the tune of double digit >> percentage points compared to regular LTO+CFI. >> > >> > Looking into possible causes I see that under ThinLTO+CFI iCall type >> checks almost always generate jump table entries for indirect calls, which >> creates another level of indirection for every such call. On top of that it >> breaks the link order layout because real function names point to jump >> table entries. It appears that I’m hitting a limitation in ThinLTO on how >> much information it can propagate across modules, particularly information >> about constants. In the example below, the fact that “i” is effectively a >> constant, is lost under ThinLTO, and the inlined copy of b.c:get_fptr() in >> a.c does not eliminate the conditional, which, for CFI purposes requires to >> generate a type check/jump table. >> > >> > I was wondering if there was a way to mitigate this limitation. >> > >> > a.c >> > ============================>> > typedef int (*fptr_t) (void); >> > fptr_t get_fptr(); >> > int main(int argc, char *argv[]) >> > { >> > fptr_t fp = get_fptr(); >> > return fp(); >> > } >> > >> > >> > b.c >> > ============================>> > typedef int (*fptr_t) (void); >> > int foo(void) { return 11; } >> > int bar(void) { return 22; } >> > >> > static fptr_t fptr = bar; >> > static int i = 53; >> > >> > fptr_t get_fptr(void) >> > { >> > if (i >= 0) >> > fptr = foo; >> > else >> > fptr = bar; >> > >> > return fptr; >> > } >> > >> > _______________________________________________ >> > LLVM Developers mailing list >> > llvm-dev at lists.llvm.org >> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >> > > -- > Teresa Johnson | Software Engineer | tejohnson at google.com | > 408-460-2413 > > >-- Teresa Johnson | Software Engineer | tejohnson at google.com | 408-460-2413 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180418/edb8959d/attachment.html>