Adve, Vikram Sadanand via llvm-dev
2016-Jul-22 19:15 UTC
[llvm-dev] ThinLTO status in trunk?
First, kudos on the ThinLTO results reported in your blog post — they’re impressive and the system sounds really well engineered. I’m starting to try it out on a large piece of software and I’d like to make sure I know what to expect. The blog said it will be available in clang-3.9 but both clang-3.8 and trunk seem to have some degree of support for it. What is the status of ThinLTO in 3.8 (preferably) or trunk (otherwise)? Are either of these usable with multiple threads? What about for a distributed build? Thanks! -—Vikram // Vikram S. Adve // Professor, Department of Computer Science // University of Illinois at Urbana-Champaign // vadve at illinois.edu // http://llvm.org
Hi Vikram, Thanks! I'm not sure what part got committed in the 3.8 timeframe - it looks like that was released back in March? A number of fixes have gone in since then so I would stick with a more recent version. The ThinLTO in trunk and (presumably 3.9 which seems to have not yet gone out?) is complete and working with multiple threads using gold. The distributed build support is also in (which needs support in the build system however), although I am fixing a few bugs right now that popped up in testing our internal apps. Which distributed build system do you use? We have support coming out in Bazel, which is our open sourced distributed build system. Let me know if you run into any issues or have any other questions! Teresa On Fri, Jul 22, 2016 at 12:15 PM, Adve, Vikram Sadanand via llvm-dev < llvm-dev at lists.llvm.org> wrote:> First, kudos on the ThinLTO results reported in your blog post — they’re > impressive and the system sounds really well engineered. > > I’m starting to try it out on a large piece of software and I’d like to > make sure I know what to expect. The blog said it will be available in > clang-3.9 but both clang-3.8 and trunk seem to have some degree of support > for it. What is the status of ThinLTO in 3.8 (preferably) or trunk > (otherwise)? Are either of these usable with multiple threads? What about > for a distributed build? > > Thanks! > > -—Vikram > > // Vikram S. Adve > // Professor, Department of Computer Science > // University of Illinois at Urbana-Champaign > // vadve at illinois.edu > // http://llvm.org > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-- Teresa Johnson | Software Engineer | tejohnson at google.com | 408-460-2413 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160722/537e007b/attachment.html>
Adve, Vikram Sadanand via llvm-dev
2016-Jul-22 21:19 UTC
[llvm-dev] ThinLTO status in trunk?
On Jul 22, 2016, at 2:36 PM, Teresa Johnson <tejohnson at google.com<mailto:tejohnson at google.com>> wrote: Hi Vikram, Thanks! I'm not sure what part got committed in the 3.8 timeframe - it looks like that was released back in March? A number of fixes have gone in since then so I would stick with a more recent version. The ThinLTO in trunk and (presumably 3.9 which seems to have not yet gone out?) is complete and working with multiple threads using gold. Sounds good. I will give this a spin. The distributed build support is also in (which needs support in the build system however), although I am fixing a few bugs right now that popped up in testing our internal apps. OK it will be a little while before I get to try that. And the build system for the set of applications I’m working with is quite painful, so I don’t know how long that will take, sigh. Which distributed build system do you use? We have support coming out in Bazel, which is our open sourced distributed build system. I don’t know yet — I’ll find out when I get to that stage. But it’s almost certainly something very simple or home-brewed. Let me know if you run into any issues or have any other questions! Teresa Thanks, -—Vikram // Vikram S. Adve // Professor, Department of Computer Science // University of Illinois at Urbana-Champaign // vadve at illinois.edu<mailto:vadve at illinois.edu> // http://llvm.org On Fri, Jul 22, 2016 at 12:15 PM, Adve, Vikram Sadanand via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote: First, kudos on the ThinLTO results reported in your blog post — they’re impressive and the system sounds really well engineered. I’m starting to try it out on a large piece of software and I’d like to make sure I know what to expect. The blog said it will be available in clang-3.9 but both clang-3.8 and trunk seem to have some degree of support for it. What is the status of ThinLTO in 3.8 (preferably) or trunk (otherwise)? Are either of these usable with multiple threads? What about for a distributed build? Thanks! -—Vikram // Vikram S. Adve // Professor, Department of Computer Science // University of Illinois at Urbana-Champaign // vadve at illinois.edu<mailto:vadve at illinois.edu> // http://llvm.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__llvm.org&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=b7uK7dJM4Fx7J_ehsuohEdD-6NdkoLyTwBFfHX-XKcc&m=px4LMbEtuCuyquW6qSGKnKyFZb4fKwVt3Wz8PbAyOLg&s=4cBsKK5PujU5xK3e6qJKaS_SAErck-yyileHZDHs-Gg&e=> _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.llvm.org_cgi-2Dbin_mailman_listinfo_llvm-2Ddev&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=b7uK7dJM4Fx7J_ehsuohEdD-6NdkoLyTwBFfHX-XKcc&m=px4LMbEtuCuyquW6qSGKnKyFZb4fKwVt3Wz8PbAyOLg&s=NtOBRnkjd1iF0YKpA2pFQcGM6o5sKpse1RSc5JaYXpc&e=> -- Teresa Johnson | Software Engineer | tejohnson at google.com<mailto:tejohnson at google.com> | 408-460-2413 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160722/26400b48/attachment.html>
> On Jul 22, 2016, at 12:36 PM, Teresa Johnson via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > Hi Vikram, > > Thanks! > > I'm not sure what part got committed in the 3.8 timeframe - it looks like that was released back in March?3.8 was branched in early January though. It has some of the work-in-progress for ThinLTO, it “could” work in simple cases I guess, but it hasn’t really been tested. Also, in 3.8, the state was basically what was presented at EuroLLVM, while (as explained in the blog post), we ended up redesigning it significantly. In 3.9 we changed the bitcode format and broke backward compatibility: 3.9 won’t be able to read a ThinLTO bitcode generated by 3.8 (this is possible only because “ThinLTO was not supported in 3.8”). — Mehdi> A number of fixes have gone in since then so I would stick with a more recent version. The ThinLTO in trunk and (presumably 3.9 which seems to have not yet gone out?) is complete and working with multiple threads using gold. > > The distributed build support is also in (which needs support in the build system however), although I am fixing a few bugs right now that popped up in testing our internal apps. Which distributed build system do you use? We have support coming out in Bazel, which is our open sourced distributed build system. > > Let me know if you run into any issues or have any other questions! > > Teresa > > > On Fri, Jul 22, 2016 at 12:15 PM, Adve, Vikram Sadanand via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: > First, kudos on the ThinLTO results reported in your blog post — they’re impressive and the system sounds really well engineered. > > I’m starting to try it out on a large piece of software and I’d like to make sure I know what to expect. The blog said it will be available in clang-3.9 but both clang-3.8 and trunk seem to have some degree of support for it. What is the status of ThinLTO in 3.8 (preferably) or trunk (otherwise)? Are either of these usable with multiple threads? What about for a distributed build? > > Thanks! > > -—Vikram > > // Vikram S. Adve > // Professor, Department of Computer Science > // University of Illinois at Urbana-Champaign > // vadve at illinois.edu <mailto:vadve at illinois.edu> > // http://llvm.org <http://llvm.org/> > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev> > > > > -- > Teresa Johnson | Software Engineer | tejohnson at google.com <mailto:tejohnson at google.com> | 408-460-2413 > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160722/ab333fdb/attachment.html>
Hi Teresa, Impressive results, indeed! (But no less is expected from an Itanium alumni... ;-)) One question, if you don't mind. In the blog post you wrote: "In a few cases ThinLTO even outperforms full LTO, most likely because the higher scalability of ThinLTO allows using a more aggressive backend optimization pipeline (similar to that of a non-LTO build)." Is it due to optimizations not working (in full LTO case) after hitting some internal limits? If so, my understanding is that these limits are expressed in units not affected by IPA representation (like function size, number of basic blocks, etc, *not* total memory consumption) -- how ThinLTO helps to stay within them? Yours, Andrey> 22 июля 2016 г., в 22:36, Teresa Johnson via llvm-dev <llvm-dev at lists.llvm.org> написал(а): > > Hi Vikram, > > Thanks! > > I'm not sure what part got committed in the 3.8 timeframe - it looks like that was released back in March? A number of fixes have gone in since then so I would stick with a more recent version. The ThinLTO in trunk and (presumably 3.9 which seems to have not yet gone out?) is complete and working with multiple threads using gold. > > The distributed build support is also in (which needs support in the build system however), although I am fixing a few bugs right now that popped up in testing our internal apps. Which distributed build system do you use? We have support coming out in Bazel, which is our open sourced distributed build system. > > Let me know if you run into any issues or have any other questions! > > Teresa > > >> On Fri, Jul 22, 2016 at 12:15 PM, Adve, Vikram Sadanand via llvm-dev <llvm-dev at lists.llvm.org> wrote: >> First, kudos on the ThinLTO results reported in your blog post — they’re impressive and the system sounds really well engineered. >> >> I’m starting to try it out on a large piece of software and I’d like to make sure I know what to expect. The blog said it will be available in clang-3.9 but both clang-3.8 and trunk seem to have some degree of support for it. What is the status of ThinLTO in 3.8 (preferably) or trunk (otherwise)? Are either of these usable with multiple threads? What about for a distributed build? >> >> Thanks! >> >> -—Vikram >> >> // Vikram S. Adve >> // Professor, Department of Computer Science >> // University of Illinois at Urbana-Champaign >> // vadve at illinois.edu >> // http://llvm.org >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > -- > Teresa Johnson | Software Engineer | tejohnson at google.com | 408-460-2413 > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160723/16215d52/attachment.html>