llvm dev - Jul 2016 - ThinLTO status in trunk?

First, kudos on the ThinLTO results reported in your blog post — they’re
impressive and the system sounds really well engineered.

I’m starting to try it out on a large piece of software and I’d like to make
sure I know what to expect.  The blog said it will be available in clang-3.9 but
both clang-3.8 and trunk seem to have some degree of support for it.  What is
the status of ThinLTO in 3.8 (preferably) or trunk (otherwise)?  Are either of
these usable with multiple threads?  What about for a distributed build?

Thanks!

-—Vikram

// Vikram S. Adve
// Professor, Department of Computer Science
// University of Illinois at Urbana-Champaign
// vadve at illinois.edu
// http://llvm.org

Hi Vikram,

Thanks!

I'm not sure what part got committed in the 3.8 timeframe - it looks like
that was released back in March? A number of fixes have gone in since then
so I would stick with a more recent version. The ThinLTO in trunk and
(presumably 3.9 which seems to have not yet gone out?) is complete and
working with multiple threads using gold.

The distributed build support is also in (which needs support in the build
system however), although I am fixing a few bugs right now that popped up
in testing our internal apps. Which distributed build system do you use? We
have support coming out in Bazel, which is our open sourced distributed
build system.

Let me know if you run into any issues or have any other questions!

Teresa


On Fri, Jul 22, 2016 at 12:15 PM, Adve, Vikram Sadanand via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> First, kudos on the ThinLTO results reported in your blog post — they’re
> impressive and the system sounds really well engineered.
>
> I’m starting to try it out on a large piece of software and I’d like to
> make sure I know what to expect.  The blog said it will be available in
> clang-3.9 but both clang-3.8 and trunk seem to have some degree of support
> for it.  What is the status of ThinLTO in 3.8 (preferably) or trunk
> (otherwise)?  Are either of these usable with multiple threads?  What about
> for a distributed build?
>
> Thanks!
>
> -—Vikram
>
> // Vikram S. Adve
> // Professor, Department of Computer Science
> // University of Illinois at Urbana-Champaign
> // vadve at illinois.edu
> // http://llvm.org
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>


-- 
Teresa Johnson |  Software Engineer |  tejohnson at google.com |  408-460-2413
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160722/537e007b/attachment.html>

On Jul 22, 2016, at 2:36 PM, Teresa Johnson <tejohnson at
google.com<mailto:tejohnson at google.com>> wrote:

Hi Vikram,

Thanks!

I'm not sure what part got committed in the 3.8 timeframe - it looks like
that was released back in March? A number of fixes have gone in since then so I
would stick with a more recent version. The ThinLTO in trunk and (presumably 3.9
which seems to have not yet gone out?) is complete and working with multiple
threads using gold.

Sounds good.  I will give this a spin.


The distributed build support is also in (which needs support in the build
system however), although I am fixing a few bugs right now that popped up in
testing our internal apps.

OK it will be a little while before I get to try that.  And the build system for
the set of applications I’m working with is quite painful, so I don’t know how
long that will take, sigh.


Which distributed build system do you use? We have support coming out in Bazel,
which is our open sourced distributed build system.

I don’t know yet — I’ll find out when I get to that stage.  But it’s almost
certainly something very simple or home-brewed.


Let me know if you run into any issues or have any other questions!

Teresa

Thanks,

-—Vikram

// Vikram S. Adve
// Professor, Department of Computer Science
// University of Illinois at Urbana-Champaign
// vadve at illinois.edu<mailto:vadve at illinois.edu>
// http://llvm.org




On Fri, Jul 22, 2016 at 12:15 PM, Adve, Vikram Sadanand via llvm-dev
<llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>>
wrote:
First, kudos on the ThinLTO results reported in your blog post — they’re
impressive and the system sounds really well engineered.

I’m starting to try it out on a large piece of software and I’d like to make
sure I know what to expect.  The blog said it will be available in clang-3.9 but
both clang-3.8 and trunk seem to have some degree of support for it.  What is
the status of ThinLTO in 3.8 (preferably) or trunk (otherwise)?  Are either of
these usable with multiple threads?  What about for a distributed build?

Thanks!

-—Vikram

// Vikram S. Adve
// Professor, Department of Computer Science
// University of Illinois at Urbana-Champaign
// vadve at illinois.edu<mailto:vadve at illinois.edu>
//
http://llvm.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__llvm.org&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=b7uK7dJM4Fx7J_ehsuohEdD-6NdkoLyTwBFfHX-XKcc&m=px4LMbEtuCuyquW6qSGKnKyFZb4fKwVt3Wz8PbAyOLg&s=4cBsKK5PujU5xK3e6qJKaS_SAErck-yyileHZDHs-Gg&e=>

_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.llvm.org_cgi-2Dbin_mailman_listinfo_llvm-2Ddev&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=b7uK7dJM4Fx7J_ehsuohEdD-6NdkoLyTwBFfHX-XKcc&m=px4LMbEtuCuyquW6qSGKnKyFZb4fKwVt3Wz8PbAyOLg&s=NtOBRnkjd1iF0YKpA2pFQcGM6o5sKpse1RSc5JaYXpc&e=>



--
Teresa Johnson |         Software Engineer |     tejohnson at
google.com<mailto:tejohnson at google.com> |     408-460-2413

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160722/26400b48/attachment.html>

> On Jul 22, 2016, at 12:36 PM, Teresa Johnson via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hi Vikram,
> 
> Thanks!
> 
> I'm not sure what part got committed in the 3.8 timeframe - it looks
like that was released back in March?
3.8 was branched in early January though. 
It has some of the work-in-progress for ThinLTO, it “could” work in simple cases
I guess, but it hasn’t really been tested.
Also, in 3.8, the state was basically what was presented at EuroLLVM, while (as
explained in the blog post), we ended up redesigning it significantly.
In 3.9 we changed the bitcode format and broke backward compatibility: 3.9 won’t
be able to read a ThinLTO bitcode generated by 3.8 (this is possible only
because “ThinLTO was not supported in 3.8”).

— 
Mehdi



> A number of fixes have gone in since then so I would stick with a more
recent version. The ThinLTO in trunk and (presumably 3.9 which seems to have not
yet gone out?) is complete and working with multiple threads using gold.
> 
> The distributed build support is also in (which needs support in the build
system however), although I am fixing a few bugs right now that popped up in
testing our internal apps. Which distributed build system do you use? We have
support coming out in Bazel, which is our open sourced distributed build system.
> 
> Let me know if you run into any issues or have any other questions!
> 
> Teresa
> 
> 
> On Fri, Jul 22, 2016 at 12:15 PM, Adve, Vikram Sadanand via llvm-dev
<llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
wrote:
> First, kudos on the ThinLTO results reported in your blog post — they’re
impressive and the system sounds really well engineered.
> 
> I’m starting to try it out on a large piece of software and I’d like to
make sure I know what to expect.  The blog said it will be available in
clang-3.9 but both clang-3.8 and trunk seem to have some degree of support for
it.  What is the status of ThinLTO in 3.8 (preferably) or trunk (otherwise)? 
Are either of these usable with multiple threads?  What about for a distributed
build?
> 
> Thanks!
> 
> -—Vikram
> 
> // Vikram S. Adve
> // Professor, Department of Computer Science
> // University of Illinois at Urbana-Champaign
> // vadve at illinois.edu <mailto:vadve at illinois.edu>
> // http://llvm.org <http://llvm.org/>
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> 
> 
> 
> -- 
> Teresa Johnson |	 Software Engineer |	 tejohnson at google.com
<mailto:tejohnson at google.com> |	 408-460-2413
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160722/ab333fdb/attachment.html>

Hi Teresa,

Impressive results, indeed! (But no less is expected from an Itanium alumni...
;-))

One question, if you don't mind. In the blog post you wrote: "In a few
cases ThinLTO even outperforms full LTO, most likely because the higher
scalability of ThinLTO allows using a more aggressive backend optimization
pipeline (similar to that of a non-LTO build)." Is it due to optimizations
not working (in full LTO case) after hitting some internal limits? If so, my
understanding is that these limits are expressed in units not affected by IPA
representation (like function size, number of basic blocks, etc, *not* total
memory consumption) -- how ThinLTO helps to stay within them?

Yours,
Andrey
> 22 июля 2016 г., в 22:36, Teresa Johnson via llvm-dev <llvm-dev at
lists.llvm.org> написал(а):
> 
> Hi Vikram,
> 
> Thanks!
> 
> I'm not sure what part got committed in the 3.8 timeframe - it looks
like that was released back in March? A number of fixes have gone in since then
so I would stick with a more recent version. The ThinLTO in trunk and
(presumably 3.9 which seems to have not yet gone out?) is complete and working
with multiple threads using gold.
> 
> The distributed build support is also in (which needs support in the build
system however), although I am fixing a few bugs right now that popped up in
testing our internal apps. Which distributed build system do you use? We have
support coming out in Bazel, which is our open sourced distributed build system.
> 
> Let me know if you run into any issues or have any other questions!
> 
> Teresa
> 
> 
>> On Fri, Jul 22, 2016 at 12:15 PM, Adve, Vikram Sadanand via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>> First, kudos on the ThinLTO results reported in your blog post —
they’re impressive and the system sounds really well engineered.
>> 
>> I’m starting to try it out on a large piece of software and I’d like to
make sure I know what to expect.  The blog said it will be available in
clang-3.9 but both clang-3.8 and trunk seem to have some degree of support for
it.  What is the status of ThinLTO in 3.8 (preferably) or trunk (otherwise)? 
Are either of these usable with multiple threads?  What about for a distributed
build?
>> 
>> Thanks!
>> 
>> -—Vikram
>> 
>> // Vikram S. Adve
>> // Professor, Department of Computer Science
>> // University of Illinois at Urbana-Champaign
>> // vadve at illinois.edu
>> // http://llvm.org
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 
> 
> 
> -- 
> Teresa Johnson |	 Software Engineer |	 tejohnson at google.com |	
408-460-2413
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160723/16215d52/attachment.html>