thr3ads.net - llvm dev - [llvm-dev] getting nowhere with thinLTO [Nov 2017]

If this information is useful, please help other people find it:
Share via:

Davis, Alan via llvm-dev

2017-Nov-10 17:12 UTC

[llvm-dev] getting nowhere with thinLTO

Teresa, thanks, that’s helpful.

I discovered the resolution-based API and am now converting to that. I can also
see that the summaries are in the bitcode files, via llvm-bcanalyzer.

I hope the llvm-dis support will happen. Also it would be nice if the
documentation were more up-to-date. But this will definitely help get me moving
again.

For whatever reason in this simple example t3 is not being inlined, but I can
set that aside for now.

// t2.c:
int t3();

int t2()
{
   if (t3()) return 1;
   else return 0;
}

// t3.c:
int t3()
{
   return 10;
}

% clang -c -O2 -flto=thin t2.c t3.c
% llvm-lto t2.o t3.o --exported-symbol=t2 --thinlto-action=run
-debug-only=function-import -thinlto-save-temps=.
Live root: 11079814748834529807
2 symbols Live, and 1 symbols Dead
Computing import for Module 't3.o'
Ignores Dead GUID: 8059958135287337232
Initialize import for 14314392591860533259
Computing import for Module 't2.o'
Initialize import for 11079814748834529807
Import/Export lists for 2 modules:
* Module t3.o exports 0 functions. Imports from 0 modules.
* Module t2.o exports 0 functions. Imports from 0 modules.
Starting import for Module t2.o
Imported 0 functions for Module t2.o
Starting import for Module t3.o
Imported 0 functions for Module t3.o

-Alan

From: Teresa Johnson [mailto:tejohnson at google.com]
Sent: Thursday, November 09, 2017 10:33 AM
To: Davis, Alan
Cc: llvm-dev at lists.llvm.org; Peter Collingbourne
Subject: [EXTERNAL] Re: [llvm-dev] getting nowhere with thinLTO



On Wed, Nov 8, 2017 at 2:31 PM, Davis, Alan via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
I’m trying to incorporate thinLTO into our proprietary linker. I can’t seem to
make it work at all (although I do have the ‘thick’ form working).

Sorry to hear that, hopefully we can help get that sorted out.

When I try to invoke the ThinGenerator API it crashes somewhere way down in the
bowels of IPO. So stepping away from the linker, I am just trying to explore the
processing steps using standalone off-the shelf llvm built from tip-of-tree.

For a new implementation I would highly recommend using the new LTO API, which
utilizes linker resolution info and is better supported at this point. See
http://llvm-cs.pcc.me.uk/include/llvm/LTO/LTO.h#234.
See also gold-plugin.cpp and the lld sources - both utilize this API. To test it
use the llvm-lto2 tool instead of llvm-lto. Because this interface uses linker
resolution info, invoking via llvm-lto2 is more tedious since resolutions must
be specified for all symbols in the input files. See invocations in the llvm
tests for examples.


1. When I do ‘clang -c -flto=thin t1.c’ on a simple input file, the resulting .o
file is a bitcode file as expected, but when I llvm-dis it there is no evidence
of the module summary information. Is there a way to view the summary index for
a given module?

Unfortunately not. This has come up a few times and the long-term plan is to
serialize it through llvm assembly, but no one has had the bandwidth to
completely flesh this out yet, and there was disagreement about what to do in
the short term, so nothing went in. For now, you have to use llvm-bcanalyzer
-dump, but that is a raw output format, although it will at least allow you to
confirm the summary sections are there.


2. When I take two or three .o files generated using -flto=thin and pass them to
the standalone llvm-lto tool, as in:
   llvm-lto --thinlto t2.o t3.o --exported-symbol=t2 -o index
the combined index is (apparently) empty.

Use llvm-bcanalyzer -dump to see the contents of the output index. It
shouldn't be empty.

3. When I take a program and pass all the .o files to llvm-lto
--thinlto-action=run, I get recompiled .o files for each but there seems to be
no IPO (no inlining, etc).

How do you know? BTW you can invoke llvm-lto with the
-thinlto-save-temps=<pathprefix> option to get bitcode files emitted from
the backends after the major phases of ThinLTO backend compilation. I.e. to see
what was imported.
You can also pass internal llvm options to llvm-lto, i.e.
-debug-only=function-import will give you debug info from the function importing
done both during the thin link and during the thinlto backends.

Let me know what you find.

Teresa


What am I missing?

-Alan

_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev



--
Teresa Johnson |

 Software Engineer |

 tejohnson at google.com<mailto:tejohnson at google.com> |

 408-460-2413<tel:(408)%20460-2413>


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171110/ea1c562d/attachment.html>

Teresa Johnson via llvm-dev

2017-Nov-10 20:09 UTC

head link

[llvm-dev] getting nowhere with thinLTO

Hi Alan,

I am seeing the inline of t3 into t2 after it is imported. Look at the
post-opt bitcode file generated by -thinlto-save-temps:

llvm-dis -o - .0.4.opt.bc
; ModuleID = '.0.4.opt.bc'
source_filename = "t2.c"
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-unknown-linux-gnu"

; Function Attrs: norecurse nounwind readnone uwtable
define i32 @t2() local_unnamed_addr #0 {
entry:
  ret i32 1
}
...

Also, if you do -debug-only=inline instead of -debug-only=function-import,
you will see:

llvm-lto t2.o t3.o --exported-symbol=t2 --thinlto-action=run
-debug-only=inline -thinlto-save-temps=.Inliner visiting SCC: t3: 0 call
sites.
Inliner visiting SCC: t3: 0 call sites.
Inliner visiting SCC: INDIRECTNODE: 0 call sites.
Inliner visiting SCC: t2: 1 call sites.
    Inlining: cost=-30, thres=337, Call:   %call = tail call i32 @t3() #2
Inliner visiting SCC: INDIRECTNODE: 0 call sites.

Are you seeing something different?
Teresa

On Fri, Nov 10, 2017 at 9:12 AM, Davis, Alan <adavis at ti.com> wrote:
> Teresa, thanks, that’s helpful.
>
>
>
> I discovered the resolution-based API and am now converting to that. I can
> also see that the summaries are in the bitcode files, via llvm-bcanalyzer.
>
>
>
> I hope the llvm-dis support will happen. Also it would be nice if the
> documentation were more up-to-date. But this will definitely help get me
> moving again.
>
>
>
> For whatever reason in this simple example t3 is not being inlined, but I
> can set that aside for now.
>
>
>
> // t2.c:
>
> int t3();
>
>
>
> int t2()
>
> {
>
>    if (t3()) return 1;
>
>    else return 0;
>
> }
>
>
>
> // t3.c:
>
> int t3()
>
> {
>
>    return 10;
>
> }
>
>
>
> % clang -c -O2 -flto=thin t2.c t3.c
>
> % llvm-lto t2.o t3.o --exported-symbol=t2 --thinlto-action=run
> -debug-only=function-import -thinlto-save-temps=.
>
> Live root: 11079814748834529807
>
> 2 symbols Live, and 1 symbols Dead
>
> Computing import for Module 't3.o'
>
> Ignores Dead GUID: 8059958135287337232
>
> Initialize import for 14314392591860533259
>
> Computing import for Module 't2.o'
>
> Initialize import for 11079814748834529807
>
> Import/Export lists for 2 modules:
>
> * Module t3.o exports 0 functions. Imports from 0 modules.
>
> * Module t2.o exports 0 functions. Imports from 0 modules.
>
> Starting import for Module t2.o
>
> Imported 0 functions for Module t2.o
>
> Starting import for Module t3.o
>
> Imported 0 functions for Module t3.o
>
>
>
> -Alan
>
>
>
> *From:* Teresa Johnson [mailto:tejohnson at google.com]
> *Sent:* Thursday, November 09, 2017 10:33 AM
> *To:* Davis, Alan
> *Cc:* llvm-dev at lists.llvm.org; Peter Collingbourne
> *Subject:* [EXTERNAL] Re: [llvm-dev] getting nowhere with thinLTO
>
>
>
>
>
>
>
> On Wed, Nov 8, 2017 at 2:31 PM, Davis, Alan via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> I’m trying to incorporate thinLTO into our proprietary linker. I can’t
> seem to make it work at all (although I do have the ‘thick’ form working).
>
>
>
> Sorry to hear that, hopefully we can help get that sorted out.
>
>
>
> When I try to invoke the ThinGenerator API it crashes somewhere way down
> in the bowels of IPO. So stepping away from the linker, I am just trying to
> explore the processing steps using standalone off-the shelf llvm built from
> tip-of-tree.
>
>
>
> For a new implementation I would highly recommend using the new LTO API,
> which utilizes linker resolution info and is better supported at this
> point. See http://llvm-cs.pcc.me.uk/include/llvm/LTO/LTO.h#234.
>
> See also gold-plugin.cpp and the lld sources - both utilize this API. To
> test it use the llvm-lto2 tool instead of llvm-lto. Because this interface
> uses linker resolution info, invoking via llvm-lto2 is more tedious since
> resolutions must be specified for all symbols in the input files. See
> invocations in the llvm tests for examples.
>
>
>
>
>
> 1. When I do ‘clang -c -flto=thin t1.c’ on a simple input file, the
> resulting .o file is a bitcode file as expected, but when I llvm-dis it
> there is no evidence of the module summary information. Is there a way to
> view the summary index for a given module?
>
>
>
> Unfortunately not. This has come up a few times and the long-term plan is
> to serialize it through llvm assembly, but no one has had the bandwidth to
> completely flesh this out yet, and there was disagreement about what to do
> in the short term, so nothing went in. For now, you have to use
> llvm-bcanalyzer -dump, but that is a raw output format, although it will at
> least allow you to confirm the summary sections are there.
>
>
>
>
>
> 2. When I take two or three .o files generated using -flto=thin and pass
> them to the standalone llvm-lto tool, as in:
>
>    llvm-lto --thinlto t2.o t3.o --exported-symbol=t2 -o index
>
> the combined index is (apparently) empty.
>
>
>
> Use llvm-bcanalyzer -dump to see the contents of the output index. It
> shouldn't be empty.
>
>
>
> 3. When I take a program and pass all the .o files to llvm-lto
> --thinlto-action=run, I get recompiled .o files for each but there seems to
> be no IPO (no inlining, etc).
>
>
>
> How do you know? BTW you can invoke llvm-lto with the
-thinlto-save-temps=<pathprefix>
> option to get bitcode files emitted from the backends after the major
> phases of ThinLTO backend compilation. I.e. to see what was imported.
>
> You can also pass internal llvm options to llvm-lto,
> i.e. -debug-only=function-import will give you debug info from the
> function importing done both during the thin link and during the thinlto
> backends.
>
>
>
> Let me know what you find.
>
>
>
> Teresa
>
>
>
>
>
> What am I missing?
>
>
>
> -Alan
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
>
>
> --
>
> Teresa Johnson |
>
>  Software Engineer |
>
>  tejohnson at google.com |
>
>  408-460-2413 <(408)%20460-2413>
>
>
>


-- 
Teresa Johnson |  Software Engineer |  tejohnson at google.com |  408-460-2413
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171110/2e98ca43/attachment.html>

Tobias Edler von Koch via llvm-dev

2017-Nov-14 17:26 UTC

head link

[llvm-dev] getting nowhere with thinLTO

Hi Alan,

On 11/10/2017 11:12 AM, Davis, Alan via llvm-dev wrote:>
> For whatever reason in this simple example t3 is not being inlined, 
> but I can set that aside for now.
>
> // t2.c:
>
> int t3();
>
Does inlining happen if you change this to:

int t3(*void*);

Without the void, it's a non-prototype declaration (a pre-ANSI C 
holdover) and will get compiled to "declare void t3(...)". I'm
guessing
you'll see a bitcast at the call site which the inliner can't look
through.


Tobias

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171114/2594c462/attachment.html>

llvm dev - Nov 2017 - getting nowhere with thinLTO

[llvm-dev] getting nowhere with thinLTO

[llvm-dev] getting nowhere with thinLTO

[llvm-dev] getting nowhere with thinLTO