llvm dev - Sep 2019 - [RFC] Propeller: A frame work for Post Link Optimizations

On Thu, Sep 26, 2019 at 6:16 PM Eli Friedman <efriedma at quicinc.com>
wrote:
>
> > -----Original Message-----
> > From: Sriraman Tallam <tmsriram at google.com>
> > Sent: Thursday, September 26, 2019 5:31 PM
> > To: Eli Friedman <efriedma at quicinc.com>
> > Cc: Xinliang David Li <xinliangli at gmail.com>; llvm-dev
<llvm-dev at lists.llvm.org>
> > Subject: [EXT] Re: [llvm-dev] [RFC] Propeller: A frame work for Post
Link
> > Optimizations
> >
> > For more details, We strongly considered this.  We could run something
> > like a thin link in thin lto figure out the global layout and hand out
> > the relevant  subsets of the global decision  to each module.  This
> > looked more complicated and the individual pieces from each module
> > should still be globally laid out again by the linker.  This
> > constraints us on what we can do for layout and also does not work
> > well with future optimizations like global alignment like David
> > pointed out.
>
> Basically, you need to do branch relaxation etc. in the linker because the
layout isn't sufficiently finalized until then?
Yes, this is correct.  Relaxation is needed because the layout is not
finalized until link time.




And you *only* need to rewrite branches this way; never anything else,
because other instructions don't need relaxation?

I cannot think of any other instance where such relaxation is needed.
>
> On most targets, we do branch relaxation in the compiler, unlike x86 which
does it in the assembler.  I guess we could teach the compiler to emit some
specific "relaxed" pattern, and teach the linker to reverse it.  Might
be a little tricky to handle certain odd cases well; currently, for example, on
AArch64 we sometimes insert a cpsr clobber as part of relaxation.  I'm not
really happy with having to reimplement that for every target, but if we're
sure it's limited to branches, I guess it's okay.
Understood.
>
> > > Why are you proposing to add a bunch of options to clang to
manipulate LLVM
> > code generation, given none of those options are actually used for
Propeller
> > workflows?
> >
> > Where do you suggest labelling and section options should exist?  We
> > looked at  basic block sections to be similar to function sections in
> > terms of option handling?
>
> Generating bitcode with/without propeller doesn't actually affect the
generated bitcode, right?  So you could say that Propeller is enabled with
"-Wl,--enable-propeller", and not change clang at all.  I'm not a
fan of adding options just because we can.  If basic block sections are only
used as a sort of secret handshake between the propeller compiler and the
propeller linker, we can change that interface later, if we want; if we expose
it as a clang option, we're committing to making basic block sections
continue to work the same way until the end of time.
The generated MIR code is slightly different as the later passes have
more CFI instructions, basic block labels and extra direct jumps which
would have been fall throughs otherwise.  So, some llvm option is
needed.

I envisioned that basic block sections could also be useful on its own
outside of propeller.   Hence, we made it like function-sections with
a separate option -fbasicblock-sections.  The propeller option is an
umbrella option to make it easy to invoke Propeller.

Thanks
Sri
>
> -Eli

> -----Original Message-----
> From: Sriraman Tallam <tmsriram at google.com>
> Sent: Friday, September 27, 2019 9:43 AM
> To: Eli Friedman <efriedma at quicinc.com>
> Cc: Xinliang David Li <xinliangli at gmail.com>; llvm-dev
<llvm-dev at lists.llvm.org>
> Subject: [EXT] Re: [llvm-dev] [RFC] Propeller: A frame work for Post Link
> Optimizations
>
> >
> > > > Why are you proposing to add a bunch of options to clang to
manipulate
> LLVM
> > > code generation, given none of those options are actually used
for Propeller
> > > workflows?
> > >
> > > Where do you suggest labelling and section options should exist? 
We
> > > looked at  basic block sections to be similar to function
sections in
> > > terms of option handling?
> >
> > Generating bitcode with/without propeller doesn't actually affect
the
> generated bitcode, right?  So you could say that Propeller is enabled with
"-Wl,--
> enable-propeller", and not change clang at all.  I'm not a fan of
adding options
> just because we can.  If basic block sections are only used as a sort of
secret
> handshake between the propeller compiler and the propeller linker, we can
> change that interface later, if we want; if we expose it as a clang option,
we're
> committing to making basic block sections continue to work the same way
until
> the end of time.
>
> The generated MIR code is slightly different as the later passes have
> more CFI instructions, basic block labels and extra direct jumps which
> would have been fall throughs otherwise.  So, some llvm option is
> needed.
At link (LTO codegen) time, yes.  But clang's bitcode generation doesn't
change; only LTO code generation in lld changes.
> I envisioned that basic block sections could also be useful on its own
> outside of propeller.   Hence, we made it like function-sections with
> a separate option -fbasicblock-sections.  The propeller option is an
> umbrella option to make it easy to invoke Propeller.
I don't think this is really true.  Basic block labels means "insert
labels that are useful for propeller profiling", and basic block sections
means "split the function into multiple chunks that are convenient for
propeller optimization".  So it's really only useful for propeller-like
workflows.  And I'd be concerned that future changes to propeller could
affect the behavior of these options in the future, if the behavior isn't
specified in some rigorous way.

In any case, if we're adding flags to clang other than "enable
propeller", I think we should have a separate thread on cfe-dev to discuss
them. We don't expose every possible LLVM optimization and code generation
option through clang. Users have a strong expectation of stability for clang
options, and that means we need to evaluate each new option carefully, to decide
whether it's something we really want to support indefinitely.

-Eli

On Fri, Sep 27, 2019 at 1:16 PM Eli Friedman <efriedma at quicinc.com>
wrote:
>
> > -----Original Message-----
> > From: Sriraman Tallam <tmsriram at google.com>
> > Sent: Friday, September 27, 2019 9:43 AM
> > To: Eli Friedman <efriedma at quicinc.com>
> > Cc: Xinliang David Li <xinliangli at gmail.com>; llvm-dev
<llvm-dev at lists.llvm.org>
> > Subject: [EXT] Re: [llvm-dev] [RFC] Propeller: A frame work for Post
Link
> > Optimizations
> >
> > >
> > > > > Why are you proposing to add a bunch of options to
clang to manipulate
> > LLVM
> > > > code generation, given none of those options are actually
used for Propeller
> > > > workflows?
> > > >
> > > > Where do you suggest labelling and section options should
exist?  We
> > > > looked at  basic block sections to be similar to function
sections in
> > > > terms of option handling?
> > >
> > > Generating bitcode with/without propeller doesn't actually
affect the
> > generated bitcode, right?  So you could say that Propeller is enabled
with "-Wl,--
> > enable-propeller", and not change clang at all.  I'm not a
fan of adding options
> > just because we can.  If basic block sections are only used as a sort
of secret
> > handshake between the propeller compiler and the propeller linker, we
can
> > change that interface later, if we want; if we expose it as a clang
option, we're
> > committing to making basic block sections continue to work the same
way until
> > the end of time.
> >
> > The generated MIR code is slightly different as the later passes have
> > more CFI instructions, basic block labels and extra direct jumps which
> > would have been fall throughs otherwise.  So, some llvm option is
> > needed.
>
> At link (LTO codegen) time, yes.  But clang's bitcode generation
doesn't change; only LTO code generation in lld changes.
I see what you mean here.
>
> > I envisioned that basic block sections could also be useful on its own
> > outside of propeller.   Hence, we made it like function-sections with
> > a separate option -fbasicblock-sections.  The propeller option is an
> > umbrella option to make it easy to invoke Propeller.
>
> I don't think this is really true.  Basic block labels means
"insert labels that are useful for propeller profiling", and basic
block sections means "split the function into multiple chunks that are
convenient for propeller optimization".  So it's really only useful for
propeller-like workflows.  And I'd be concerned that future changes to
propeller could affect the behavior of these options in the future, if the
behavior isn't specified in some rigorous way.
The idea of basic block sections was seeded by Rui Ueyama.  When basic
block sections was originally looked at, Propeller was not designed.
We looked at basic block sections as finer granularity than function
sections.  Section prefix based code layout where you just create
smaller code sections and let the linker do what it does today would
have much better results with basic block sections.  Symbol ordering
file with basic block sections to do random orderings can be done
without invoking Propeller.  Function sections has found uses after it
was invented like Identical Code Folding.
>
> In any case, if we're adding flags to clang other than "enable
propeller", I think we should have a separate thread on cfe-dev to discuss
them. We don't expose every possible LLVM optimization and code generation
option through clang. Users have a strong expectation of stability for clang
options, and that means we need to evaluate each new option carefully, to decide
whether it's something we really want to support indefinitely.
Understood.

Thanks
Sri
>
> -Eli

On Fri, Sep 27, 2019 at 1:16 PM Eli Friedman <efriedma at quicinc.com>
wrote:
> > -----Original Message-----
> > From: Sriraman Tallam <tmsriram at google.com>
> > Sent: Friday, September 27, 2019 9:43 AM
> > To: Eli Friedman <efriedma at quicinc.com>
> > Cc: Xinliang David Li <xinliangli at gmail.com>; llvm-dev <
> llvm-dev at lists.llvm.org>
> > Subject: [EXT] Re: [llvm-dev] [RFC] Propeller: A frame work for Post
Link
> > Optimizations
> >
> > >
> > > > > Why are you proposing to add a bunch of options to
clang to
> manipulate
> > LLVM
> > > > code generation, given none of those options are actually
used for
> Propeller
> > > > workflows?
> > > >
> > > > Where do you suggest labelling and section options should
exist?  We
> > > > looked at  basic block sections to be similar to function
sections in
> > > > terms of option handling?
> > >
> > > Generating bitcode with/without propeller doesn't actually
affect the
> > generated bitcode, right?  So you could say that Propeller is enabled
> with "-Wl,--
> > enable-propeller", and not change clang at all.  I'm not a
fan of adding
> options
> > just because we can.  If basic block sections are only used as a sort
of
> secret
> > handshake between the propeller compiler and the propeller linker, we
can
> > change that interface later, if we want; if we expose it as a clang
> option, we're
> > committing to making basic block sections continue to work the same
way
> until
> > the end of time.
> >
> > The generated MIR code is slightly different as the later passes have
> > more CFI instructions, basic block labels and extra direct jumps which
> > would have been fall throughs otherwise.  So, some llvm option is
> > needed.
>
> At link (LTO codegen) time, yes.  But clang's bitcode generation
doesn't
> change; only LTO code generation in lld changes.
>
After having spent more time on this, I think what you are suggesting is to
impose this work-flow for Propeller always:

Step 1: generate labels binary

clang -emit-llvm -c -O2 foo.cc  bar.cc (generates optimized foo.bc and
bar.bc)
./lld foo.bc bar.bc --basicblock-labels -o a.out.labels (generates labels
binary)

Step 2: Profile into perf.data just like now

Step 3: Convert profiles just like now

Step 4: Re-optimize with propeller by invoking lld directly
./lld foo.bc bar.bc --basicblock-sections --profile=perf.propeller -o a.out

If this workflow is adopted we might be able to get away without clang
options but I still think Step 1 could use a clang option to combine the 2
steps into one and may be one more option in Step 4 might be useful if we
want the invocation to be via clang.

Notice how this work-flow is very similar to ThinLTO's, and even thinlto
needs command-line options, -flto=thin, to break this down into the steps.

However, the issue with this work-flow imposes LTO like workflow on
Propeller. Propeller is LTO agnostic and works even without LTO or regular
FDO. The above does not support the simple workflows where you build each
module with -O3 and then dump the native object files and link.  I think it
would be beneficial to allow Propeller to be used everywhere by modifying
the make files to adopt it rather than constrain the work flow.

Thanks
Sri



>
> > I envisioned that basic block sections could also be useful on its own
> > outside of propeller.   Hence, we made it like function-sections with
> > a separate option -fbasicblock-sections.  The propeller option is an
> > umbrella option to make it easy to invoke Propeller.
>
> I don't think this is really true.  Basic block labels means
"insert
> labels that are useful for propeller profiling", and basic block
sections
> means "split the function into multiple chunks that are convenient for
> propeller optimization".  So it's really only useful for
propeller-like
> workflows.  And I'd be concerned that future changes to propeller could
> affect the behavior of these options in the future, if the behavior
isn't
> specified in some rigorous way.
>
> In any case, if we're adding flags to clang other than "enable
propeller",
> I think we should have a separate thread on cfe-dev to discuss them. We
> don't expose every possible LLVM optimization and code generation
option
> through clang. Users have a strong expectation of stability for clang
> options, and that means we need to evaluate each new option carefully, to
> decide whether it's something we really want to support indefinitely.
>
> -Eli
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190928/8582020c/attachment.html>