llvm dev - Jul 2017 - [ThinLTO] Making ThinLTO functions not fail hasExactDefinition (specifically preventing it from being derefined)

Hi Charles,

On Tue, Jul 11, 2017 at 12:27 PM, Davide Italiano <davide at freebsd.org>
wrote:
>> I'm working on adding interprocedural FunctionAttrs optimization
>> (http://llvm-cs.pcc.me.uk/lib/Transforms/IPO/FunctionAttrs.cpp) to
ThinLTO
>> so it does something similar to what LTO is doing
>> (https://bugs.llvm.org/show_bug.cgi?id=33648). I've hit a problem
with how
>> the FunctionAttrs optimization expects linkage types.
>>
>> In ThinLTO since the linkage type is set to External or
AvailableExternally
FunctionAttrs should still kick in for external linkage, can you post
an IR sample of where that's not happening (since that's a bug IIUC)?

As far as available_externally is concerned, preventing FunctionAttrs
on them is by design and I'm not sure if there is anything in ThinLTO
to fix that (assuming that the functions are, in fact,
available_externally).

There is some background information on this issue at
https://www.playingwithpointers.com/ipo-and-derefinement.html

-- Sanjoy

>> it is unable to apply any of the functionattr optimizations that LTO
can
>> (since LTO is able to convert everything to Internal). This is a
problem
>> because every functionattr optimization expects an exact definition by
>> calling hasExactDefinition
>> (http://llvm-cs.pcc.me.uk/include/llvm/IR/GlobalValue.h#387) which
doesn't
>> want a derefined function (mayBeDerefined
>> http://llvm-cs.pcc.me.uk/include/llvm/IR/GlobalValue.h#119). According
to
>> the current definition, an AvailableExternally function could be
derefined
>> so hasExactDefinition fails.
>>
>> Does anyone have any ideas on how to make a function with
>> External/AvailableExternally linkage type have an exact definition?
>>
>
> + Sanjoy, who did all the derefinement work originally.
>
> --
> Davide

2017-07-11 14:54 GMT-07:00 Sanjoy Das via llvm-dev <llvm-dev at
lists.llvm.org>
:
> Hi Charles,
>
> On Tue, Jul 11, 2017 at 12:27 PM, Davide Italiano <davide at
freebsd.org>
> wrote:
> >> I'm working on adding interprocedural FunctionAttrs
optimization
> >> (http://llvm-cs.pcc.me.uk/lib/Transforms/IPO/FunctionAttrs.cpp) to
> ThinLTO
> >> so it does something similar to what LTO is doing
> >> (https://bugs.llvm.org/show_bug.cgi?id=33648). I've hit a
problem with
> how
> >> the FunctionAttrs optimization expects linkage types.
> >>
> >> In ThinLTO since the linkage type is set to External or
> AvailableExternally
>
> FunctionAttrs should still kick in for external linkage, can you post
> an IR sample of where that's not happening (since that's a bug
IIUC)?
>
> As far as available_externally is concerned, preventing FunctionAttrs
> on them is by design and I'm not sure if there is anything in ThinLTO
> to fix that (assuming that the functions are, in fact,
> available_externally).
>
I think ThinLTO can fix most of it. My idea was to propagate function
attribute even for simple function declarations!

The reason I believe LTO can circumvent the "derefinement" issue is
because
the attributes computed after optimization during the compile phase are
propagated only after the linker decided which "version" of an ODR
function
we will pick in the final binary.

I believe this is valid, as long as later optimizations in the ThinLTO
backend can't "undo" an attribute (or can't change the
attribute list to be
less restrictive). For example if a function is `read_none` after the
compile phase. During the ThinLTO thin-link we can propagate this attribute
to every uses (after deciding that this is the copy we want to keep for
this function). However if a later optimization invalidate the read_none
attributes, it's over. I doubt this can happen though (as in general it
would invalidate previous optimizations).

-- 
Mehdi



>
> There is some background information on this issue at
> https://www.playingwithpointers.com/ipo-and-derefinement.html
>
> -- Sanjoy
>
>
> >> it is unable to apply any of the functionattr optimizations that
LTO can
> >> (since LTO is able to convert everything to Internal). This is a
problem
> >> because every functionattr optimization expects an exact
definition by
> >> calling hasExactDefinition
> >> (http://llvm-cs.pcc.me.uk/include/llvm/IR/GlobalValue.h#387) which
> doesn't
> >> want a derefined function (mayBeDerefined
> >> http://llvm-cs.pcc.me.uk/include/llvm/IR/GlobalValue.h#119).
According
> to
> >> the current definition, an AvailableExternally function could be
> derefined
> >> so hasExactDefinition fails.
> >>
> >> Does anyone have any ideas on how to make a function with
> >> External/AvailableExternally linkage type have an exact
definition?
> >>
> >
> > + Sanjoy, who did all the derefinement work originally.
> >
> > --
> > Davide
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170712/396edd2b/attachment.html>

Hey Mehdi,
> I think ThinLTO can fix most of it. My idea was to propagate function
attribute even for simple function declarations!

Alright - I'll look into this approach - seems like the way too go.

Thanks,
Charles


On Wed, Jul 12, 2017 at 12:51 PM, Mehdi AMINI via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
>
>
> 2017-07-11 14:54 GMT-07:00 Sanjoy Das via llvm-dev <
> llvm-dev at lists.llvm.org>:
>
>> Hi Charles,
>>
>> On Tue, Jul 11, 2017 at 12:27 PM, Davide Italiano <davide at
freebsd.org>
>> wrote:
>> >> I'm working on adding interprocedural FunctionAttrs
optimization
>> >>
(http://llvm-cs.pcc.me.uk/lib/Transforms/IPO/FunctionAttrs.cpp) to
>> ThinLTO
>> >> so it does something similar to what LTO is doing
>> >> (https://bugs.llvm.org/show_bug.cgi?id=33648). I've hit a
problem
>> with how
>> >> the FunctionAttrs optimization expects linkage types.
>> >>
>> >> In ThinLTO since the linkage type is set to External or
>> AvailableExternally
>>
>> FunctionAttrs should still kick in for external linkage, can you post
>> an IR sample of where that's not happening (since that's a bug
IIUC)?
>>
>> As far as available_externally is concerned, preventing FunctionAttrs
>> on them is by design and I'm not sure if there is anything in
ThinLTO
>> to fix that (assuming that the functions are, in fact,
>> available_externally).
>>
>
> I think ThinLTO can fix most of it. My idea was to propagate function
> attribute even for simple function declarations!
>
> The reason I believe LTO can circumvent the "derefinement" issue
is
> because the attributes computed after optimization during the compile phase
> are propagated only after the linker decided which "version" of
an ODR
> function we will pick in the final binary.
>
> I believe this is valid, as long as later optimizations in the ThinLTO
> backend can't "undo" an attribute (or can't change the
attribute list to be
> less restrictive). For example if a function is `read_none` after the
> compile phase. During the ThinLTO thin-link we can propagate this attribute
> to every uses (after deciding that this is the copy we want to keep for
> this function). However if a later optimization invalidate the read_none
> attributes, it's over. I doubt this can happen though (as in general it
> would invalidate previous optimizations).
>
> --
> Mehdi
>
>
>
>
>>
>> There is some background information on this issue at
>> https://www.playingwithpointers.com/ipo-and-derefinement.html
>>
>> -- Sanjoy
>>
>>
>> >> it is unable to apply any of the functionattr optimizations
that LTO
>> can
>> >> (since LTO is able to convert everything to Internal). This is
a
>> problem
>> >> because every functionattr optimization expects an exact
definition by
>> >> calling hasExactDefinition
>> >> (http://llvm-cs.pcc.me.uk/include/llvm/IR/GlobalValue.h#387)
which
>> doesn't
>> >> want a derefined function (mayBeDerefined
>> >> http://llvm-cs.pcc.me.uk/include/llvm/IR/GlobalValue.h#119).
>> According to
>> >> the current definition, an AvailableExternally function could
be
>> derefined
>> >> so hasExactDefinition fails.
>> >>
>> >> Does anyone have any ideas on how to make a function with
>> >> External/AvailableExternally linkage type have an exact
definition?
>> >>
>> >
>> > + Sanjoy, who did all the derefinement work originally.
>> >
>> > --
>> > Davide
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170712/affb175e/attachment.html>