llvm dev - Feb 2018 - ThinLTO and linkonce_odr + unnamed_addr

> But it is interesting in general because according to the definition for
local_unnamed_addr, the symbol has to be linkonce_odr to be auto hide as
well. ThinLTO promotion can break that as well.

What do you mean that the promotion can break that?

The original description I found here: https://reviews.llvm.org/D20348 says
that it is possible to exclude a global from the symbol table if three
things are true:

   - This attribute is present on every instance of the global (which means
   that the normal rule that the global must have a unique address can be
   broken without being observable by the program by performing comparisons
   against the global's address)
   - The global has linkonce_odr linkage (which means that each linkage
   unit must have its own copy of the global if it requires one, and the copy
   in each linkage unit must be the same)
   - It is a constant or a function (which means that the program cannot
   observe that the unique-address rule has been broken by writing to the
   global)


When promoting from a linkonce_odr, it seems safe to me to *keep* the
local_unnamed_addr
on the weak_odr since we know the symbol was linkonce_odr in the first
place.

-- 
Mehdi

2018-02-07 12:12 GMT-08:00 Steven Wu <stevenwu at apple.com>:
> We didn't drop unnamed_addr. I just don't think weakodr_addr +
> unnamed_addr is safe to hide in all cases.
>
> I don't know if I interpreted local_unnamed_addr correctly but I think
it
> is mostly the same in thinLTO for ld64. The code generator only looks at
> the individual module and ld64 will be in charge of merging all the symbols
> with autohide. It doesn't really help in this case. But it is
interesting
> in general because according to the definition for local_unnamed_addr, the
> symbol has to be linkonce_odr to be auto hide as well. ThinLTO promotion
> can break that as well.
>
> Steven
>
>
>
> On Feb 7, 2018, at 11:44 AM, Mehdi AMINI <joker.eph at gmail.com>
wrote:
>
> Something I haven't seen mentioned: why are we dropping the
unnamed_addr?
> Can't we preserve it with the weak symbol? Would it be OK to add
auto-hide
> in this case (maybe it would already happen)?
> Can we use the new local_unnamed_addr that was added (by pcc or Rafael I
> don't remember)? I think this attribute matches exactly the `auto-hide`
> semantic. Wasn't the idea that this could be added any time by a
> module-level `infer_attribute` pass?
>
> --
> Mehdi
>
>
> 2018-02-07 11:36 GMT-08:00 Steven Wu via llvm-dev <llvm-dev at
lists.llvm.org
> >:
>
>> I didn't realize that that WeakDefCanBeHiddenDirective is only
available
>> on Darwin. So if we are doing it for #2, it should be a Darwin only fix
as
>> well.
>>
>> Steven
>>
>>
>> On Feb 7, 2018, at 11:29 AM, Reid Kleckner <rnk at google.com>
wrote:
>>
>> I agree with Teresa, we should probably do #2 to preserve behavior for
>> now.
>>
>> On Wed, Feb 7, 2018 at 9:34 AM, Teresa Johnson via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Hi Steven,
>>>
>>> I'd prefer not to inhibit importing. I am also concerned about
putting
>>> these symbols in the llvm.compiler_used (I don't recall earlier
discussion
>>> around this, but it seems like it could have effects on
optimization as you
>>> mention).
>>>
>>> What are the downsides of #2 (adding visibility hidden)? We already
do
>>> this when promoting internal linkage to external due to importing.
I'm not
>>> an expert on how this would affect link semantics.
>>>
>>> Thanks,
>>> Teresa
>>>
>>> On Tue, Feb 6, 2018 at 5:35 PM, Steven Wu <stevenwu at
apple.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I recently found that thinLTO doesn't deal with globals
that has
>>>> linkonce_odr and unnamed_addr (for macho at least) because it
prohibits the
>>>> autohide optimization during link time.
>>>>
>>>> In LLVM, we tagged a global linkonce_odr and unnamed_addr to
indicate
>>>> to the linker can hide them from symbol table if they were
picked (aka,
>>>> linkonce_odr_auto_hide linkage). It is very commonly used for
some type of
>>>> Tables for c++ code in clang for example.
>>>> However, thinLTO is promoting these symbols to weak_odr +
unnamed_addr,
>>>> which lose the property. As a result, it introduces unnecessary
weak
>>>> external symbols and weak external are not good for performance
on darwin
>>>> platforms.
>>>>
>>>> I have few proposed solutions for this issue but I don't
know which one
>>>> works the best for none macho platforms and other LTO clients
like lld.
>>>>
>>>> 1. Use llvm.compiler_used.
>>>> As far as I know, the linkage promote are just there to keep
the symbol
>>>> through internalize and codegen so adding them to compiler used
should
>>>> solve this issue. I was told that there was some objections to
do that in
>>>> the first place. Is it because the globals added to compiler
used is
>>>> ignored by the optimizer so they cannot be internalized and
they cannot be
>>>> optimized away? This works well for the case I am looking at
because c++
>>>> VTable can't really be optimized and for darwin platforms
because we can
>>>> rely on ld64 to do dead_stripping if needed.
>>>>
>>>> 2. Add visibility hidden when promote linkonce_odr +
unnamed_addr.
>>>> Well,this doesn't really preserve the link semantics, but
neither does
>>>> promoting linkonce_odr to weak_odr. The global will still end
up in the
>>>> symbol table but at least it isn't external so it
doesn't come with a
>>>> performance cost.
>>>>
>>>> 3. We can teach function importer that it cannot just reference
to
>>>> linkonce_odr + unnamed_addr symbols without importing them. I
have some
>>>> thoughts about how to do this so I can propose something if
people are
>>>> interested going down this route. I am expecting at least add
an entry in
>>>> the global summery and change the cost of importing symbols
that references
>>>> to linkonce_odr + unnamed_addr symbols.
>>>>
>>>> 4. As a temporary fix, just targeting at the VTables for c++.
We can
>>>> put a special case for global constants that uses this linkage
so they are
>>>> never promoted and their parents are never imported into other
modules. The
>>>> benefit for inlining global constants is very minimal and I
don't think we
>>>> are doing it currently.
>>>>
>>>> Let me know if any of those solutions work for other LTO
client.
>>>>
>>>> Thanks
>>>>
>>>> Steven
>>>>
>>>
>>>
>>>
>>> --
>>> Teresa Johnson |  Software Engineer |  tejohnson at google.com |
>>> 408-460-2413 <(408)%20460-2413>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180207/928d9820/attachment.html>

> On Feb 7, 2018, at 12:36 PM, Mehdi AMINI <joker.eph at gmail.com>
wrote:
> 
> > But it is interesting in general because according to the definition
for local_unnamed_addr, the symbol has to be linkonce_odr to be auto hide as
well. ThinLTO promotion can break that as well.
> 
> What do you mean that the promotion can break that?
> 
> The original description I found here: https://reviews.llvm.org/D20348
<https://reviews.llvm.org/D20348> says that it is possible to exclude a
global from the symbol table if three things are true:
> This attribute is present on every instance of the global (which means that
the normal rule that the global must have a unique address can be broken without
being observable by the program by performing comparisons against the
global's address)
> The global has linkonce_odr linkage (which means that each linkage unit
must have its own copy of the global if it requires one, and the copy in each
linkage unit must be the same)
> It is a constant or a function (which means that the program cannot observe
that the unique-address rule has been broken by writing to the global)
> 
> When promoting from a linkonce_odr, it seems safe to me to *keep* the
local_unnamed_addr on the weak_odr since we know the symbol was linkonce_odr in
the first place.
I don't think there is such guarantees. The description doesn't prevent
local_unnamed_addr on other linkage types. If your assumption is correct, then
the description can simply state "symbols with local_unnamed_addr can be
hidden from the symbol table, regardless of the linkage type). I guess I will
leave pcc to interpret this.

If my interpretation is correct, then a constant with linkonce_odr +
local_unnamed_addr can be hidden from symbol table before promotion. After
promotion, it no longer satisfies rule 2, so it has be in the symbol table.

Steven
> 
> -- 
> Mehdi
> 
> 2018-02-07 12:12 GMT-08:00 Steven Wu <stevenwu at apple.com
<mailto:stevenwu at apple.com>>:
> We didn't drop unnamed_addr. I just don't think weakodr_addr +
unnamed_addr is safe to hide in all cases.
> 
> I don't know if I interpreted local_unnamed_addr correctly but I think
it is mostly the same in thinLTO for ld64. The code generator only looks at the
individual module and ld64 will be in charge of merging all the symbols with
autohide. It doesn't really help in this case. But it is interesting in
general because according to the definition for local_unnamed_addr, the symbol
has to be linkonce_odr to be auto hide as well. ThinLTO promotion can break that
as well.
> 
> Steven
> 
> 
> 
>> On Feb 7, 2018, at 11:44 AM, Mehdi AMINI <joker.eph at gmail.com
<mailto:joker.eph at gmail.com>> wrote:
>> 
>> Something I haven't seen mentioned: why are we dropping the
unnamed_addr? Can't we preserve it with the weak symbol? Would it be OK to
add auto-hide in this case (maybe it would already happen)?
>> Can we use the new local_unnamed_addr that was added (by pcc or Rafael
I don't remember)? I think this attribute matches exactly the `auto-hide`
semantic. Wasn't the idea that this could be added any time by a
module-level `infer_attribute` pass?
>> 
>> -- 
>> Mehdi
>> 
>> 
>> 2018-02-07 11:36 GMT-08:00 Steven Wu via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>:
>> I didn't realize that that WeakDefCanBeHiddenDirective is only
available on Darwin. So if we are doing it for #2, it should be a Darwin only
fix as well.
>> 
>> Steven
>> 
>> 
>>> On Feb 7, 2018, at 11:29 AM, Reid Kleckner <rnk at google.com
<mailto:rnk at google.com>> wrote:
>>> 
>>> I agree with Teresa, we should probably do #2 to preserve behavior
for now.
>>> 
>>> On Wed, Feb 7, 2018 at 9:34 AM, Teresa Johnson via llvm-dev
<llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
wrote:
>>> Hi Steven,
>>> 
>>> I'd prefer not to inhibit importing. I am also concerned about
putting these symbols in the llvm.compiler_used (I don't recall earlier
discussion around this, but it seems like it could have effects on optimization
as you mention).
>>> 
>>> What are the downsides of #2 (adding visibility hidden)? We already
do this when promoting internal linkage to external due to importing. I'm
not an expert on how this would affect link semantics.
>>> 
>>> Thanks,
>>> Teresa
>>> 
>>> On Tue, Feb 6, 2018 at 5:35 PM, Steven Wu <stevenwu at apple.com
<mailto:stevenwu at apple.com>> wrote:
>>> Hi,
>>> 
>>> I recently found that thinLTO doesn't deal with globals that
has linkonce_odr and unnamed_addr (for macho at least) because it prohibits the
autohide optimization during link time.
>>> 
>>> In LLVM, we tagged a global linkonce_odr and unnamed_addr to
indicate to the linker can hide them from symbol table if they were picked (aka,
linkonce_odr_auto_hide linkage). It is very commonly used for some type of
Tables for c++ code in clang for example.
>>> However, thinLTO is promoting these symbols to weak_odr +
unnamed_addr, which lose the property. As a result, it introduces unnecessary
weak external symbols and weak external are not good for performance on darwin
platforms.
>>> 
>>> I have few proposed solutions for this issue but I don't know
which one works the best for none macho platforms and other LTO clients like
lld.
>>> 
>>> 1. Use llvm.compiler_used.
>>> As far as I know, the linkage promote are just there to keep the
symbol through internalize and codegen so adding them to compiler used should
solve this issue. I was told that there was some objections to do that in the
first place. Is it because the globals added to compiler used is ignored by the
optimizer so they cannot be internalized and they cannot be optimized away? This
works well for the case I am looking at because c++ VTable can't really be
optimized and for darwin platforms because we can rely on ld64 to do
dead_stripping if needed.
>>> 
>>> 2. Add visibility hidden when promote linkonce_odr + unnamed_addr.
>>> Well,this doesn't really preserve the link semantics, but
neither does promoting linkonce_odr to weak_odr. The global will still end up in
the symbol table but at least it isn't external so it doesn't come with
a performance cost.
>>> 
>>> 3. We can teach function importer that it cannot just reference to
linkonce_odr + unnamed_addr symbols without importing them. I have some thoughts
about how to do this so I can propose something if people are interested going
down this route. I am expecting at least add an entry in the global summery and
change the cost of importing symbols that references to linkonce_odr +
unnamed_addr symbols.
>>> 
>>> 4. As a temporary fix, just targeting at the VTables for c++. We
can put a special case for global constants that uses this linkage so they are
never promoted and their parents are never imported into other modules. The
benefit for inlining global constants is very minimal and I don't think we
are doing it currently.
>>> 
>>> Let me know if any of those solutions work for other LTO client.
>>> 
>>> Thanks
>>> 
>>> Steven
>>> 
>>> 
>>> 
>>> -- 
>>> Teresa Johnson |	 Software Engineer |	 tejohnson at google.com
<mailto:tejohnson at google.com> |	 408-460-2413
<tel:(408)%20460-2413>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>>> 
>>> 
>> 
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>> 
>> 
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180207/f2b2acee/attachment.html>

2018-02-07 12:45 GMT-08:00 Steven Wu <stevenwu at apple.com>:
>
>
> On Feb 7, 2018, at 12:36 PM, Mehdi AMINI <joker.eph at gmail.com>
wrote:
>
> > But it is interesting in general because according to the definition
> for local_unnamed_addr, the symbol has to be linkonce_odr to be auto hide
> as well. ThinLTO promotion can break that as well.
>
> What do you mean that the promotion can break that?
>
> The original description I found here: https://reviews.llvm.org/D20348
> says that it is possible to exclude a global from the symbol table if
> three things are true:
>
>    - This attribute is present on every instance of the global (which
>    means that the normal rule that the global must have a unique address
can
>    be broken without being observable by the program by performing
comparisons
>    against the global's address)
>    - The global has linkonce_odr linkage (which means that each linkage
>    unit must have its own copy of the global if it requires one, and the
copy
>    in each linkage unit must be the same)
>    - It is a constant or a function (which means that the program cannot
>    observe that the unique-address rule has been broken by writing to the
>    global)
>
>
> When promoting from a linkonce_odr, it seems safe to me to *keep* the
local_unnamed_addr
> on the weak_odr since we know the symbol was linkonce_odr in the first
> place.
>
>
> I don't think there is such guarantees. The description doesn't
prevent
> local_unnamed_addr on other linkage types. If your assumption is correct,
> then the description can simply state "symbols with local_unnamed_addr
can
> be hidden from the symbol table, regardless of the linkage type). I guess I
> will leave pcc to interpret this.
>
> If my interpretation is correct, then a constant with linkonce_odr +
> local_unnamed_addr can be hidden from symbol table before promotion. After
> promotion, it no longer satisfies rule 2, so it has be in the symbol table.
>
I don't see what would justify this, but I can miss some subtleties here.
If we can't do this with promotion, then it would be very unfortunate: the
whole point of these was to allow to "auto-hide".

-- 
Mehdi


>
> --
> Mehdi
>
> 2018-02-07 12:12 GMT-08:00 Steven Wu <stevenwu at apple.com>:
>
>> We didn't drop unnamed_addr. I just don't think weakodr_addr +
>> unnamed_addr is safe to hide in all cases.
>>
>> I don't know if I interpreted local_unnamed_addr correctly but I
think it
>> is mostly the same in thinLTO for ld64. The code generator only looks
at
>> the individual module and ld64 will be in charge of merging all the
symbols
>> with autohide. It doesn't really help in this case. But it is
interesting
>> in general because according to the definition for local_unnamed_addr,
the
>> symbol has to be linkonce_odr to be auto hide as well. ThinLTO
promotion
>> can break that as well.
>>
>> Steven
>>
>>
>>
>> On Feb 7, 2018, at 11:44 AM, Mehdi AMINI <joker.eph at gmail.com>
wrote:
>>
>> Something I haven't seen mentioned: why are we dropping the
unnamed_addr?
>> Can't we preserve it with the weak symbol? Would it be OK to add
auto-hide
>> in this case (maybe it would already happen)?
>> Can we use the new local_unnamed_addr that was added (by pcc or Rafael
I
>> don't remember)? I think this attribute matches exactly the
`auto-hide`
>> semantic. Wasn't the idea that this could be added any time by a
>> module-level `infer_attribute` pass?
>>
>> --
>> Mehdi
>>
>>
>> 2018-02-07 11:36 GMT-08:00 Steven Wu via llvm-dev <
>> llvm-dev at lists.llvm.org>:
>>
>>> I didn't realize that that WeakDefCanBeHiddenDirective is only
available
>>> on Darwin. So if we are doing it for #2, it should be a Darwin only
fix as
>>> well.
>>>
>>> Steven
>>>
>>>
>>> On Feb 7, 2018, at 11:29 AM, Reid Kleckner <rnk at
google.com> wrote:
>>>
>>> I agree with Teresa, we should probably do #2 to preserve behavior
for
>>> now.
>>>
>>> On Wed, Feb 7, 2018 at 9:34 AM, Teresa Johnson via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> Hi Steven,
>>>>
>>>> I'd prefer not to inhibit importing. I am also concerned
about putting
>>>> these symbols in the llvm.compiler_used (I don't recall
earlier discussion
>>>> around this, but it seems like it could have effects on
optimization as you
>>>> mention).
>>>>
>>>> What are the downsides of #2 (adding visibility hidden)? We
already do
>>>> this when promoting internal linkage to external due to
importing. I'm not
>>>> an expert on how this would affect link semantics.
>>>>
>>>> Thanks,
>>>> Teresa
>>>>
>>>> On Tue, Feb 6, 2018 at 5:35 PM, Steven Wu <stevenwu at
apple.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I recently found that thinLTO doesn't deal with globals
that has
>>>>> linkonce_odr and unnamed_addr (for macho at least) because
it prohibits the
>>>>> autohide optimization during link time.
>>>>>
>>>>> In LLVM, we tagged a global linkonce_odr and unnamed_addr
to indicate
>>>>> to the linker can hide them from symbol table if they were
picked (aka,
>>>>> linkonce_odr_auto_hide linkage). It is very commonly used
for some type of
>>>>> Tables for c++ code in clang for example.
>>>>> However, thinLTO is promoting these symbols to weak_odr +
>>>>> unnamed_addr, which lose the property. As a result, it
introduces
>>>>> unnecessary weak external symbols and weak external are not
good for
>>>>> performance on darwin platforms.
>>>>>
>>>>> I have few proposed solutions for this issue but I
don't know which
>>>>> one works the best for none macho platforms and other LTO
clients like lld.
>>>>>
>>>>> 1. Use llvm.compiler_used.
>>>>> As far as I know, the linkage promote are just there to
keep the
>>>>> symbol through internalize and codegen so adding them to
compiler used
>>>>> should solve this issue. I was told that there was some
objections to do
>>>>> that in the first place. Is it because the globals added to
compiler used
>>>>> is ignored by the optimizer so they cannot be internalized
and they cannot
>>>>> be optimized away? This works well for the case I am
looking at because c++
>>>>> VTable can't really be optimized and for darwin
platforms because we can
>>>>> rely on ld64 to do dead_stripping if needed.
>>>>>
>>>>> 2. Add visibility hidden when promote linkonce_odr +
unnamed_addr.
>>>>> Well,this doesn't really preserve the link semantics,
but neither does
>>>>> promoting linkonce_odr to weak_odr. The global will still
end up in the
>>>>> symbol table but at least it isn't external so it
doesn't come with a
>>>>> performance cost.
>>>>>
>>>>> 3. We can teach function importer that it cannot just
reference to
>>>>> linkonce_odr + unnamed_addr symbols without importing them.
I have some
>>>>> thoughts about how to do this so I can propose something if
people are
>>>>> interested going down this route. I am expecting at least
add an entry in
>>>>> the global summery and change the cost of importing symbols
that references
>>>>> to linkonce_odr + unnamed_addr symbols.
>>>>>
>>>>> 4. As a temporary fix, just targeting at the VTables for
c++. We can
>>>>> put a special case for global constants that uses this
linkage so they are
>>>>> never promoted and their parents are never imported into
other modules. The
>>>>> benefit for inlining global constants is very minimal and I
don't think we
>>>>> are doing it currently.
>>>>>
>>>>> Let me know if any of those solutions work for other LTO
client.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Steven
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Teresa Johnson |  Software Engineer |  tejohnson at google.com
|
>>>> 408-460-2413 <(408)%20460-2413>
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180207/721bcc95/attachment.html>