llvm dev - Mar 2018 - LLVM gold plugin do not add llvm instrinsics symbols to the linker symbol table

Dear community,

Recently I discovered that llvm gold linker plugin (LLVMgold.so) doesn't add
llvm instrinsics symbols to the linker symbol table. I do not claim that
something is necessary wrong, just want to share my observations with the
community.

               Brief summary

If I create a static library with a custom version of 'exp()' math
function and link it as follows:
$ clang -O0 -ffast-math -flto a.o -L. -Wl,-Bstatic -lexp -Wl,-Bdynamic -lm
exp gets resolved from libm instead of my static library (libexp.a) even though
the static library appears first on the link command line.

               Reproducer

1) As a reproducer I built a tiny static library (with the name libexp.a) with
exp function defined:
$ cat exp.c
double exp (double x)
{
  return x;
}
$ clang -c exp.c
$ ar -r libexp.a exp.o
$ nm -S libexp.a | grep exp
0000000000000000 0000000000000010 T exp

2) I have another small piece of code which will use 'exp' from my
library (libexp.a):

$ cat a.c
#include <math.h>
int main()
{
  return exp(1.0);
}
$ clang -c -O0 -ffast-math -flto -o a.o a.c

After rC318193 we started lowering math libcalls to equivalent LLVM intrinsics:

$ llvm-dis a.o -o - | grep llvm.exp
  %2 = call fast double @llvm.exp.f64(double 1.000000e+00)

3) I link my library (libexp.a) statically but link libm dynamically (as a
fallback option):

$ clang -O0 -ffast-math -flto a.o -L. -Wl,-Bstatic -lexp -Wl,-Bdynamic -lm
$ nm -S a.out | grep exp
               U exp@@GLIBC_2.2.5

So, exp symbol was taken from libm, not from my static library. Notice, that
'llvm.exp.f64' should be lowered into __exp_finite (see
https://bugs.llvm.org/show_bug.cgi?id=35672#c9 ).
To understand why this happened I traced 'exp' symbol:

$ clang -O0 -ffast-math -flto a.o -L. -Wl,-Bstatic -lexp -Wl,-Bdynamic -lm
-Wl,--trace-symbol=exp
/usr/lib64/libm.so: definition of exp
/tmp/lto-llvm-8f9b9a.o: reference to exp

               How linker works in those cases?

In LTO build linker works in 2 phases:
1. Runs through all the inputs and identifies which of them should be processed
by the plugin. But while doing that linker already starts filling it's
symbol table.
   Plugin at that point knows what symbols are defined and what referenced by
the input files with bitcode inside (because it knows how to parse it). So,
plugin reports that symbols to the linker.
2. After all inputs are read linker signals to the plugin to do it's magic.
   We run LTO and produce one "fat" relocatable object file and feed
it back to the linker.
   Linker finishes it's job as usual.

More details of this process are described here:
https://gcc.gnu.org/wiki/whopr/driver .

If we apply the process described above to our case, we will understand what
happened:
1st pass:
  We see a.o. It is claimed by the plugin. But plugin ignored llvm.exp.f64
intrinsic, so 'exp' symbol was not reported to the linker as referenced
symbol.
  We see static linking of libexp. We are linking static libraries only if
someone referenced any symbols defined in it, so we skip libexp (no one
referenced 'exp').
  We see dynamic linking of libm. We will link to it unconditionally, so linker
captures all symbols in libm. It found definition of 'exp' and insert it
to the linker symbol table.
  We run LTO stage. /tmp/lto-llvm-8f9b9a.o was added to the linker inputs.
2nd pass:
  There is a reference to exp in /tmp/lto-llvm-8f9b9a.o, but we have it's
definition already (in libm).
  We see libexp again, but it's too late, we already have exp defined in
libm, and it won't be overridden.

I confirmed all the above by adding debug prints in LLVMGold plugin
(LLVMgold.so) and built debug version of binutils (added debug prints to gold
linker).
Behavior is similar for gold and bfd linkers.

This problem goes away if I add '-u exp' to the linker command. This
option adds undefined symbol 'exp' to the linker symbol right at the
beginning of the process, so it will be taken in 1st pass from libexp. But of
course, it can't be a solution to this issue.

               How it affects us?

We have our own runtime library which provides implementation for math functions
like exp, log, etc. So, we use it as a replacement for libm.
We link libm dynamically because not every distribution is supplied with static
version of libm.

It's not so trivial to report symbols that correspond to llvm intrinsics
because we don't know for sure to which exact symbols they will be expanded.
For example, in rL322087 we started lowering llvm.exp.* to
"__exp_finite" if -ffast-math (or -Ofast) is provided.

Any thoughts on this?

-Denis Bakhvalov
Employee of Intel, compiler development team.
--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial
Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 |
Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i
moze zawierac informacje poufne. W razie przypadkowego otrzymania tej
wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie;
jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). If you are not the intended recipient, please
contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180323/87ab3241/attachment.html>

A couple questions/notes so I can understand better:

Without -flto, a.o ends up with a reference to __exp_finite, which also
would not be satifisfied out of libexp.a. Do you also have an
implementation of __exp_finite in your libexp.a?

Can you build with -fno-builtin, or -fno-builtin-exp etc? That results in a
reference to __exp_finite in the .o bitcode (which of course has the same
issue I mentioned above, but is consistent). That seems to be the option
you should use here when you want to use your own implementations.

Teresa


On Fri, Mar 23, 2018 at 8:39 AM, Bakhvalov, Denis via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Dear community,
>
>
>
> Recently I discovered that llvm gold linker plugin (LLVMgold.so)
doesn't
> add llvm instrinsics symbols to the linker symbol table. I do not claim
> that something is necessary wrong, just want to share my observations with
> the community.
>
>
>
>                Brief summary
>
>
>
> If I create a static library with a custom version of ‘exp()’ math
> function and link it as follows:
>
> $ clang -O0 -ffast-math -flto a.o -L. -Wl,-Bstatic -lexp -Wl,-Bdynamic -lm
>
> exp gets resolved from libm instead of my static library (libexp.a) even
> though the static library appears first on the link command line.
>
>
>
>                Reproducer
>
>
>
> 1) As a reproducer I built a tiny static library (with the name libexp.a)
> with exp function defined:
>
> $ cat exp.c
>
> double exp (double x)
>
> {
>
>   return x;
>
> }
>
> $ clang –c exp.c
>
> $ ar -r libexp.a exp.o
>
> $ nm -S libexp.a | grep exp
>
> 0000000000000000 0000000000000010 T exp
>
>
>
> 2) I have another small piece of code which will use 'exp' from my
library
> (libexp.a):
>
>
>
> $ cat a.c
>
> #include <math.h>
>
> int main()
>
> {
>
>   return exp(1.0);
>
> }
>
> $ clang -c -O0 -ffast-math -flto -o a.o a.c
>
>
>
> After rC318193 we started lowering math libcalls to equivalent LLVM
> intrinsics:
>
>
>
> $ llvm-dis a.o -o - | grep llvm.exp
>
>   %2 = call fast double @llvm.exp.f64(double 1.000000e+00)
>
>
>
> 3) I link my library (libexp.a) statically but link libm dynamically (as a
> fallback option):
>
>
>
> $ clang -O0 -ffast-math -flto a.o -L. -Wl,-Bstatic -lexp -Wl,-Bdynamic –lm
>
> $ nm -S a.out | grep exp
>
>                U exp@@GLIBC_2.2.5
>
>
>
> So, exp symbol was taken from libm, not from my static library. Notice,
> that 'llvm.exp.f64' should be lowered into __exp_finite (see
> https://bugs.llvm.org/show_bug.cgi?id=35672#c9 ).
>
> To understand why this happened I traced ‘exp’ symbol:
>
>
>
> $ clang -O0 -ffast-math -flto a.o -L. -Wl,-Bstatic -lexp -Wl,-Bdynamic -lm
> -Wl,--trace-symbol=exp
>
> /usr/lib64/libm.so: definition of exp
>
> /tmp/lto-llvm-8f9b9a.o: reference to exp
>
>
>
>                How linker works in those cases?
>
>
>
> In LTO build linker works in 2 phases:
>
> 1. Runs through all the inputs and identifies which of them should be
> processed by the plugin. But while doing that linker already starts filling
> it’s symbol table.
>
>    Plugin at that point knows what symbols are defined and what referenced
> by the input files with bitcode inside (because it knows how to parse it).
> So, plugin reports that symbols to the linker.
>
> 2. After all inputs are read linker signals to the plugin to do it’s magic.
>
>    We run LTO and produce one “fat” relocatable object file and feed it
> back to the linker.
>
>    Linker finishes it’s job as usual.
>
>
>
> More details of this process are described here: https://gcc.gnu.org/wiki/
> whopr/driver .
>
>
>
> If we apply the process described above to our case, we will understand
> what happened:
>
> 1st pass:
>
>   We see a.o. It is claimed by the plugin. But plugin ignored llvm.exp.f64
> intrinsic, so ‘exp’ symbol was not reported to the linker as referenced
> symbol.
>
>   We see static linking of libexp. We are linking static libraries only if
> someone referenced any symbols defined in it, so we skip libexp (no one
> referenced ‘exp’).
>
>   We see dynamic linking of libm. We will link to it unconditionally, so
> linker captures all symbols in libm. It found definition of ‘exp’ and
> insert it to the linker symbol table.
>
>   We run LTO stage. /tmp/lto-llvm-8f9b9a.o was added to the linker inputs.
>
> 2nd pass:
>
>   There is a reference to exp in /tmp/lto-llvm-8f9b9a.o, but we have it’s
> definition already (in libm).
>
>   We see libexp again, but it’s too late, we already have exp defined in
> libm, and it won’t be overridden.
>
>
>
> I confirmed all the above by adding debug prints in LLVMGold plugin
> (LLVMgold.so) and built debug version of binutils (added debug prints to
> gold linker).
>
> Behavior is similar for gold and bfd linkers.
>
>
>
> This problem goes away if I add ‘-u exp’ to the linker command. This
> option adds undefined symbol ‘exp’ to the linker symbol right at the
> beginning of the process, so it will be taken in 1st pass from libexp. But
> of course, it can't be a solution to this issue.
>
>
>
>                How it affects us?
>
>
>
> We have our own runtime library which provides implementation for math
> functions like exp, log, etc. So, we use it as a replacement for libm.
>
> We link libm dynamically because not every distribution is supplied with
> static version of libm.
>
>
>
> It's not so trivial to report symbols that correspond to llvm
intrinsics
> because we don't know for sure to which exact symbols they will be
> expanded. For example, in rL322087 we started lowering llvm.exp.* to
> “__exp_finite” if –ffast-math (or -Ofast) is provided.
>
>
>
> Any thoughts on this?
>
>
>
> -Denis Bakhvalov
>
> Employee of Intel, compiler development team.
>
> ---------------------------------------------------------------------
>
> *Intel Technology Poland sp. z o.o.*ul. S&#322owackiego 173 | 80-298
> Gda&#324sk | S&#261d Rejonowy Gda&#324sk P&#243&#322noc
| VII Wydzia&#322
> Gospodarczy Krajowego Rejestru S&#261dowego - KRS 101882 | NIP
> 957-07-52-316 | Kapita&#322 zak&#322adowy 200.000 PLN.
>
> Ta wiadomo&#347&#263 wraz z za&#322&#261cznikami jest
przeznaczona dla
> okre&#347lonego adresata i mo&#380e zawiera&#263 informacje
poufne. W razie
> przypadkowego otrzymania tej wiadomo&#347ci, prosimy o powiadomienie
> nadawcy oraz trwa&#322e jej usuni&#281cie; jakiekolwiek
przegl&#261danie
> lub rozpowszechnianie jest zabronione.
> This e-mail and any attachments may contain confidential material for the
> sole use of the intended recipient(s). If you are not the intended
> recipient, please contact the sender and delete all copies; any review or
> distribution by others is strictly prohibited.
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>

-- 
Teresa Johnson |  Software Engineer |  tejohnson at google.com |  408-460-2413
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180323/cb5e8129/attachment-0001.html>

Hello Teresa,
> Without -flto, a.o ends up with a reference to __exp_finite,
That’s correct.
> which also would not be satifisfied out of libexp.a.
That’s not correct. Even if libexp.a would have __exp_finite, it wouldn’t be
resolved from libexp.a, because of the behavior described in my first message.
> Do you also have an implementation of __exp_finite in your libexp.a?
No, I don’t have __exp_finite in libexp.a. I presented complete code in my first
message. You can reproduce it yourself on current trunk.
> Can you build with -fno-builtin, or -fno-builtin-exp etc?
> That results in a reference to __exp_finite in the .o bitcode
> (which of course has the same issue I mentioned above, but is consistent).
> That seems to be the option you should use here when you want to use your
own implementations.
Yes, -fno-builtin results in call to __exp_finite in the bitcode. But the
problem I try to address is not what version of exp libcall we want to generate:
exp or __exp_finite.
The problem is that if the exp call was lowered to llvm intrinsic, linker
doesn’t resolve it from my static library in LTO build, because plugin
(LLVMgold.so) doesn’t report those symbols (yet in intrinsic form) to the
linker.

Best regards,
Denis Bakhvalov.
--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial
Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 |
Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i
moze zawierac informacje poufne. W razie przypadkowego otrzymania tej
wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie;
jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). If you are not the intended recipient, please
contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180323/cbe4d534/attachment.html>