llvm dev - Sep 2020 - LLJIT vs. thread-local storage (again)

Hmm, I'm confused. The DSO is compiled with gcc. Do I need to compile
it with clang instead? I don't believe the JITted code uses the thread
support library directly, although I suppose it may be hidden with
templates and/or inline functions...

On Mon, Sep 28, 2020 at 10:43 PM Lang Hames <lhames at gmail.com>
wrote:
>
> Hi Geoff,
>
> If you want to access the variable directly from JIT'd code you will
need to compile the DSO with -femulated-tls. Alternatively you could introduce a
wrapper function to return the variable's address.
>
> This doesn't help you right now, but for what it is worth the road-map
to native TLS is becoming clearer:
>
> (1) Land the orcv1-removal branch.
> (2) Land the ORC remote-target-support branch -- This breaks ORC up into
three libraries: One for the JIT, one for the target process and one of shared
data structures.
> (3) Create an ORC runtime library. I'm not sure whether this should
live in LLVM or compiler-rt, but it would include implementations of, among
other things, native TLS variable setup.
> (4) Teach JITLink to handle native TLS relocations. This should be
relatively straightforward on MachO. It may be more involved for ELF, but I
haven't had a chance to dig in to this yet.
>
> Steps  (1) and (2) are mostly done. Step (3) shouldn't be too
difficult. Step (4) will be the interesting part.
>
> -- Lang.
>
> On Mon, Sep 28, 2020 at 7:35 AM Geoff Levner <glevner at gmail.com>
wrote:
>>
>> Hi JITters,
>>
>> I have some JIT-compiled C++ code that uses symbols from a DSO which
>> uses C++14's thread support library. When I compile it I get the
>> following error message:
>>
>> Symbols not found: [ __emutls_v._ZSt15__once_callable,
>> __emutls_v._ZSt11__once_call ]
>>
>> Those seem to correspond to std::__once_callable and std::__once_call,
>> which are indeed present in the C++ standard library. And I imagine
>> they are needed because the DSO makes calls to std::call_once(). But
>> ORC seems to be looking for them with the prefix
"__emutls_v".
>>
>> I had a similar problem in the past, which I thought Lang had fixed...
>> See this thread:
>>
>> http://lists.llvm.org/pipermail/llvm-dev/2020-January/138402.html
>>
>> Perhaps it wasn't as fixed as I thought? Or maybe there is
something I
>> need to do for JIT to fully support thread-local storage?
>>
>> Geoff

Hi Geoff,

Sorry -- I misread your question!

I have some JIT-compiled C++ code that uses symbols from a DSO which
uses
> C++14's thread support library. When I compile it I get the following
error
> message:
> Symbols not found: [ __emutls_v._ZSt15__once_callable,
> __emutls_v._ZSt11__once_call ]

That error message implies that your JIT'd code is directly referencing a
TLS variable, but the definition of the TLS variable has not been compiled
with -femulated-tls.

Did you expect your JIT'd code to reference call_once? Or was this perhaps
due to some unexpected inlining?

If JIT'd code doesn't access TLS variables at all then you don't
have to
worry about -femulated-tls. If JIT'd code uses TLS variables (either
defining or referencing them) then you need to turn -femulted-tls on for
the JIT'd code and any libraries that define TLS variables used by JIT'd
code. Unfortunately this can snowball: Once the library is complied with
emulated-tls, if it depends on TLS variables in a second library then that
also needs to be compiled with -femulated-tls, and so on.

I had a similar problem in the past, which I thought Lang had fixed...


That was a different emulated-tls problem: The JIT tracks symbol
definitions very carefully to ensure that they're not dropped (or error out
if they are) and the implicit renaming used by -femulated-tls was breaking
this. The fix taught the JIT to anticipate the renaming in order to fix the
tracking. This allows -femulated-tls to be used in JIT'd code. but you
still need to use it consistently across library boundaries.

Native TLS should fix this whole mess once we can support it.

-- Lang.


On Mon, Sep 28, 2020 at 2:02 PM Geoff Levner <glevner at gmail.com> wrote:
> Hmm, I'm confused. The DSO is compiled with gcc. Do I need to compile
> it with clang instead? I don't believe the JITted code uses the thread
> support library directly, although I suppose it may be hidden with
> templates and/or inline functions...
>
> On Mon, Sep 28, 2020 at 10:43 PM Lang Hames <lhames at gmail.com>
wrote:
> >
> > Hi Geoff,
> >
> > If you want to access the variable directly from JIT'd code you
will
> need to compile the DSO with -femulated-tls. Alternatively you could
> introduce a wrapper function to return the variable's address.
> >
> > This doesn't help you right now, but for what it is worth the
road-map
> to native TLS is becoming clearer:
> >
> > (1) Land the orcv1-removal branch.
> > (2) Land the ORC remote-target-support branch -- This breaks ORC up
into
> three libraries: One for the JIT, one for the target process and one of
> shared data structures.
> > (3) Create an ORC runtime library. I'm not sure whether this
should live
> in LLVM or compiler-rt, but it would include implementations of, among
> other things, native TLS variable setup.
> > (4) Teach JITLink to handle native TLS relocations. This should be
> relatively straightforward on MachO. It may be more involved for ELF, but I
> haven't had a chance to dig in to this yet.
> >
> > Steps  (1) and (2) are mostly done. Step (3) shouldn't be too
difficult.
> Step (4) will be the interesting part.
> >
> > -- Lang.
> >
> > On Mon, Sep 28, 2020 at 7:35 AM Geoff Levner <glevner at
gmail.com> wrote:
> >>
> >> Hi JITters,
> >>
> >> I have some JIT-compiled C++ code that uses symbols from a DSO
which
> >> uses C++14's thread support library. When I compile it I get
the
> >> following error message:
> >>
> >> Symbols not found: [ __emutls_v._ZSt15__once_callable,
> >> __emutls_v._ZSt11__once_call ]
> >>
> >> Those seem to correspond to std::__once_callable and
std::__once_call,
> >> which are indeed present in the C++ standard library. And I
imagine
> >> they are needed because the DSO makes calls to std::call_once().
But
> >> ORC seems to be looking for them with the prefix
"__emutls_v".
> >>
> >> I had a similar problem in the past, which I thought Lang had
fixed...
> >> See this thread:
> >>
> >> http://lists.llvm.org/pipermail/llvm-dev/2020-January/138402.html
> >>
> >> Perhaps it wasn't as fixed as I thought? Or maybe there is
something I
> >> need to do for JIT to fully support thread-local storage?
> >>
> >> Geoff
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200928/6eacf1b4/attachment.html>

Hi Lang,

Yes, it turns out the JITted code accidentally references thread-local
stuff via a function template defined by the DSO which calls
std::call_once(). And I don't think compiling external libraries with
clang and -femulated-tls is a reasonable solution. So I guess we will
have to just wait until LLJIT supports native TLS...

Thanks for all your help,
Geoff

On Mon, Sep 28, 2020 at 11:54 PM Lang Hames <lhames at gmail.com>
wrote:
>
> Hi Geoff,
>
> Sorry -- I misread your question!
>
>> I have some JIT-compiled C++ code that uses symbols from a DSO which
uses C++14's thread support library. When I compile it I get the following
error message:
>> Symbols not found: [ __emutls_v._ZSt15__once_callable,
>> __emutls_v._ZSt11__once_call ]
>
>
> That error message implies that your JIT'd code is directly referencing
a TLS variable, but the definition of the TLS variable has not been compiled
with -femulated-tls.
>
> Did you expect your JIT'd code to reference call_once? Or was this
perhaps due to some unexpected inlining?
>
> If JIT'd code doesn't access TLS variables at all then you
don't have to worry about -femulated-tls. If JIT'd code uses TLS
variables (either defining or referencing them) then you need to turn
-femulted-tls on for the JIT'd code and any libraries that define TLS
variables used by JIT'd code. Unfortunately this can snowball: Once the
library is complied with emulated-tls, if it depends on TLS variables in a
second library then that also needs to be compiled with -femulated-tls, and so
on.
>
>> I had a similar problem in the past, which I thought Lang had fixed...
>
>
> That was a different emulated-tls problem: The JIT tracks symbol
definitions very carefully to ensure that they're not dropped (or error out
if they are) and the implicit renaming used by -femulated-tls was breaking this.
The fix taught the JIT to anticipate the renaming in order to fix the tracking.
This allows -femulated-tls to be used in JIT'd code. but you still need to
use it consistently across library boundaries.
>
> Native TLS should fix this whole mess once we can support it.
>
> -- Lang.
>
>
> On Mon, Sep 28, 2020 at 2:02 PM Geoff Levner <glevner at gmail.com>
wrote:
>>
>> Hmm, I'm confused. The DSO is compiled with gcc. Do I need to
compile
>> it with clang instead? I don't believe the JITted code uses the
thread
>> support library directly, although I suppose it may be hidden with
>> templates and/or inline functions...
>>
>> On Mon, Sep 28, 2020 at 10:43 PM Lang Hames <lhames at gmail.com>
wrote:
>> >
>> > Hi Geoff,
>> >
>> > If you want to access the variable directly from JIT'd code
you will need to compile the DSO with -femulated-tls. Alternatively you could
introduce a wrapper function to return the variable's address.
>> >
>> > This doesn't help you right now, but for what it is worth the
road-map to native TLS is becoming clearer:
>> >
>> > (1) Land the orcv1-removal branch.
>> > (2) Land the ORC remote-target-support branch -- This breaks ORC
up into three libraries: One for the JIT, one for the target process and one of
shared data structures.
>> > (3) Create an ORC runtime library. I'm not sure whether this
should live in LLVM or compiler-rt, but it would include implementations of,
among other things, native TLS variable setup.
>> > (4) Teach JITLink to handle native TLS relocations. This should be
relatively straightforward on MachO. It may be more involved for ELF, but I
haven't had a chance to dig in to this yet.
>> >
>> > Steps  (1) and (2) are mostly done. Step (3) shouldn't be too
difficult. Step (4) will be the interesting part.
>> >
>> > -- Lang.
>> >
>> > On Mon, Sep 28, 2020 at 7:35 AM Geoff Levner <glevner at
gmail.com> wrote:
>> >>
>> >> Hi JITters,
>> >>
>> >> I have some JIT-compiled C++ code that uses symbols from a DSO
which
>> >> uses C++14's thread support library. When I compile it I
get the
>> >> following error message:
>> >>
>> >> Symbols not found: [ __emutls_v._ZSt15__once_callable,
>> >> __emutls_v._ZSt11__once_call ]
>> >>
>> >> Those seem to correspond to std::__once_callable and
std::__once_call,
>> >> which are indeed present in the C++ standard library. And I
imagine
>> >> they are needed because the DSO makes calls to
std::call_once(). But
>> >> ORC seems to be looking for them with the prefix
"__emutls_v".
>> >>
>> >> I had a similar problem in the past, which I thought Lang had
fixed...
>> >> See this thread:
>> >>
>> >>
http://lists.llvm.org/pipermail/llvm-dev/2020-January/138402.html
>> >>
>> >> Perhaps it wasn't as fixed as I thought? Or maybe there is
something I
>> >> need to do for JIT to fully support thread-local storage?
>> >>
>> >> Geoff