llvm dev - Aug 2020 - JIT interaction with linkonce_odr global variables

Hello,

I recently hit an issue when JIT'ing my generated IR using
llvm::orc::LLJIT. My IR contains the following definition of a global
variable:
> $_ZZ23TestStaticVarInFunctionbE1x = comdat any
> @_ZZ23TestStaticVarInFunctionbE1x = linkonce_odr dso_local global i32 123,
> comdat, align 4
>
And in my host process, there exists the same symbol. I would expect LLJIT
to resolve the global variable above to the address inside the host
process, since it has linkage 'linkonce_odr'. However, it turns out that
LLJIT resolved it as if it were a conventional global variable definition
and gave it its own address (I have added 'GetForCurrentProcess'
generator
for symbol resolution of course).

I can workaround this issue by making the symbol a declaration (drop the
initializer, comdat and make the linkage external), but I'm wondering if it
is expected behavior that LLJIT does not respect linkonce_odr specifier,
since the documentation says LLJIT's symbol resolution should work as if I
were running a normal linker.

Best,
Haoran
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200807/a0c882c0/attachment.html>

Hi Haoran

Yes, that's expected behavior. Definition generators only get invoked
for symbols that are not defined in your modules. The documentation in
https://llvm.org/docs/ORCv2.html says:
> If a definition generator is attached to a JITDylib, then any
unsuccessful lookup on that JITDylib will fall back to calling the
definition generator

I guess the situation is hard to compare with the context a static
linker runs in. It's an interesting idea though. Maybe it's possible
achieve with a JITLink plugin?

Best
Stefan

On 07/08/2020 11:51, Haoran Xu via llvm-dev wrote:
> Hello,
>
> I recently hit an issue when JIT'ing my generated IR using
> llvm::orc::LLJIT. My IR contains the following definition of a global
> variable:
>
>     $_ZZ23TestStaticVarInFunctionbE1x = comdat any
>     @_ZZ23TestStaticVarInFunctionbE1x = linkonce_odr dso_local global
>     i32 123, comdat, align 4
>
>
> And in my host process, there exists the same symbol. I would expect
> LLJIT to resolve the global variable above to the address inside the
> host process, since it has linkage 'linkonce_odr'. However, it
turns
> out that LLJIT resolved it as if it were a conventional global
> variable definition and gave it its own address (I have added
> 'GetForCurrentProcess' generator for symbol resolution of course).
>
> I can workaround this issue by making the symbol a declaration (drop
> the initializer, comdat and make the linkage external), but I'm
> wondering if it is expected behavior that LLJIT does not respect
> linkonce_odr specifier, since the documentation says LLJIT's symbol
> resolution should work as if I were running a normal linker. 
>
> Best,
> Haoran
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-- 
https://flowcrypt.com/pub/stefan.graenitz at gmail.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200807/a0a7e4b1/attachment.html>

Hi Haoran, Stefan,

This case is poorly supported at the moment. I think your workaround
(changing it to a decl) is good for now. You could also use a JITLink
plugin to replace the duplicate definition with an absolute symbol at link
time, but it's probably better to drop the definition before adding and
save yourself some compile time.

Eventually I would like to improve weak symbol handling by adding a new
WeakSymbolManager abstraction. This would be attached to the
ExecutionSession and would be called for each MaterializationUnit before it
is added. It would enable proper resolution of weak symbols across JITDylib
boundaries (also currently unsupported) and could scan the process
symbols first. It's on my todo list, but I have to get some JITLink work
and the removable dylibs feature done first. If anyone wants to tackle this
as a project I'd be happy to help with pointers. :)

Regards,
Lang.



On Fri, Aug 7, 2020 at 9:14 AM Stefan Gränitz <stefan.graenitz at
gmail.com>
wrote:
> Hi Haoran
>
> Yes, that's expected behavior. Definition generators only get invoked
for
> symbols that are not defined in your modules. The documentation in
> https://llvm.org/docs/ORCv2.html says:
>
> > If a definition generator is attached to a JITDylib, then any
> unsuccessful lookup on that JITDylib will fall back to calling the
> definition generator
>
> I guess the situation is hard to compare with the context a static linker
> runs in. It's an interesting idea though. Maybe it's possible
achieve with
> a JITLink plugin?
>
> Best
> Stefan
>
> On 07/08/2020 11:51, Haoran Xu via llvm-dev wrote:
>
> Hello,
>
> I recently hit an issue when JIT'ing my generated IR using
> llvm::orc::LLJIT. My IR contains the following definition of a global
> variable:
>
>> $_ZZ23TestStaticVarInFunctionbE1x = comdat any
>> @_ZZ23TestStaticVarInFunctionbE1x = linkonce_odr dso_local global i32
>> 123, comdat, align 4
>>
>
> And in my host process, there exists the same symbol. I would expect LLJIT
> to resolve the global variable above to the address inside the host
> process, since it has linkage 'linkonce_odr'. However, it turns out
that
> LLJIT resolved it as if it were a conventional global variable definition
> and gave it its own address (I have added 'GetForCurrentProcess'
generator
> for symbol resolution of course).
>
> I can workaround this issue by making the symbol a declaration (drop the
> initializer, comdat and make the linkage external), but I'm wondering
if it
> is expected behavior that LLJIT does not respect linkonce_odr specifier,
> since the documentation says LLJIT's symbol resolution should work as
if I
> were running a normal linker.
>
> Best,
> Haoran
>
>
>
> _______________________________________________
> LLVM Developers mailing listllvm-dev at
lists.llvm.orghttps://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> -- https://flowcrypt.com/pub/stefan.graenitz at gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200809/de98caac/attachment.html>