llvm dev - Apr 2020 - [ORC JIT] Getting symbols from object files before any lookup is done

Hello LLVM-Mailing-List and hello Lang (I figured somebody would add you....so I
did it directly. Sorry),

I'm still playing around with the ORC JIT version which is available in LLVM
10. My goal is to get the address of every defined symbol in a module and an
object file before the code is emitted. With the help of Lang I do this now with
the lookup function of the Execution Session. Thank you for this!

However, my next issue is: How can I find every defined symbol in my object
files? Currently I attached a callback function with the
"setNotifyLoaded" function of "RTDyldObjectLinkingLayer".
Which is really handy, because I can simply iterate over the symbols of the
object files.
However this only works when I already know what is inside my object files,
which kinda kills the purpose.

For a workaround I iterate through all the global symbols and functions of the
IR module I'm loading, skipping symbols that are not defined in the module.
However, I noticed when I do a lookup on those symbols, I will get some
"additional" symbols from the notify loaded callback function having
addresses I will miss then. Also this does not help me when I'm loading an
object file from disk cause I don't have such a function there.

Okay... explaining stuff is not my best practise so.... Here a tl;dr:
How can I get all symbols from an object file when it is loaded/added to ORC
JIT?

Sorry that I still have questions >o<

Kind greetings
Björn
Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr.
DE 114 165 789 Geschäftsführer: Dr. Hiroshi Nakamura, Dr. Robert Plank, Markus
Bode, Heiko Lampert, Takashi Nagano, Junichi Tajika, Ergin Cansiz.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200422/1cf1c05b/attachment.html>

Hi Bjoern,

The general answer is to scan the interface of whatever program
representation you are adding to the JIT. In the object file case you would
create an llvm::object::ObjectFile and then iterate over its symbols. As
you noticed you have to be careful with linkage (e.g. private and
available-externally definitions aren't really part of the interface) and
you have to be aware that the compilation of higher level program
representations can cause new definitions to be created (e.g. for constant
pool entries).

What are you planning to do with the symbols once you have them? If you are
going to issue a lookup you will have to be careful with the lookup and
visibility flags. A lookup using
JITDylibLookupFlags::MatchExportedSymbolsOnly will fail if you look up a
hidden (i.e. non-exported) symbol. You can either not look up hidden
symbols, or you can use JITDylibLookupFlags::MatchAllSymbols.

Regards,
Lang.

On Wed, Apr 22, 2020 at 4:51 AM Gaier, Bjoern <Bjoern.Gaier at horiba.com>
wrote:
> Hello LLVM-Mailing-List and hello Lang (I figured somebody would add
> you….so I did it directly. Sorry),
>
>
>
> I'm still playing around with the ORC JIT version which is available in
> LLVM 10. My goal is to get the address of every defined symbol in a module
> and an object file before the code is emitted. With the help of Lang I do
> this now with the lookup function of the Execution Session. Thank you for
> this!
>
>
>
> However, my next issue is: How can I find every defined symbol in my
> object files? Currently I attached a callback function with the
> "setNotifyLoaded" function of
"RTDyldObjectLinkingLayer". Which is really
> handy, because I can simply iterate over the symbols of the object files.
>
> However this only works when I already know what is inside my object
> files, which kinda kills the purpose.
>
>
>
> For a workaround I iterate through all the global symbols and functions of
> the IR module I'm loading, skipping symbols that are not defined in the
> module. However, I noticed when I do a lookup on those symbols, I will get
> some "additional" symbols from the notify loaded callback
function having
> addresses I will miss then. Also this does not help me when I'm loading
an
> object file from disk cause I don't have such a function there.
>
>
>
> Okay... explaining stuff is not my best practise so.... Here a tl;dr:
>
> How can I get all symbols from an object file when it is loaded/added to
> ORC JIT?
>
>
>
> Sorry that I still have questions >o<
>
>
>
> Kind greetings
>
> Björn
> Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816,
> USt.ID-Nr. DE 114 165 789 Geschäftsführer: Dr. Hiroshi Nakamura, Dr. Robert
> Plank, Markus Bode, Heiko Lampert, Takashi Nagano, Junichi Tajika, Ergin
> Cansiz.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200429/50ab4017/attachment.html>

Hey Lang,

Yes my intention was to use the “lookup” function of the executionsession to get
the address of every symbol in the object file or module. In general I would
prefer to even get the address of the hidden symbols. Just for learning though.
However I have a big issue, if I search in the LLVM code base on my machine for
“JITDylibLookupFlags” or “MatchAllSymbols” I will not find a thing.

Seems like I’m not running the latest LLVM version – I will download the LLVM10
for those changes now - or are they in some branch I’m not aware of?

Thank you for the help :3

Kind greetings
Björn

From: Lang Hames <lhames at gmail.com>
Sent: 29 April 2020 23:10
To: Gaier, Bjoern <Bjoern.Gaier at horiba.com>
Cc: LLVM Developers Mailing List <llvm-dev at lists.llvm.org>
Subject: Re: [ORC JIT] Getting symbols from object files before any lookup is
done

Hi Bjoern,

The general answer is to scan the interface of whatever program representation
you are adding to the JIT. In the object file case you would create an
llvm::object::ObjectFile and then iterate over its symbols. As you noticed you
have to be careful with linkage (e.g. private and available-externally
definitions aren't really part of the interface) and you have to be aware
that the compilation of higher level program representations can cause new
definitions to be created (e.g. for constant pool entries).

What are you planning to do with the symbols once you have them? If you are
going to issue a lookup you will have to be careful with the lookup and
visibility flags. A lookup using JITDylibLookupFlags::MatchExportedSymbolsOnly
will fail if you look up a hidden (i.e. non-exported) symbol. You can either not
look up hidden symbols, or you can use JITDylibLookupFlags::MatchAllSymbols.

Regards,
Lang.

On Wed, Apr 22, 2020 at 4:51 AM Gaier, Bjoern <Bjoern.Gaier at
horiba.com<mailto:Bjoern.Gaier at horiba.com>> wrote:
Hello LLVM-Mailing-List and hello Lang (I figured somebody would add you….so I
did it directly. Sorry),

I'm still playing around with the ORC JIT version which is available in LLVM
10. My goal is to get the address of every defined symbol in a module and an
object file before the code is emitted. With the help of Lang I do this now with
the lookup function of the Execution Session. Thank you for this!

However, my next issue is: How can I find every defined symbol in my object
files? Currently I attached a callback function with the
"setNotifyLoaded" function of "RTDyldObjectLinkingLayer".
Which is really handy, because I can simply iterate over the symbols of the
object files.
However this only works when I already know what is inside my object files,
which kinda kills the purpose.

For a workaround I iterate through all the global symbols and functions of the
IR module I'm loading, skipping symbols that are not defined in the module.
However, I noticed when I do a lookup on those symbols, I will get some
"additional" symbols from the notify loaded callback function having
addresses I will miss then. Also this does not help me when I'm loading an
object file from disk cause I don't have such a function there.

Okay... explaining stuff is not my best practise so.... Here a tl;dr:
How can I get all symbols from an object file when it is loaded/added to ORC
JIT?

Sorry that I still have questions >o<

Kind greetings
Björn
Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr.
DE 114 165 789 Geschäftsführer: Dr. Hiroshi Nakamura, Dr. Robert Plank, Markus
Bode, Heiko Lampert, Takashi Nagano, Junichi Tajika, Ergin Cansiz.
Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr.
DE 114 165 789 Geschäftsführer: Dr. Hiroshi Nakamura, Dr. Robert Plank, Markus
Bode, Heiko Lampert, Takashi Nagano, Junichi Tajika, Ergin Cansiz.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200430/a7765586/attachment-0001.html>