Lang Hames via llvm-dev
2020-Oct-02 16:33 UTC
[llvm-dev] LLVM Developers Meeting JIT BoF -- Request for Topics of Interest
Hi Geoff, I have only encountered one problem. If a static library has not been> compiled with -fPIC and uses symbols from a shared library, LLJIT does not > complain, but the code may crash without warning when it is executed.Was the static library compiled with large code model too? I think this is probably a RuntimeDyld bug: It's not great at error reporting. A few people in the community are working on an ELF/x86-64 implementation of JITLink at the moment -- Once we have that you should get small code model support and better error handling. -- Lang. On Fri, Oct 2, 2020 at 4:03 AM Geoff Levner <glevner at gmail.com> wrote:> StaticLibraryDefinitionGenerator works as promised on ELF archives! > > I have only encountered one problem. If a static library has not been > compiled with -fPIC and uses symbols from a shared library, LLJIT does not > complain, but the code may crash without warning when it is executed. > > Geoff > > On Mon, Sep 28, 2020 at 11:58 PM Lang Hames <lhames at gmail.com> wrote: > >> Hi Geoff, >> >> We use LLJIT. Do addObjectFile() and StaticLibraryDefinitionGenerator >>> work for ELF objects? >> >> >> They do. :) >> >> I've not tested StaticLibraryDefinitionGenerator extensively on Linux. >> but we have a regression test checking basic usage. If you run into any >> trouble at all please file a bug and assign it to me. >> >> -- Lang. >> >> >> >> On Mon, Sep 28, 2020 at 2:05 PM Geoff Levner <glevner at gmail.com> wrote: >> >>> We use LLJIT. Do addObjectFile() and StaticLibraryDefinitionGenerator >>> work for ELF objects? >>> >>> On Mon, Sep 28, 2020 at 10:57 PM Lang Hames <lhames at gmail.com> wrote: >>> > >>> > Hi Geoff, >>> > >>> >> Importing symbols into the JIT from an object file or static >>> library...? >>> > >>> > >>> > Sure! Are you interested in doing this with the C API, LLJIT, or raw >>> OrcV2 components? >>> > >>> > The high-level answer here (which we can dig into further in the BoF) >>> is: >>> > >>> > For object files: >>> > - For raw OrcV2 components you'll want to create an >>> RTDyldObjectLinkingLayer or ObjectLinkingLayer and use the 'add' method. >>> > - For an LLJIT instance you can just call the 'addObjectFile' method. >>> > - For the OrcV2 C API you can call the 'LLVMOrcLLJITAddObjectFile' >>> function. >>> > >>> > For static libraries: >>> > - For raw OrcV2 components or an LLJIT instance you can attach a >>> StaticLibraryDefinitionGenerator [1] to the JITDylib that you would like to >>> load the library into. See [2] for an example. >>> > - For the OrcV2 C API we will just need to add a method to wrap the >>> operation above. We already have >>> 'LLVMOrcCreateDynamicLibrarySearchGeneratorForProcess'. We should add >>> counterparts to that to load dynamic and static libraries from paths. >>> > >>> > -- Lang. >>> > >>> > [1] >>> https://github.com/llvm/llvm-project/blob/5aa56b242951ab0f0181386ea58509f19b09206e/llvm/include/llvm/ExecutionEngine/Orc/ExecutionUtils.h#L361 >>> > [2] >>> https://github.com/llvm/llvm-project/blob/e7549dafcd33ced4280a81ca1d1ee4cc78ed253f/llvm/tools/lli/lli.cpp#L968 >>> > >>> > >>> > On Mon, Sep 28, 2020 at 7:18 AM Geoff Levner <glevner at gmail.com> >>> wrote: >>> >> >>> >> Importing symbols into the JIT from an object file or static >>> library...? >>> >> >>> >> On Mon, Sep 28, 2020 at 5:29 AM Lang Hames <lhames at gmail.com> wrote: >>> >>> >>> >>> Hi Andres, >>> >>> >>> >>>> A topic that I'd find interesting is what it'd take to make it >>> easier to >>> >>>> use profile guided optimization in the context of JIT. A few years >>> back >>> >>>> I crudely hacked this together, but it was fairly ugly. It'd be >>> nice to >>> >>>> make that easier. Possibly too complicated for that type of >>> session? >>> >>> >>> >>> >>> >>> We couldn't go into detail, but we could kick off a discussion and >>> identify who is interested in working on this. My pitch: We should solve >>> this by investing in JITLink and implementing profiling (and debugger >>> support) via ObjectLinkingLayer::Plugins. Maybe a brief overview of / >>> discussion on how these work would be useful? >>> >>> >>> >>> -- Lang. >>> >>> >>> >>> >>> >>> On Fri, Sep 25, 2020 at 7:19 PM Andres Freund <andres at anarazel.de> >>> wrote: >>> >>>> >>> >>>> Hi, >>> >>>> >>> >>>> On 2020-09-25 19:05:42 -0700, Lang Hames wrote: >>> >>>> > The 2020 Virtual LLVM Developer's Meeting is coming up the week >>> after next. >>> >>>> > I'll be hosting a JIT Birds-of-a-Feather session on Thursday the >>> 8th at >>> >>>> > 10:55am PDT (See http://llvm.org/devmtg/2020-09/schedule/). I'm >>> planning to >>> >>>> > run this more like a Round Table: Minimal introduction, plenty of >>> time for >>> >>>> > discussion. >>> >>>> > >>> >>>> > Does anyone have any LLVM JIT related topics that they would >>> particularly >>> >>>> > like to discuss at this session? Any questions that they think >>> could be >>> >>>> > best answered by a round-table style discussion? I'll aim to rope >>> in the >>> >>>> > right people and prepare answers to make sure we get the most out >>> of the >>> >>>> > session. >>> >>>> >>> >>>> A topic that I'd find interesting is what it'd take to make it >>> easier to >>> >>>> use profile guided optimization in the context of JIT. A few years >>> back >>> >>>> I crudely hacked this together, but it was fairly ugly. It'd be >>> nice to >>> >>>> make that easier. Possibly too complicated for that type of >>> session? >>> >>>> >>> >>>> I dimly recall that the main issue was that there is no good way to >>> >>>> extract the profile information without going through files / >>> applying >>> >>>> the profile data without going through files. >>> >>>> >>> >>>> Greetings, >>> >>>> >>> >>>> Andres Freund >>> >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201002/6cf932ca/attachment.html>
Geoff Levner via llvm-dev
2020-Oct-02 17:02 UTC
[llvm-dev] LLVM Developers Meeting JIT BoF -- Request for Topics of Interest
On Fri, Oct 2, 2020 at 6:33 PM Lang Hames <lhames at gmail.com> wrote:> Hi Geoff, > > I have only encountered one problem. If a static library has not been >> compiled with -fPIC and uses symbols from a shared library, LLJIT does not >> complain, but the code may crash without warning when it is executed. > > > Was the static library compiled with large code model too? >I don't think so... All libraries are compiled with gcc 6.3.1, and the gcc documentation says that the default code model is small. I don't know what the default code model is for clang...> I think this is probably a RuntimeDyld bug: It's not great at error > reporting. A few people in the community are working on an ELF/x86-64 > implementation of JITLink at the moment -- Once we have that you should get > small code model support and better error handling. > > -- Lang. >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201002/247f722a/attachment.html>
Lang Hames via llvm-dev
2020-Oct-02 17:12 UTC
[llvm-dev] LLVM Developers Meeting JIT BoF -- Request for Topics of Interest
Hi Geoff, I believe clang also defaults to small for ELF / x86-64 (I'd expect it to be the same as gcc). RuntimeDyld currently assumes large code model. Testing and support for the small code model are spotty at best -- it's essentially unsupported. JITLink *will* support small code model (already does for MachO). The JITLink ELF/x86-64 implementation isn't quite ready for testing yet, but I'll make an announcement as soon as it is. -- Lang. On Fri, Oct 2, 2020 at 10:02 AM Geoff Levner <glevner at gmail.com> wrote:> > On Fri, Oct 2, 2020 at 6:33 PM Lang Hames <lhames at gmail.com> wrote: > >> Hi Geoff, >> >> I have only encountered one problem. If a static library has not been >>> compiled with -fPIC and uses symbols from a shared library, LLJIT does not >>> complain, but the code may crash without warning when it is executed. >> >> >> Was the static library compiled with large code model too? >> > > I don't think so... All libraries are compiled with gcc 6.3.1, and the gcc > documentation says that the default code model is small. I don't know what > the default code model is for clang... > > >> I think this is probably a RuntimeDyld bug: It's not great at error >> reporting. A few people in the community are working on an ELF/x86-64 >> implementation of JITLink at the moment -- Once we have that you should get >> small code model support and better error handling. >> >> -- Lang. >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201002/bed5212b/attachment-0001.html>