So this is likely just an accident rather than on purpose. There's totally room for that to happen, but it'll be the job of the client and not MCJIT itself. Basically whomever should call dlopen if they want to and it's the problem of the client application (which could be lli as the canonical mcjit example) to link in the correct bits. Make sense? If you can see a use case in particular that would have this functionality being linked into mcjit we'd love to hear it, but I haven't been able to come up with one. -eric On Mon, Oct 1, 2012 at 10:06 AM, Jim Grosbach <grosbach at apple.com> wrote:> Hi James, > > In that scenario, it would be the responsibility of the client to implement a memory manager for the MCJIT that knows how to import those symbols from the relevant shared library (or resolve them directly to the statically compiled symbols). > > -Jim > > On Oct 1, 2012, at 3:09 AM, James Molloy <James.Molloy at arm.com> wrote: > >> Hi all, >> >> There are symbols in libgcc (and compiler-rt) that JIT-compiled modules >> may need. These are currently linked correctly because lli and friends >> are linked against libgcc_s.so (i.e. shared library version of libgcc, >> so dlopen/dlsym works). >> >> However if a consumer links statically, dlsym won't find all compiler >> required functions (only the ones that were required by the JIT-compiler >> itself will be linked). >> >> So the question is, do we want to support users of the MCJIT linking >> statically to libgcc/compiler-rt? Or explicitly not handle this >> use-case? >> >> Cheers, >> >> James >> >> >> >> _______________________________________________ >> LLVM Developers mailing list >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Hi Eric,
This makes total sense, thanks.
One thing though; it seems a lot of functionality is being foisted onto
the client. This is all well and good, but it seems that clients are
expected to reinvent the wheel quite a bit for no real purpose - an
example is the implementation of getPointerToNamedFunction,
allocate{Code,Data}Section and invalidateInstructionCache in lli.cpp.
Many clients will just want "a standard implementation that works but
can be customised" for many of these functions, and this is currently
not available. So they have to copy-paste code from lli.cpp and keep it
in sync when bugs gets found.
Would it not be nicer to hoist this behaviour that could be customised
but will suit most people's needs (when JITting on host on a unix OS)
into some superclass that could be subclassed in a client?
Cheers,
James
On Mon, 2012-10-01 at 21:16 +0100, Eric Christopher
wrote:> So this is likely just an accident rather than on purpose. There's
> totally room for that to happen, but it'll be the job of the client
> and not MCJIT itself.
>
> Basically whomever should call dlopen if they want to and it's the
> problem of the client application (which could be lli as the canonical
> mcjit example) to link in the correct bits.
>
> Make sense? If you can see a use case in particular that would have
> this functionality being linked into mcjit we'd love to hear it, but I
> haven't been able to come up with one.
>
> -eric
>
> On Mon, Oct 1, 2012 at 10:06 AM, Jim Grosbach <grosbach at apple.com>
wrote:
> > Hi James,
> >
> > In that scenario, it would be the responsibility of the client to
implement a memory manager for the MCJIT that knows how to import those symbols
from the relevant shared library (or resolve them directly to the statically
compiled symbols).
> >
> > -Jim
> >
> > On Oct 1, 2012, at 3:09 AM, James Molloy <James.Molloy at
arm.com> wrote:
> >
> >> Hi all,
> >>
> >> There are symbols in libgcc (and compiler-rt) that JIT-compiled
modules
> >> may need. These are currently linked correctly because lli and
friends
> >> are linked against libgcc_s.so (i.e. shared library version of
libgcc,
> >> so dlopen/dlsym works).
> >>
> >> However if a consumer links statically, dlsym won't find all
compiler
> >> required functions (only the ones that were required by the
JIT-compiler
> >> itself will be linked).
> >>
> >> So the question is, do we want to support users of the MCJIT
linking
> >> statically to libgcc/compiler-rt? Or explicitly not handle this
> >> use-case?
> >>
> >> Cheers,
> >>
> >> James
> >>
> >>
> >>
> >> _______________________________________________
> >> LLVM Developers mailing list
> >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
On Oct 2, 2012, at 12:39 AM, James Molloy <James.Molloy at arm.com> wrote:> Hi Eric, > > This makes total sense, thanks. > > One thing though; it seems a lot of functionality is being foisted onto > the client. This is all well and good, but it seems that clients are > expected to reinvent the wheel quite a bit for no real purpose - an > example is the implementation of getPointerToNamedFunction, > allocate{Code,Data}Section and invalidateInstructionCache in lli.cpp. > > Many clients will just want "a standard implementation that works but > can be customised" for many of these functions, and this is currently > not available. So they have to copy-paste code from lli.cpp and keep it > in sync when bugs gets found. > > Would it not be nicer to hoist this behaviour that could be customised > but will suit most people's needs (when JITting on host on a unix OS) > into some superclass that could be subclassed in a client? >That's the end-goal, yes. Right now the focus is on making the division of responsibility a very clear one and get the MCJIT functionality fleshed out. Doing lots of both at once leads to intermingling of functionality out of convenience. For example, consider the many patches proposed that are rejected due to violations of the client/target layering. By making all clients do some work now, it helps ensure everyone thinks about the needs of the broader architecture, not just what will make things work for their use-case (which, as you say, is typically a self-hosted Unix-ish environment). Put another way, the old JIT intermingled lots of host and client logic, and that was one of its major design flaws. We're fixing that, but it's going to take a while for everyone's thinking to adjust and internalize the new model. That's totally reasonable. The idea is to structure the code and enforce divisions of labor in the code via client vs. library that help reinforce the new, better layered, design. If you really want to get things moving on this sooner rather than later, a step in that direction would be to flesh out the lli support for the "-remote-mcjit" memory manager so that all of the ExecutionEngine tests pass with that enabled. As it stands, that's primarily good support for external symbol resolution, I believe, perhaps with a bit of additional relocation support. If we can turn that on for testing in all regression test runs, that will help identify at "make check" time any gratuitous layering violations, as they often don't survive the address translations involved. Then we can start building up a set of memory managers that support the common cases that can be plug-and-play used by MCJIT clients. -Jim> Cheers, > > James > > On Mon, 2012-10-01 at 21:16 +0100, Eric Christopher wrote: >> So this is likely just an accident rather than on purpose. There's >> totally room for that to happen, but it'll be the job of the client >> and not MCJIT itself. >> >> Basically whomever should call dlopen if they want to and it's the >> problem of the client application (which could be lli as the canonical >> mcjit example) to link in the correct bits. >> >> Make sense? If you can see a use case in particular that would have >> this functionality being linked into mcjit we'd love to hear it, but I >> haven't been able to come up with one. >> >> -eric >> >> On Mon, Oct 1, 2012 at 10:06 AM, Jim Grosbach <grosbach at apple.com> wrote: >>> Hi James, >>> >>> In that scenario, it would be the responsibility of the client to implement a memory manager for the MCJIT that knows how to import those symbols from the relevant shared library (or resolve them directly to the statically compiled symbols). >>> >>> -Jim >>> >>> On Oct 1, 2012, at 3:09 AM, James Molloy <James.Molloy at arm.com> wrote: >>> >>>> Hi all, >>>> >>>> There are symbols in libgcc (and compiler-rt) that JIT-compiled modules >>>> may need. These are currently linked correctly because lli and friends >>>> are linked against libgcc_s.so (i.e. shared library version of libgcc, >>>> so dlopen/dlsym works). >>>> >>>> However if a consumer links statically, dlsym won't find all compiler >>>> required functions (only the ones that were required by the JIT-compiler >>>> itself will be linked). >>>> >>>> So the question is, do we want to support users of the MCJIT linking >>>> statically to libgcc/compiler-rt? Or explicitly not handle this >>>> use-case? >>>> >>>> Cheers, >>>> >>>> James >>>> >>>> >>>> >>>> _______________________________________________ >>>> LLVM Developers mailing list >>>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >>> >>> _______________________________________________ >>> LLVM Developers mailing list >>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >> > > >