Lang Hames via llvm-dev
2021-Apr-12 22:36 UTC
[llvm-dev] Inclusion of the ORC runtime in compiler-rt.
Hi Petr, I'd like to better understand the structure of the ORC runtime and its> dependencies (both build and runtime). Does it use the C or C++ standard > library? Does it depend on other parts of LLVM? Do you plan on reusing some > of the existing compiler-rt libraries like sanitizer_common?The ORC runtime currently uses the C++ standard library. Since the ORC runtime needs to communicate with the LLVM ORC library it also currently uses some header-only includes from LLVM. It does not depend on any LLVM libraries. We could duplicate this code, but I'd prefer to share it if possible. I have not used sanitizer_common, but some parts of it look like they may be useful. I gravitated towards implementing the ORC runtime in compiler-rt because I need to be able to write parts of it in platform-specific assembly (which compiler-rt supports), and because the runtime should be build for all targets, not just the host (which seems to be the standard way that compiler-rt is configured). To give a bit more background on why I'm interested, compiler-rt has grown> fairly organically this has been making the maintenance more and more > difficult, at least from the build perspective. There are some runtimes > that only use C, some that use C++, some that use C++ standard library. > When building compiler-rt together with other runtimes like libc or libc++, > it's difficult to pick up the right order which is why we have several > entry points into compiler-rt's build system to build different subsets and > that's been a maintenance headache.> I've been thinking about this quite a bit recently and I repeatedly came > to the conclusion that compiler-rt would ideally be broken up into several > subprojects, but that should probably be discussed as a separate topic. > However, understanding the build and runtimes dependencies of the ORC > runtime could help us decide whether it should be a part of compiler-rt or > be a separate subproject.That makes sense to me. I think of the ORC runtime as a compiler-rt-style runtime with a libc++ dependency. In that sense I think it's similar to libFuzzer, and whatever solution we come up with for libFuzzer would probably also be applicable to the ORC runtime too. -- Lang. On Mon, Apr 12, 2021 at 2:36 PM Petr Hosek <phosek at google.com> wrote:> I'd like to better understand the structure of the ORC runtime and its > dependencies (both build and runtime). Does it use the C or C++ standard > library? Does it depend on other parts of LLVM? Do you plan on reusing some > of the existing compiler-rt libraries like sanitizer_common? > > To give a bit more background on why I'm interested, compiler-rt has grown > fairly organically this has been making the maintenance more and more > difficult, at least from the build perspective. There are some runtimes > that only use C, some that use C++, some that use C++ standard library. > When building compiler-rt together with other runtimes like libc or libc++, > it's difficult to pick up the right order which is why we have several > entry points into compiler-rt's build system to build different subsets and > that's been a maintenance headache. > > I've been thinking about this quite a bit recently and I repeatedly came > to the conclusion that compiler-rt would ideally be broken up into several > subprojects, but that should probably be discussed as a separate topic. > However, understanding the build and runtimes dependencies of the ORC > runtime could help us decide whether it should be a part of compiler-rt or > be a separate subproject. > > On Mon, Apr 12, 2021 at 12:26 PM Lang Hames <lhames at gmail.com> wrote: > >> Hi All, >> >> I'd like to add the ORC runtime library (preview available at >> https://github.com/lhames/llvm-project/tree/orc-runtime-preview) to >> compiler-rt. >> >> Background: >> >> ORC, like MCJIT, can link JIT'd code either into the current process >> ("in-process" mode) or into a remote executor process ("cross-process" >> mode). Some JIT features require support code in the executor process, but >> the existing ORC libraries are only linked into the JIT process. This has >> made cross-process mode support for those features (which include static >> initializers, thread local variables, exception handling, and others) >> awkward or impractical to implement. The ORC runtime library aims to >> provide the necessary support code in a form that is loadable by the JIT >> itself, which should allow these features to work uniformly in both modes. >> >> My prototype branch of the ORC runtime (available at >> https://github.com/lhames/llvm-project/tree/orc-runtime-preview) has >> advanced to the point where it can provide uniform support for static >> initializers, destructors, exceptions, thread locals, and language >> registration for Objective C and Swift code. This support is all >> MachO/Darwin only so far, but should be easily adaptable for ELF/Linux/BSD >> support. >> >> Proposal: >> >> The proof of concept implementation has been very successful, so I would >> like to move future development to the LLVM main branch so that others can >> benefit from this. >> >> Before I start posting patches, though: >> >> Does anyone see any problems with including this in compiler-rt? >> >> Does anyone think that there is a more reasonable home for the ORC >> runtime within the llvm-project? I considered LLVM itself, or a new >> top-level project, but neither seemed as natural a fit as compiler-rt. >> >> Finally, if everyone is happy for it to be included in principle, are >> there any volunteers to review ORC runtime patches? >> >> Regards, >> Lang. >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210412/ae954bb5/attachment.html>
Eric Christopher via llvm-dev
2021-Apr-12 22:51 UTC
[llvm-dev] Inclusion of the ORC runtime in compiler-rt.
On Mon, Apr 12, 2021 at 6:36 PM Lang Hames <lhames at gmail.com> wrote:> Hi Petr, > > I'd like to better understand the structure of the ORC runtime and its >> dependencies (both build and runtime). Does it use the C or C++ standard >> library? Does it depend on other parts of LLVM? Do you plan on reusing some >> of the existing compiler-rt libraries like sanitizer_common? > > > The ORC runtime currently uses the C++ standard library. > Since the ORC runtime needs to communicate with the LLVM ORC library it > also currently uses some header-only includes from LLVM. It does not depend > on any LLVM libraries. We could duplicate this code, but I'd prefer to > share it if possible. > I have not used sanitizer_common, but some parts of it look like they may > be useful. > > I gravitated towards implementing the ORC runtime in compiler-rt because I > need to be able to write parts of it in platform-specific assembly (which > compiler-rt supports), and because the runtime should be build for all > targets, not just the host (which seems to be the standard way that > compiler-rt is configured). > > >From this it sounds like "convenient reusing of the build system" ratherthan "should be included in compiler-rt as a library"? If that's the case maybe making it clear or lifting the common build system support out might be maintainable without the "this is a runtime library for the system" sort of thing? -eric> To give a bit more background on why I'm interested, compiler-rt has grown >> fairly organically this has been making the maintenance more and more >> difficult, at least from the build perspective. There are some runtimes >> that only use C, some that use C++, some that use C++ standard library. >> When building compiler-rt together with other runtimes like libc or libc++, >> it's difficult to pick up the right order which is why we have several >> entry points into compiler-rt's build system to build different subsets and >> that's been a maintenance headache. > > >> I've been thinking about this quite a bit recently and I repeatedly came >> to the conclusion that compiler-rt would ideally be broken up into several >> subprojects, but that should probably be discussed as a separate topic. >> However, understanding the build and runtimes dependencies of the ORC >> runtime could help us decide whether it should be a part of compiler-rt or >> be a separate subproject. > > > That makes sense to me. I think of the ORC runtime as a compiler-rt-style > runtime with a libc++ dependency. In that sense I think it's similar to > libFuzzer, and whatever solution we come up with for libFuzzer would > probably also be applicable to the ORC runtime too. > > -- Lang. > > On Mon, Apr 12, 2021 at 2:36 PM Petr Hosek <phosek at google.com> wrote: > >> I'd like to better understand the structure of the ORC runtime and its >> dependencies (both build and runtime). Does it use the C or C++ standard >> library? Does it depend on other parts of LLVM? Do you plan on reusing some >> of the existing compiler-rt libraries like sanitizer_common? >> >> To give a bit more background on why I'm interested, compiler-rt has >> grown fairly organically this has been making the maintenance more and more >> difficult, at least from the build perspective. There are some runtimes >> that only use C, some that use C++, some that use C++ standard library. >> When building compiler-rt together with other runtimes like libc or libc++, >> it's difficult to pick up the right order which is why we have several >> entry points into compiler-rt's build system to build different subsets and >> that's been a maintenance headache. >> >> I've been thinking about this quite a bit recently and I repeatedly came >> to the conclusion that compiler-rt would ideally be broken up into several >> subprojects, but that should probably be discussed as a separate topic. >> However, understanding the build and runtimes dependencies of the ORC >> runtime could help us decide whether it should be a part of compiler-rt or >> be a separate subproject. >> >> On Mon, Apr 12, 2021 at 12:26 PM Lang Hames <lhames at gmail.com> wrote: >> >>> Hi All, >>> >>> I'd like to add the ORC runtime library (preview available at >>> https://github.com/lhames/llvm-project/tree/orc-runtime-preview) to >>> compiler-rt. >>> >>> Background: >>> >>> ORC, like MCJIT, can link JIT'd code either into the current process >>> ("in-process" mode) or into a remote executor process ("cross-process" >>> mode). Some JIT features require support code in the executor process, but >>> the existing ORC libraries are only linked into the JIT process. This has >>> made cross-process mode support for those features (which include static >>> initializers, thread local variables, exception handling, and others) >>> awkward or impractical to implement. The ORC runtime library aims to >>> provide the necessary support code in a form that is loadable by the JIT >>> itself, which should allow these features to work uniformly in both modes. >>> >>> My prototype branch of the ORC runtime (available at >>> https://github.com/lhames/llvm-project/tree/orc-runtime-preview) has >>> advanced to the point where it can provide uniform support for static >>> initializers, destructors, exceptions, thread locals, and language >>> registration for Objective C and Swift code. This support is all >>> MachO/Darwin only so far, but should be easily adaptable for ELF/Linux/BSD >>> support. >>> >>> Proposal: >>> >>> The proof of concept implementation has been very successful, so I would >>> like to move future development to the LLVM main branch so that others can >>> benefit from this. >>> >>> Before I start posting patches, though: >>> >>> Does anyone see any problems with including this in compiler-rt? >>> >>> Does anyone think that there is a more reasonable home for the ORC >>> runtime within the llvm-project? I considered LLVM itself, or a new >>> top-level project, but neither seemed as natural a fit as compiler-rt. >>> >>> Finally, if everyone is happy for it to be included in principle, are >>> there any volunteers to review ORC runtime patches? >>> >>> Regards, >>> Lang. >>> >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210412/f3bc19e7/attachment.html>