Lang Hames via llvm-dev
2021-Apr-14 17:34 UTC
[llvm-dev] Inclusion of the ORC runtime in compiler-rt.
Hi All, Petr -- since the ORC runtime's dependencies are similar to libFuzzer's, is there any reason not to land the ORC runtime in compiler-rt now and then break it out again later? If the compiler-rt refactor is likely to happen soon then it's worth waiting, otherwise I think landing it in compiler-rt sooner rather than later is the best option, so that any kinks in the integration can be worked out before any future compiler-rt refactor. -- Lang. On Mon, Apr 12, 2021 at 4:38 PM Lang Hames <lhames at gmail.com> wrote:> From this it sounds like "convenient reusing of the build system" rather >> than "should be included in compiler-rt as a library"? If that's the case >> maybe making it clear or lifting the common build system support out might >> be maintainable without the "this is a runtime library for the system" sort >> of thing? > > > Yeah. It sounds like in an ideal world we'd lift out the common build > system support, then have a new set of sub-projects that re-use that > generic build system. > > Do you have any sense of how difficult it would be to lift out that common > build system code? If that's relatively easy then maybe the right approach > is to do that first, then land the ORC runtime. Otherwise the ORC runtime > could go in to compiler-rt for now, then be split out with the rest of > compiler-rt when it's broken up -- it doesn't require any meaningful > changes to existing compiler-rt code, so it should be very easy to break > back out again later. > > -- Lang. > > > On Mon, Apr 12, 2021 at 3:51 PM Eric Christopher <echristo at gmail.com> > wrote: > >> >> >> On Mon, Apr 12, 2021 at 6:36 PM Lang Hames <lhames at gmail.com> wrote: >> >>> Hi Petr, >>> >>> I'd like to better understand the structure of the ORC runtime and its >>>> dependencies (both build and runtime). Does it use the C or C++ standard >>>> library? Does it depend on other parts of LLVM? Do you plan on reusing some >>>> of the existing compiler-rt libraries like sanitizer_common? >>> >>> >>> The ORC runtime currently uses the C++ standard library. >>> Since the ORC runtime needs to communicate with the LLVM ORC library it >>> also currently uses some header-only includes from LLVM. It does not depend >>> on any LLVM libraries. We could duplicate this code, but I'd prefer to >>> share it if possible. >>> I have not used sanitizer_common, but some parts of it look like they >>> may be useful. >>> >>> I gravitated towards implementing the ORC runtime in compiler-rt because >>> I need to be able to write parts of it in platform-specific assembly (which >>> compiler-rt supports), and because the runtime should be build for all >>> targets, not just the host (which seems to be the standard way that >>> compiler-rt is configured). >>> >>> >> From this it sounds like "convenient reusing of the build system" rather >> than "should be included in compiler-rt as a library"? If that's the case >> maybe making it clear or lifting the common build system support out might >> be maintainable without the "this is a runtime library for the system" sort >> of thing? >> >> -eric >> >> >>> To give a bit more background on why I'm interested, compiler-rt has >>>> grown fairly organically this has been making the maintenance more and more >>>> difficult, at least from the build perspective. There are some runtimes >>>> that only use C, some that use C++, some that use C++ standard library. >>>> When building compiler-rt together with other runtimes like libc or libc++, >>>> it's difficult to pick up the right order which is why we have several >>>> entry points into compiler-rt's build system to build different subsets and >>>> that's been a maintenance headache. >>> >>> >>>> I've been thinking about this quite a bit recently and I repeatedly >>>> came to the conclusion that compiler-rt would ideally be broken up into >>>> several subprojects, but that should probably be discussed as a >>>> separate topic. However, understanding the build and runtimes dependencies >>>> of the ORC runtime could help us decide whether it should be a part of >>>> compiler-rt or be a separate subproject. >>> >>> >>> That makes sense to me. I think of the ORC runtime as a >>> compiler-rt-style runtime with a libc++ dependency. In that sense I think >>> it's similar to libFuzzer, and whatever solution we come up with for >>> libFuzzer would probably also be applicable to the ORC runtime too. >>> >>> -- Lang. >>> >>> On Mon, Apr 12, 2021 at 2:36 PM Petr Hosek <phosek at google.com> wrote: >>> >>>> I'd like to better understand the structure of the ORC runtime and its >>>> dependencies (both build and runtime). Does it use the C or C++ standard >>>> library? Does it depend on other parts of LLVM? Do you plan on reusing some >>>> of the existing compiler-rt libraries like sanitizer_common? >>>> >>>> To give a bit more background on why I'm interested, compiler-rt has >>>> grown fairly organically this has been making the maintenance more and more >>>> difficult, at least from the build perspective. There are some runtimes >>>> that only use C, some that use C++, some that use C++ standard library. >>>> When building compiler-rt together with other runtimes like libc or libc++, >>>> it's difficult to pick up the right order which is why we have several >>>> entry points into compiler-rt's build system to build different subsets and >>>> that's been a maintenance headache. >>>> >>>> I've been thinking about this quite a bit recently and I repeatedly >>>> came to the conclusion that compiler-rt would ideally be broken up into >>>> several subprojects, but that should probably be discussed as a >>>> separate topic. However, understanding the build and runtimes dependencies >>>> of the ORC runtime could help us decide whether it should be a part of >>>> compiler-rt or be a separate subproject. >>>> >>>> On Mon, Apr 12, 2021 at 12:26 PM Lang Hames <lhames at gmail.com> wrote: >>>> >>>>> Hi All, >>>>> >>>>> I'd like to add the ORC runtime library (preview available at >>>>> https://github.com/lhames/llvm-project/tree/orc-runtime-preview) to >>>>> compiler-rt. >>>>> >>>>> Background: >>>>> >>>>> ORC, like MCJIT, can link JIT'd code either into the current process >>>>> ("in-process" mode) or into a remote executor process ("cross-process" >>>>> mode). Some JIT features require support code in the executor process, but >>>>> the existing ORC libraries are only linked into the JIT process. This has >>>>> made cross-process mode support for those features (which include static >>>>> initializers, thread local variables, exception handling, and others) >>>>> awkward or impractical to implement. The ORC runtime library aims to >>>>> provide the necessary support code in a form that is loadable by the JIT >>>>> itself, which should allow these features to work uniformly in both modes. >>>>> >>>>> My prototype branch of the ORC runtime (available at >>>>> https://github.com/lhames/llvm-project/tree/orc-runtime-preview) has >>>>> advanced to the point where it can provide uniform support for static >>>>> initializers, destructors, exceptions, thread locals, and language >>>>> registration for Objective C and Swift code. This support is all >>>>> MachO/Darwin only so far, but should be easily adaptable for ELF/Linux/BSD >>>>> support. >>>>> >>>>> Proposal: >>>>> >>>>> The proof of concept implementation has been very successful, so I >>>>> would like to move future development to the LLVM main branch so that >>>>> others can benefit from this. >>>>> >>>>> Before I start posting patches, though: >>>>> >>>>> Does anyone see any problems with including this in compiler-rt? >>>>> >>>>> Does anyone think that there is a more reasonable home for the ORC >>>>> runtime within the llvm-project? I considered LLVM itself, or a new >>>>> top-level project, but neither seemed as natural a fit as compiler-rt. >>>>> >>>>> Finally, if everyone is happy for it to be included in principle, are >>>>> there any volunteers to review ORC runtime patches? >>>>> >>>>> Regards, >>>>> Lang. >>>>> >>>>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210414/76597b15/attachment-0001.html>
Lang Hames via llvm-dev
2021-Apr-17 20:51 UTC
[llvm-dev] Inclusion of the ORC runtime in compiler-rt.
Hi All, I've broken the compiler-rt cmake changes and new directories out of the ORC runtime prototype and posted them for review in https://reviews.llvm.org/D100711. Most of this was adapted from xray's cmake files and project layout. I'm not a CMake expert, so I expect there's room for improvement here, but otherwise I'm hoping it's a pretty canonical "new compiler-rt library". Kind Regards, Lang. On Wed, Apr 14, 2021 at 10:34 AM Lang Hames <lhames at gmail.com> wrote:> Hi All, > > Petr -- since the ORC runtime's dependencies are similar to libFuzzer's, > is there any reason not to land the ORC runtime in compiler-rt now and then > break it out again later? If the compiler-rt refactor is likely to happen > soon then it's worth waiting, otherwise I think landing it in compiler-rt > sooner rather than later is the best option, so that any kinks in the > integration can be worked out before any future compiler-rt refactor. > > -- Lang. > > > > On Mon, Apr 12, 2021 at 4:38 PM Lang Hames <lhames at gmail.com> wrote: > >> From this it sounds like "convenient reusing of the build system" rather >>> than "should be included in compiler-rt as a library"? If that's the case >>> maybe making it clear or lifting the common build system support out might >>> be maintainable without the "this is a runtime library for the system" sort >>> of thing? >> >> >> Yeah. It sounds like in an ideal world we'd lift out the common build >> system support, then have a new set of sub-projects that re-use that >> generic build system. >> >> Do you have any sense of how difficult it would be to lift out that >> common build system code? If that's relatively easy then maybe the right >> approach is to do that first, then land the ORC runtime. Otherwise the ORC >> runtime could go in to compiler-rt for now, then be split out with the rest >> of compiler-rt when it's broken up -- it doesn't require any meaningful >> changes to existing compiler-rt code, so it should be very easy to break >> back out again later. >> >> -- Lang. >> >> >> On Mon, Apr 12, 2021 at 3:51 PM Eric Christopher <echristo at gmail.com> >> wrote: >> >>> >>> >>> On Mon, Apr 12, 2021 at 6:36 PM Lang Hames <lhames at gmail.com> wrote: >>> >>>> Hi Petr, >>>> >>>> I'd like to better understand the structure of the ORC runtime and its >>>>> dependencies (both build and runtime). Does it use the C or C++ standard >>>>> library? Does it depend on other parts of LLVM? Do you plan on reusing some >>>>> of the existing compiler-rt libraries like sanitizer_common? >>>> >>>> >>>> The ORC runtime currently uses the C++ standard library. >>>> Since the ORC runtime needs to communicate with the LLVM ORC library it >>>> also currently uses some header-only includes from LLVM. It does not depend >>>> on any LLVM libraries. We could duplicate this code, but I'd prefer to >>>> share it if possible. >>>> I have not used sanitizer_common, but some parts of it look like they >>>> may be useful. >>>> >>>> I gravitated towards implementing the ORC runtime in compiler-rt >>>> because I need to be able to write parts of it in platform-specific >>>> assembly (which compiler-rt supports), and because the runtime should be >>>> build for all targets, not just the host (which seems to be the standard >>>> way that compiler-rt is configured). >>>> >>>> >>> From this it sounds like "convenient reusing of the build system" rather >>> than "should be included in compiler-rt as a library"? If that's the case >>> maybe making it clear or lifting the common build system support out might >>> be maintainable without the "this is a runtime library for the system" sort >>> of thing? >>> >>> -eric >>> >>> >>>> To give a bit more background on why I'm interested, compiler-rt has >>>>> grown fairly organically this has been making the maintenance more and more >>>>> difficult, at least from the build perspective. There are some runtimes >>>>> that only use C, some that use C++, some that use C++ standard library. >>>>> When building compiler-rt together with other runtimes like libc or libc++, >>>>> it's difficult to pick up the right order which is why we have several >>>>> entry points into compiler-rt's build system to build different subsets and >>>>> that's been a maintenance headache. >>>> >>>> >>>>> I've been thinking about this quite a bit recently and I repeatedly >>>>> came to the conclusion that compiler-rt would ideally be broken up into >>>>> several subprojects, but that should probably be discussed as a >>>>> separate topic. However, understanding the build and runtimes dependencies >>>>> of the ORC runtime could help us decide whether it should be a part of >>>>> compiler-rt or be a separate subproject. >>>> >>>> >>>> That makes sense to me. I think of the ORC runtime as a >>>> compiler-rt-style runtime with a libc++ dependency. In that sense I think >>>> it's similar to libFuzzer, and whatever solution we come up with for >>>> libFuzzer would probably also be applicable to the ORC runtime too. >>>> >>>> -- Lang. >>>> >>>> On Mon, Apr 12, 2021 at 2:36 PM Petr Hosek <phosek at google.com> wrote: >>>> >>>>> I'd like to better understand the structure of the ORC runtime and its >>>>> dependencies (both build and runtime). Does it use the C or C++ standard >>>>> library? Does it depend on other parts of LLVM? Do you plan on reusing some >>>>> of the existing compiler-rt libraries like sanitizer_common? >>>>> >>>>> To give a bit more background on why I'm interested, compiler-rt has >>>>> grown fairly organically this has been making the maintenance more and more >>>>> difficult, at least from the build perspective. There are some runtimes >>>>> that only use C, some that use C++, some that use C++ standard library. >>>>> When building compiler-rt together with other runtimes like libc or libc++, >>>>> it's difficult to pick up the right order which is why we have several >>>>> entry points into compiler-rt's build system to build different subsets and >>>>> that's been a maintenance headache. >>>>> >>>>> I've been thinking about this quite a bit recently and I repeatedly >>>>> came to the conclusion that compiler-rt would ideally be broken up into >>>>> several subprojects, but that should probably be discussed as a >>>>> separate topic. However, understanding the build and runtimes dependencies >>>>> of the ORC runtime could help us decide whether it should be a part of >>>>> compiler-rt or be a separate subproject. >>>>> >>>>> On Mon, Apr 12, 2021 at 12:26 PM Lang Hames <lhames at gmail.com> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> I'd like to add the ORC runtime library (preview available at >>>>>> https://github.com/lhames/llvm-project/tree/orc-runtime-preview) to >>>>>> compiler-rt. >>>>>> >>>>>> Background: >>>>>> >>>>>> ORC, like MCJIT, can link JIT'd code either into the current process >>>>>> ("in-process" mode) or into a remote executor process ("cross-process" >>>>>> mode). Some JIT features require support code in the executor process, but >>>>>> the existing ORC libraries are only linked into the JIT process. This has >>>>>> made cross-process mode support for those features (which include static >>>>>> initializers, thread local variables, exception handling, and others) >>>>>> awkward or impractical to implement. The ORC runtime library aims to >>>>>> provide the necessary support code in a form that is loadable by the JIT >>>>>> itself, which should allow these features to work uniformly in both modes. >>>>>> >>>>>> My prototype branch of the ORC runtime (available at >>>>>> https://github.com/lhames/llvm-project/tree/orc-runtime-preview) has >>>>>> advanced to the point where it can provide uniform support for static >>>>>> initializers, destructors, exceptions, thread locals, and language >>>>>> registration for Objective C and Swift code. This support is all >>>>>> MachO/Darwin only so far, but should be easily adaptable for ELF/Linux/BSD >>>>>> support. >>>>>> >>>>>> Proposal: >>>>>> >>>>>> The proof of concept implementation has been very successful, so I >>>>>> would like to move future development to the LLVM main branch so that >>>>>> others can benefit from this. >>>>>> >>>>>> Before I start posting patches, though: >>>>>> >>>>>> Does anyone see any problems with including this in compiler-rt? >>>>>> >>>>>> Does anyone think that there is a more reasonable home for the ORC >>>>>> runtime within the llvm-project? I considered LLVM itself, or a new >>>>>> top-level project, but neither seemed as natural a fit as compiler-rt. >>>>>> >>>>>> Finally, if everyone is happy for it to be included in principle, are >>>>>> there any volunteers to review ORC runtime patches? >>>>>> >>>>>> Regards, >>>>>> Lang. >>>>>> >>>>>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210417/0357aca9/attachment.html>