Lang Hames via llvm-dev
2020-Jan-24 03:27 UTC
[llvm-dev] ORC JIT Weekly #2 -- COFF COMDAT Constants and Emulated TLS
Hi All, This week I've been focused on removing some of the blockers for people transitioning from ORCv1 to ORCv2. Issue #1 (http://llvm.org/PR40074, http://llvm.org/PR44337): When LLVM codegens floating point constants for COFF we produce named constant pool entries of the form __real@<bitval>. These are stored in COFF COMDAT sections [1] which allow duplicate symbol definitions to be discarded (roughly equivalent to the concept of a weak symbol). This posed two problems: (1) ORC had no idea about the (rough) mapping between COMDAT and Weak, and (2) did not support late introduction of weak symbols via MaterializationResponsibility::defineMaterializing. This caused use of the same floating point constant in multiple files to result in duplicate definition errors for the constant pool entries. Both issues have been addressed in 84217ad6611, and floating point constants should now work in COFF. If you run in to any trouble with them please file a bug and CC me. Issue #2: Emulated TLS and name mangling (raised by Geoff Levner on llvm-dev): Use of emulated-tls affects the name mangling of LLVM IR symbols. For a thread local symbol with name <Name>, when emulated-tls is enabled we will produce at least one emulated-tls symbol: __emutls_v.<Name> (which must then be linker-mangled to get the final symbol name). If that symbol has a non-zero initializer a second symbol will also be produced: __emutls_t.<Name>. ORCv2 requires up-front specification of the exact set of linker-level symbol names that a module produces, so that it can dispatch and track compiles, however none of the existing mangling code took emulated-tls into account. This has been fixed in ce2207abaf9 (with follow up fixes for the various examples I broke), and you should now be able to use thread-locals in ORCv2. Besides these two bugs, I have continued work on the big static initializer patch. Unfortunately issue #2 interacts with it, so I have to do some integration work before it is ready to land, but I'm hopeful that it will be ready some time next week. Finally, I have added a new example (bb7a5707ac0)to llvm/examples/LLJITExamples: LLJITWithObjectLinkingLayerPlugin demonstrating how to write a custom ObjectLinkingLayer plugin. ObjectLinkingLayer plugins can be used to receive and respond to events in the linking layer, and to inspect and modify JITLink LinkGraphs during the linking process. ORC uses this internally to register EH frames, but the system is likely to be of interest to anyone who wants low level control/visibility into the JIT linker. — Lang. [1] https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#comdat-sections-object-only -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200123/affc2a8d/attachment.html>
David Chisnall via llvm-dev
2020-Jan-24 13:08 UTC
[llvm-dev] ORC JIT Weekly #2 -- COFF COMDAT Constants and Emulated TLS
Hi Lang, Thank you very much for writing these. I suspect that they are likely to be drowned in the noise of this list and hidden from a lot of the people that would find them valuable. Would it be possible for them to be hosted somewhere permanently? Perhaps they could either be incorporated as a subsection of LLVM Weekly or put in a different location on the same server? David On 24/01/2020 03:27, Lang Hames via llvm-dev wrote:> Hi All, > > This week I've been focused on removing some of the blockers for people > transitioning from ORCv1 to ORCv2. > > Issue #1 (http://llvm.org/PR40074, http://llvm.org/PR44337): > > When LLVM codegens floating point constants for COFF we produce named > constant pool entries of the form __real@<bitval>. These are stored in > COFF COMDAT sections [1] which allow duplicate symbol definitions to be > discarded (roughly equivalent to the concept of a weak symbol). This > posed two problems: (1) ORC had no idea about the (rough) mapping > between COMDAT and Weak, and (2) did not support late introduction of > weak symbols via MaterializationResponsibility::defineMaterializing. > This caused use of the same floating point constant in multiple files to > result in duplicate definition errors for the constant pool entries. > > Both issues have been addressed in 84217ad6611, and floating point > constants should now work in COFF. If you run in to any trouble with > them please file a bug and CC me. > > Issue #2: Emulated TLS and name mangling (raised by Geoff Levner on > llvm-dev): > > Use of emulated-tls affects the name mangling of LLVM IR symbols. For a > thread local symbol with name <Name>, when emulated-tls is enabled we > will produce at least one emulated-tls symbol: __emutls_v.<Name> (which > must then be linker-mangled to get the final symbol name). If that > symbol has a non-zero initializer a second symbol will also be produced: > __emutls_t.<Name>. ORCv2 requires up-front specification of the exact > set of /linker-level/ symbol names that a module produces, so that it > can dispatch and track compiles, however none of the existing mangling > code took emulated-tls into account. This has been fixed in ce2207abaf9 > (with follow up fixes for the various examples I broke), and you should > now be able to use thread-locals in ORCv2. > > Besides these two bugs, I have continued work on the big static > initializer patch. Unfortunately issue #2 interacts with it, so I have > to do some integration work before it is ready to land, but I'm hopeful > that it will be ready some time next week. > > Finally, I have added a new example (bb7a5707ac0)to > llvm/examples/LLJITExamples: LLJITWithObjectLinkingLayerPlugin > demonstrating how to write a custom ObjectLinkingLayer plugin. > ObjectLinkingLayer plugins can be used to receive and respond to events > in the linking layer, and to inspect and modify JITLink LinkGraphs > during the linking process. ORC uses this internally to register EH > frames, but the system is likely to be of interest to anyone who wants > low level control/visibility into the JIT linker. > > — Lang. > > [1] > https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#comdat-sections-object-only > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >
Machiel van Hooren via llvm-dev
2020-Jan-24 16:06 UTC
[llvm-dev] ORC JIT Weekly #2 -- COFF COMDAT Constants and Emulated TLS
I'm very pleased that PR40074/44337 has been squashed, great work! This means I can now use unmodified LLVM for my application and perhaps soon target a stable release. Will it be part of the LLVM 10 release? Machiel On 24-Jan-20 04:27, Lang Hames via llvm-dev wrote:> Hi All, > > This week I've been focused on removing some of the blockers for > people transitioning from ORCv1 to ORCv2. > > Issue #1 (http://llvm.org/PR40074, http://llvm.org/PR44337): > > When LLVM codegens floating point constants for COFF we produce named > constant pool entries of the form __real@<bitval>. These are stored in > COFF COMDAT sections [1] which allow duplicate symbol definitions to > be discarded (roughly equivalent to the concept of a weak symbol). > This posed two problems: (1) ORC had no idea about the (rough) mapping > between COMDAT and Weak, and (2) did not support late introduction of > weak symbols via MaterializationResponsibility::defineMaterializing. > This caused use of the same floating point constant in multiple files > to result in duplicate definition errors for the constant pool entries. > > Both issues have been addressed in 84217ad6611, and floating point > constants should now work in COFF. If you run in to any trouble with > them please file a bug and CC me. > > Issue #2: Emulated TLS and name mangling (raised by Geoff Levner on > llvm-dev): > > Use of emulated-tls affects the name mangling of LLVM IR symbols. For > a thread local symbol with name <Name>, when emulated-tls is enabled > we will produce at least one emulated-tls symbol: __emutls_v.<Name> > (which must then be linker-mangled to get the final symbol name). If > that symbol has a non-zero initializer a second symbol will also be > produced: __emutls_t.<Name>. ORCv2 requires up-front specification of > the exact set of /linker-level/ symbol names that a module produces, > so that it can dispatch and track compiles, however none of the > existing mangling code took emulated-tls into account. This has been > fixed in ce2207abaf9 (with follow up fixes for the various examples I > broke), and you should now be able to use thread-locals in ORCv2. > > Besides these two bugs, I have continued work on the big static > initializer patch. Unfortunately issue #2 interacts with it, so I have > to do some integration work before it is ready to land, but I'm > hopeful that it will be ready some time next week. > > Finally, I have added a new example (bb7a5707ac0)to > llvm/examples/LLJITExamples: LLJITWithObjectLinkingLayerPlugin > demonstrating how to write a custom ObjectLinkingLayer plugin. > ObjectLinkingLayer plugins can be used to receive and respond to > events in the linking layer, and to inspect and modify JITLink > LinkGraphs during the linking process. ORC uses this internally to > register EH frames, but the system is likely to be of interest to > anyone who wants low level control/visibility into the JIT linker. > > — Lang. > > [1] > https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#comdat-sections-object-only > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200124/f701d024/attachment.html>
Lang Hames via llvm-dev
2020-Jan-28 21:46 UTC
[llvm-dev] ORC JIT Weekly #2 -- COFF COMDAT Constants and Emulated TLS
Hi Machiel, I have requested that 84217ad6611 be merged into the LLVM 10 release branch in http://llvm.org/PR44700. It's a low-risk fix, so I'm hopeful that it will make it in. -- Lang. On Fri, Jan 24, 2020 at 8:06 AM Machiel van Hooren <jcmac at machiel.info> wrote:> I'm very pleased that PR40074/44337 has been squashed, great work! This > means I can now use unmodified LLVM for my application and perhaps soon > target a stable release. Will it be part of the LLVM 10 release? > > Machiel > On 24-Jan-20 04:27, Lang Hames via llvm-dev wrote: > > Hi All, > > This week I've been focused on removing some of the blockers for people > transitioning from ORCv1 to ORCv2. > > Issue #1 (http://llvm.org/PR40074, http://llvm.org/PR44337): > > When LLVM codegens floating point constants for COFF we produce named > constant pool entries of the form __real@<bitval>. These are stored in > COFF COMDAT sections [1] which allow duplicate symbol definitions to be > discarded (roughly equivalent to the concept of a weak symbol). This posed > two problems: (1) ORC had no idea about the (rough) mapping between COMDAT > and Weak, and (2) did not support late introduction of weak symbols via > MaterializationResponsibility::defineMaterializing. This caused use of the > same floating point constant in multiple files to result in duplicate > definition errors for the constant pool entries. > > Both issues have been addressed in 84217ad6611, and floating point > constants should now work in COFF. If you run in to any trouble with them > please file a bug and CC me. > > Issue #2: Emulated TLS and name mangling (raised by Geoff Levner on > llvm-dev): > > Use of emulated-tls affects the name mangling of LLVM IR symbols. For a > thread local symbol with name <Name>, when emulated-tls is enabled we will > produce at least one emulated-tls symbol: __emutls_v.<Name> (which must > then be linker-mangled to get the final symbol name). If that symbol has a > non-zero initializer a second symbol will also be produced: > __emutls_t.<Name>. ORCv2 requires up-front specification of the exact set > of *linker-level* symbol names that a module produces, so that it can > dispatch and track compiles, however none of the existing mangling code > took emulated-tls into account. This has been fixed in ce2207abaf9 (with > follow up fixes for the various examples I broke), and you should now be > able to use thread-locals in ORCv2. > > Besides these two bugs, I have continued work on the big static > initializer patch. Unfortunately issue #2 interacts with it, so I have to > do some integration work before it is ready to land, but I'm hopeful that > it will be ready some time next week. > > Finally, I have added a new example (bb7a5707ac0)to > llvm/examples/LLJITExamples: LLJITWithObjectLinkingLayerPlugin > demonstrating how to write a custom ObjectLinkingLayer plugin. > ObjectLinkingLayer plugins can be used to receive and respond to events in > the linking layer, and to inspect and modify JITLink LinkGraphs during the > linking process. ORC uses this internally to register EH frames, but the > system is likely to be of interest to anyone who wants low level > control/visibility into the JIT linker. > > — Lang. > > [1] > https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#comdat-sections-object-only > > > _______________________________________________ > LLVM Developers mailing listllvm-dev at lists.llvm.orghttps://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200128/5184c2e1/attachment.html>
Lang Hames via llvm-dev
2020-Jan-28 22:10 UTC
[llvm-dev] ORC JIT Weekly #2 -- COFF COMDAT Constants and Emulated TLS
Hi David, Thank you very much for writing these. I suspect that they are likely> to be drowned in the noise of this list and hidden from a lot of the > people that would find them valuable. > > Would it be possible for them to be hosted somewhere permanently? > Perhaps they could either be incorporated as a subsection of LLVM Weekly > or put in a different location on the same server?The threads are archived on lists.llvm.org, but I'm not sure how discoverable they are there. I'm keen to make sure this is visible to the right people, and I'm not picky about the format. Does anyone else have any thoughts on this? Alex, how would you feel about me supplying a weekly JIT update as a subsection for LLVM Weekly, rather than starting a separate thread? (And thank you very much for posting links to the archives for the last two ORC JIT Weekly threads). -- Lang. On Fri, Jan 24, 2020 at 5:08 AM David Chisnall via llvm-dev < llvm-dev at lists.llvm.org> wrote:> Hi Lang, > > Thank you very much for writing these. I suspect that they are likely > to be drowned in the noise of this list and hidden from a lot of the > people that would find them valuable. > > Would it be possible for them to be hosted somewhere permanently? > Perhaps they could either be incorporated as a subsection of LLVM Weekly > or put in a different location on the same server? > > David > > On 24/01/2020 03:27, Lang Hames via llvm-dev wrote: > > Hi All, > > > > This week I've been focused on removing some of the blockers for people > > transitioning from ORCv1 to ORCv2. > > > > Issue #1 (http://llvm.org/PR40074, http://llvm.org/PR44337): > > > > When LLVM codegens floating point constants for COFF we produce named > > constant pool entries of the form __real@<bitval>. These are stored in > > COFF COMDAT sections [1] which allow duplicate symbol definitions to be > > discarded (roughly equivalent to the concept of a weak symbol). This > > posed two problems: (1) ORC had no idea about the (rough) mapping > > between COMDAT and Weak, and (2) did not support late introduction of > > weak symbols via MaterializationResponsibility::defineMaterializing. > > This caused use of the same floating point constant in multiple files to > > result in duplicate definition errors for the constant pool entries. > > > > Both issues have been addressed in 84217ad6611, and floating point > > constants should now work in COFF. If you run in to any trouble with > > them please file a bug and CC me. > > > > Issue #2: Emulated TLS and name mangling (raised by Geoff Levner on > > llvm-dev): > > > > Use of emulated-tls affects the name mangling of LLVM IR symbols. For a > > thread local symbol with name <Name>, when emulated-tls is enabled we > > will produce at least one emulated-tls symbol: __emutls_v.<Name> (which > > must then be linker-mangled to get the final symbol name). If that > > symbol has a non-zero initializer a second symbol will also be produced: > > __emutls_t.<Name>. ORCv2 requires up-front specification of the exact > > set of /linker-level/ symbol names that a module produces, so that it > > can dispatch and track compiles, however none of the existing mangling > > code took emulated-tls into account. This has been fixed in ce2207abaf9 > > (with follow up fixes for the various examples I broke), and you should > > now be able to use thread-locals in ORCv2. > > > > Besides these two bugs, I have continued work on the big static > > initializer patch. Unfortunately issue #2 interacts with it, so I have > > to do some integration work before it is ready to land, but I'm hopeful > > that it will be ready some time next week. > > > > Finally, I have added a new example (bb7a5707ac0)to > > llvm/examples/LLJITExamples: LLJITWithObjectLinkingLayerPlugin > > demonstrating how to write a custom ObjectLinkingLayer plugin. > > ObjectLinkingLayer plugins can be used to receive and respond to events > > in the linking layer, and to inspect and modify JITLink LinkGraphs > > during the linking process. ORC uses this internally to register EH > > frames, but the system is likely to be of interest to anyone who wants > > low level control/visibility into the JIT linker. > > > > — Lang. > > > > [1] > > > https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#comdat-sections-object-only > > > > > > _______________________________________________ > > LLVM Developers mailing list > > llvm-dev at lists.llvm.org > > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200128/52afa7c8/attachment-0001.html>