Lang Hames
2015-Mar-19 01:39 UTC
[LLVMdev] How will OrcJIT guarantee thread-safety when a function is asked to be re generated?
Hi Hayden, Dave's answer covers this pretty well. Neither Orc nor MCJIT currently reason about replacing function bodies. They may let you add duplicate definitions, but how they'll behave if you do that isn't specified in their contracts. They definitely won't replace old definitions unless you provide a custom memory manager that's rigged to lay new definitions down on top of old ones. I suspect that existing clients of MCJIT have tackled this by adding thread safety into their wrappers around MCJIT, or into the JIT'd code itself, but I'm just guessing. (CC'ing Keno and Philip, in case they have insights). I think this would be cool to build in to Orc though. Two quick thoughts: (1) Replacing function bodies at the same address is impossible if the function is already on the stack: You'd be replacing a definition that you're later going to return through. So, if you want to replace functions at the same address you'll have to have some sort of safe-point concept where you know the function you want to replace isn't on the stack. (2) Replacing function bodies at the same address isn't the only way to avoid the overhead of a trampoline. I haven't implemented this yet, but I really want to add llvm.patchpoint support to Orc. In that case you can lay down your replacement definition at a different address, update all your callsites, then delete your old definition after you're done executing it. Relative to using trampolines this lowers your execution cost (calls are direct rather than indirect), but increases your update cost (you have to update many callsites, rather than a single trampoline). Out of interest, why the desire to avoid trampolines? They do make life a lot easier here. :) Cheers, Lang. On Wed, Mar 18, 2015 at 3:13 AM, David Blaikie <dblaikie at gmail.com> wrote:> [+Lang, keeper of JITs, designer of ORCs] > > On Tue, Mar 17, 2015 at 1:27 AM, Hayden Livingston <halivingston at gmail.com > > wrote: > >> I've been playing with OrcJIT a bit, and from the looks of it I can (like >> in the previous JIT I suppose?) ask for a function to be re generated. >> >> If I've given the address of the function that LLVM gave me to an >> external party, do "I" need to ensure thread-safety? >> >> Or is it safe to ask OrcJIT to re generate code at that address and >> everything will work magically? >> > > As I understand it, Orc won't regenerate the function at the same location > unless your memory manager returns the same memory twice - so if you know > you've successfully migrated all callers off a certain chunk of allocated > memory, you might be able to recycle it back into Orc (but I think on > MacOS, the way page permissions work, this would be impossible - once a > memory page is marked executable, it's no longer writable and can't be set > back - you need a new page). > > >> I'm thinking it won't because it's quite possible some thread might be >> executing code, and we'll be asking LLVM to write bytes there. >> >> How does one generally go do such updates? I'm looking for some guidance >> without adding a trampoline in front of it. Do runtimes that support >> re-generation of code have an if check or something before entering the >> method? >> > > Without a trampoline you're probably going to have to be constrained in > some other ways - possibly (& I'm really out of my depth at this point) the > kind of safe/pause points used for GC - but perhaps more constrained than > that, such that you have safe places where your JIT'd code (or at least the > replaceable functions) isn't running. > > But again, still depends on platform - writing to executable memory isn't > possible on MacOS so far as I know (as mentioned above) so there would be > no way to replace a function there without a trampoline or at least a > global variable to load/jump to. > > - David > > >> >> _______________________________________________ >> LLVM Developers mailing list >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150319/38d37b09/attachment.html>
Keno Fischer
2015-Mar-19 02:14 UTC
[LLVMdev] How will OrcJIT guarantee thread-safety when a function is asked to be re generated?
We haven't tackled either code replacement or threading yet, hence haven't run into this particular problem yet. I'm planning to push through my TLS patch for RuntimeDyld (which is my current blocker for adding threading), but parallel codegen will probably still be a little bit off for me. On the replacing functions front, I was planning to go the patchpoint route as well. On Wed, Mar 18, 2015 at 9:39 PM, Lang Hames <lhames at gmail.com> wrote:> Hi Hayden, > > Dave's answer covers this pretty well. Neither Orc nor MCJIT currently > reason about replacing function bodies. They may let you add duplicate > definitions, but how they'll behave if you do that isn't specified in their > contracts. They definitely won't replace old definitions unless you provide > a custom memory manager that's rigged to lay new definitions down on top of > old ones. > > I suspect that existing clients of MCJIT have tackled this by adding > thread safety into their wrappers around MCJIT, or into the JIT'd code > itself, but I'm just guessing. (CC'ing Keno and Philip, in case they have > insights). > > I think this would be cool to build in to Orc though. Two quick thoughts: > > (1) Replacing function bodies at the same address is impossible if the > function is already on the stack: You'd be replacing a definition that > you're later going to return through. So, if you want to replace functions > at the same address you'll have to have some sort of safe-point concept > where you know the function you want to replace isn't on the stack. > > (2) Replacing function bodies at the same address isn't the only way to > avoid the overhead of a trampoline. I haven't implemented this yet, but I > really want to add llvm.patchpoint support to Orc. In that case you can lay > down your replacement definition at a different address, update all your > callsites, then delete your old definition after you're done executing it. > Relative to using trampolines this lowers your execution cost (calls are > direct rather than indirect), but increases your update cost (you have to > update many callsites, rather than a single trampoline). > > Out of interest, why the desire to avoid trampolines? They do make life a > lot easier here. :) > > Cheers, > Lang. > > On Wed, Mar 18, 2015 at 3:13 AM, David Blaikie <dblaikie at gmail.com> wrote: > >> [+Lang, keeper of JITs, designer of ORCs] >> >> On Tue, Mar 17, 2015 at 1:27 AM, Hayden Livingston < >> halivingston at gmail.com> wrote: >> >>> I've been playing with OrcJIT a bit, and from the looks of it I can >>> (like in the previous JIT I suppose?) ask for a function to be re generated. >>> >>> If I've given the address of the function that LLVM gave me to an >>> external party, do "I" need to ensure thread-safety? >>> >>> Or is it safe to ask OrcJIT to re generate code at that address and >>> everything will work magically? >>> >> >> As I understand it, Orc won't regenerate the function at the same >> location unless your memory manager returns the same memory twice - so if >> you know you've successfully migrated all callers off a certain chunk of >> allocated memory, you might be able to recycle it back into Orc (but I >> think on MacOS, the way page permissions work, this would be impossible - >> once a memory page is marked executable, it's no longer writable and can't >> be set back - you need a new page). >> >> >>> I'm thinking it won't because it's quite possible some thread might be >>> executing code, and we'll be asking LLVM to write bytes there. >>> >>> How does one generally go do such updates? I'm looking for some guidance >>> without adding a trampoline in front of it. Do runtimes that support >>> re-generation of code have an if check or something before entering the >>> method? >>> >> >> Without a trampoline you're probably going to have to be constrained in >> some other ways - possibly (& I'm really out of my depth at this point) the >> kind of safe/pause points used for GC - but perhaps more constrained than >> that, such that you have safe places where your JIT'd code (or at least the >> replaceable functions) isn't running. >> >> But again, still depends on platform - writing to executable memory isn't >> possible on MacOS so far as I know (as mentioned above) so there would be >> no way to replace a function there without a trampoline or at least a >> global variable to load/jump to. >> >> - David >> >> >>> >>> _______________________________________________ >>> LLVM Developers mailing list >>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >>> >>> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150318/2e6400e8/attachment.html>
Sanjoy Das
2015-Mar-19 04:00 UTC
[LLVMdev] How will OrcJIT guarantee thread-safety when a function is asked to be re generated?
On Wed, Mar 18, 2015 at 6:39 PM, Lang Hames <lhames at gmail.com> wrote:> Hi Hayden, > > Dave's answer covers this pretty well. Neither Orc nor MCJIT currently > reason about replacing function bodies. They may let you add duplicate > definitions, but how they'll behave if you do that isn't specified in their > contracts. They definitely won't replace old definitions unless you provide > a custom memory manager that's rigged to lay new definitions down on top of > old ones. > > I suspect that existing clients of MCJIT have tackled this by adding thread > safety into their wrappers around MCJIT, or into the JIT'd code itself, but > I'm just guessing. (CC'ing Keno and Philip, in case they have insights). > > I think this would be cool to build in to Orc though. Two quick thoughts: > > (1) Replacing function bodies at the same address is impossible if the > function is already on the stack: You'd be replacing a definition that > you're later going to return through.If the function you wish to replace is active on the stack, you can replace the return PC that was going to return into that active frame with a PC pointing into a stub that knows how to replace the active stack frame with something that would let the new code continue executing. The stub will then have to branch into a suitable position in the new generated code. Once you have done this for all "pending returns" into the old bit of generated code, you can throw the old code away, since nothing will ever return into it. This can be tricky to get right but if you have built OSR support already for some other reason then this is a viable option. This scheme is very similar to throwing an exception, and the semantics of "catching" an exception is to branch to a newly generated block of code.> So, if you want to replace functions > at the same address you'll have to have some sort of safe-point concept > where you know the function you want to replace isn't on the stack.That will work, but can be very hard to make happen. For instance, the method you want to replace may have called a function that has an infinite loop in it.> > (2) Replacing function bodies at the same address isn't the only way to > avoid the overhead of a trampoline. I haven't implemented this yet, but I > really want to add llvm.patchpoint support to Orc. In that case you can lay > down your replacement definition at a different address, update all your > callsites, then delete your old definition after you're done executing it. > Relative to using trampolines this lowers your execution cost (calls are > direct rather than indirect), but increases your update cost (you have to > update many callsites, rather than a single trampoline). > > Out of interest, why the desire to avoid trampolines? They do make life a > lot easier here. :) > > Cheers, > Lang. > > On Wed, Mar 18, 2015 at 3:13 AM, David Blaikie <dblaikie at gmail.com> wrote: >> >> [+Lang, keeper of JITs, designer of ORCs] >> >> On Tue, Mar 17, 2015 at 1:27 AM, Hayden Livingston >> <halivingston at gmail.com> wrote: >>> >>> I've been playing with OrcJIT a bit, and from the looks of it I can (like >>> in the previous JIT I suppose?) ask for a function to be re generated. >>> >>> If I've given the address of the function that LLVM gave me to an >>> external party, do "I" need to ensure thread-safety? >>> >>> Or is it safe to ask OrcJIT to re generate code at that address and >>> everything will work magically? >> >> >> As I understand it, Orc won't regenerate the function at the same location >> unless your memory manager returns the same memory twice - so if you know >> you've successfully migrated all callers off a certain chunk of allocated >> memory, you might be able to recycle it back into Orc (but I think on MacOS, >> the way page permissions work, this would be impossible - once a memory page >> is marked executable, it's no longer writable and can't be set back - you >> need a new page). >> >>> >>> I'm thinking it won't because it's quite possible some thread might be >>> executing code, and we'll be asking LLVM to write bytes there. >>> >>> How does one generally go do such updates? I'm looking for some guidance >>> without adding a trampoline in front of it. Do runtimes that support >>> re-generation of code have an if check or something before entering the >>> method? >> >> >> Without a trampoline you're probably going to have to be constrained in >> some other ways - possibly (& I'm really out of my depth at this point) the >> kind of safe/pause points used for GC - but perhaps more constrained than >> that, such that you have safe places where your JIT'd code (or at least the >> replaceable functions) isn't running. >> >> But again, still depends on platform - writing to executable memory isn't >> possible on MacOS so far as I know (as mentioned above) so there would be no >> way to replace a function there without a trampoline or at least a global >> variable to load/jump to. >> >> - David >> >>> >>> >>> _______________________________________________ >>> LLVM Developers mailing list >>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >>> >> > > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >
Lang Hames
2015-Mar-19 06:47 UTC
[LLVMdev] How will OrcJIT guarantee thread-safety when a function is asked to be re generated?
Hi Sanjoy,>> (1) Replacing function bodies at the same address is impossible if the >> function is already on the stack: You'd be replacing a definition that >> you're later going to return through. > > If the function you wish to replace is active on the stack, you can > replace the return PC that was going to return into that active frame > with a PC pointing into a stub that knows how to replace the active > stack frame with something that would let the new code continue > executing. The stub will then have to branch into a suitable position > in the new generated code. Once you have done this for all "pending > returns" into the old bit of generated code, you can throw the old > code away, since nothing will ever return into it. > > This can be tricky to get right but if you have built OSR support > already for some other reason then this is a viable option. This > scheme is very similar to throwing an exception, and the semantics of > "catching" an exception is to branch to a newly generated block of > code.That all makes sense. What are your thoughts on the trade-offs of this vs the patchpoint approach though? If you can modify previously executable memory it seems like the patchpoint approach would have lower overhead, unless you have a truly huge number of callsites to update?>> So, if you want to replace functions >> at the same address you'll have to have some sort of safe-point concept >> where you know the function you want to replace isn't on the stack. > > That will work, but can be very hard to make happen. For instance, > the method you want to replace may have called a function that has an > infinite loop in it.Agreed. This *might* find a home in simple REPLs where calling an infinite-loop would be undesirable/unexpected behavior, but that's also an environment where you are unlikely to want reoptimization.>> (2) Replacing function bodies at the same address isn't the only way to >> avoid the overhead of a trampoline. I haven't implemented this yet, but I >> really want to add llvm.patchpoint support to Orc. In that case you can lay >> down your replacement definition at a different address, update all your >> callsites, then delete your old definition after you're done executing it. >> Relative to using trampolines this lowers your execution cost (calls are >> direct rather than indirect), but increases your update cost (you have to >> update many callsites, rather than a single trampoline).FWIW, Pete Cooper and I have tossed around ideas about adding utilities to Orc for injecting frame-residence counting and automatic cleanup in to functions to facilitate this 2nd approach. The rough idea was that each function would increment a counter on entry and decrement it on exit. Every time the counter hits zero it would check whether it has been "deleted" (presumably due to being replaced), and if so it would free its memory. This scheme should be easy to implement, but hasn't gone past speculation on our part. - Lang. Sent from my iPad> On Mar 19, 2015, at 3:00 PM, Sanjoy Das <sanjoy at playingwithpointers.com> wrote: > >> On Wed, Mar 18, 2015 at 6:39 PM, Lang Hames <lhames at gmail.com> wrote: >> Hi Hayden, >> >> Dave's answer covers this pretty well. Neither Orc nor MCJIT currently >> reason about replacing function bodies. They may let you add duplicate >> definitions, but how they'll behave if you do that isn't specified in their >> contracts. They definitely won't replace old definitions unless you provide >> a custom memory manager that's rigged to lay new definitions down on top of >> old ones. >> >> I suspect that existing clients of MCJIT have tackled this by adding thread >> safety into their wrappers around MCJIT, or into the JIT'd code itself, but >> I'm just guessing. (CC'ing Keno and Philip, in case they have insights). >> >> I think this would be cool to build in to Orc though. Two quick thoughts: >> >> (1) Replacing function bodies at the same address is impossible if the >> function is already on the stack: You'd be replacing a definition that >> you're later going to return through. > > If the function you wish to replace is active on the stack, you can > replace the return PC that was going to return into that active frame > with a PC pointing into a stub that knows how to replace the active > stack frame with something that would let the new code continue > executing. The stub will then have to branch into a suitable position > in the new generated code. Once you have done this for all "pending > returns" into the old bit of generated code, you can throw the old > code away, since nothing will ever return into it. > > This can be tricky to get right but if you have built OSR support > already for some other reason then this is a viable option. This > scheme is very similar to throwing an exception, and the semantics of > "catching" an exception is to branch to a newly generated block of > code. > >> So, if you want to replace functions >> at the same address you'll have to have some sort of safe-point concept >> where you know the function you want to replace isn't on the stack. > > That will work, but can be very hard to make happen. For instance, > the method you want to replace may have called a function that has an > infinite loop in it. > >> >> (2) Replacing function bodies at the same address isn't the only way to >> avoid the overhead of a trampoline. I haven't implemented this yet, but I >> really want to add llvm.patchpoint support to Orc. In that case you can lay >> down your replacement definition at a different address, update all your >> callsites, then delete your old definition after you're done executing it. >> Relative to using trampolines this lowers your execution cost (calls are >> direct rather than indirect), but increases your update cost (you have to >> update many callsites, rather than a single trampoline). >> >> Out of interest, why the desire to avoid trampolines? They do make life a >> lot easier here. :) >> >> Cheers, >> Lang. >> >>> On Wed, Mar 18, 2015 at 3:13 AM, David Blaikie <dblaikie at gmail.com> wrote: >>> >>> [+Lang, keeper of JITs, designer of ORCs] >>> >>> On Tue, Mar 17, 2015 at 1:27 AM, Hayden Livingston >>> <halivingston at gmail.com> wrote: >>>> >>>> I've been playing with OrcJIT a bit, and from the looks of it I can (like >>>> in the previous JIT I suppose?) ask for a function to be re generated. >>>> >>>> If I've given the address of the function that LLVM gave me to an >>>> external party, do "I" need to ensure thread-safety? >>>> >>>> Or is it safe to ask OrcJIT to re generate code at that address and >>>> everything will work magically? >>> >>> >>> As I understand it, Orc won't regenerate the function at the same location >>> unless your memory manager returns the same memory twice - so if you know >>> you've successfully migrated all callers off a certain chunk of allocated >>> memory, you might be able to recycle it back into Orc (but I think on MacOS, >>> the way page permissions work, this would be impossible - once a memory page >>> is marked executable, it's no longer writable and can't be set back - you >>> need a new page). >>> >>>> >>>> I'm thinking it won't because it's quite possible some thread might be >>>> executing code, and we'll be asking LLVM to write bytes there. >>>> >>>> How does one generally go do such updates? I'm looking for some guidance >>>> without adding a trampoline in front of it. Do runtimes that support >>>> re-generation of code have an if check or something before entering the >>>> method? >>> >>> >>> Without a trampoline you're probably going to have to be constrained in >>> some other ways - possibly (& I'm really out of my depth at this point) the >>> kind of safe/pause points used for GC - but perhaps more constrained than >>> that, such that you have safe places where your JIT'd code (or at least the >>> replaceable functions) isn't running. >>> >>> But again, still depends on platform - writing to executable memory isn't >>> possible on MacOS so far as I know (as mentioned above) so there would be no >>> way to replace a function there without a trampoline or at least a global >>> variable to load/jump to. >>> >>> - David >>> >>>> >>>> >>>> _______________________________________________ >>>> LLVM Developers mailing list >>>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >>>> >>> >> >> >> _______________________________________________ >> LLVM Developers mailing list >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150319/cc8ce9a7/attachment.html>
Philip Reames
2015-Mar-24 16:28 UTC
[LLVMdev] How will OrcJIT guarantee thread-safety when a function is asked to be re generated?
We solve this by using MCJIT to generate the code, and then managing it ourselves. We have an instance of MCJIT per compiler thread. We use MCJIT to perform one compilation at a time, and then disconnect the generated code from MCJIT. Our runtime has existing mechanisms for patching the call sites to point to the new version of the code. We've been able to use those essentially without modification. To put it differently, we consider that out of scope for MCJIT. p.s. It's worth stating that this type of code life cycle management is *hard*. If the older version is still valid when the new one is installed, it gets a bit easier, but if you have to invalidate before install, you need to build a sophisticated deoptimization mechanism. Philip On 03/18/2015 06:39 PM, Lang Hames wrote:> Hi Hayden, > > Dave's answer covers this pretty well. Neither Orc nor MCJIT currently > reason about replacing function bodies. They may let you add duplicate > definitions, but how they'll behave if you do that isn't specified in > their contracts. They definitely won't replace old definitions unless > you provide a custom memory manager that's rigged to lay new > definitions down on top of old ones. > > I suspect that existing clients of MCJIT have tackled this by adding > thread safety into their wrappers around MCJIT, or into the JIT'd code > itself, but I'm just guessing. (CC'ing Keno and Philip, in case they > have insights). > > I think this would be cool to build in to Orc though. Two quick thoughts: > > (1) Replacing function bodies at the same address is impossible if the > function is already on the stack: You'd be replacing a definition that > you're later going to return through. So, if you want to replace > functions at the same address you'll have to have some sort of > safe-point concept where you know the function you want to replace > isn't on the stack. > > (2) Replacing function bodies at the same address isn't the only way > to avoid the overhead of a trampoline. I haven't implemented this yet, > but I really want to add llvm.patchpoint support to Orc. In that case > you can lay down your replacement definition at a different address, > update all your callsites, then delete your old definition after > you're done executing it. Relative to using trampolines this lowers > your execution cost (calls are direct rather than indirect), but > increases your update cost (you have to update many callsites, rather > than a single trampoline). > > Out of interest, why the desire to avoid trampolines? They do make > life a lot easier here. :) > > Cheers, > Lang. > > On Wed, Mar 18, 2015 at 3:13 AM, David Blaikie <dblaikie at gmail.com > <mailto:dblaikie at gmail.com>> wrote: > > [+Lang, keeper of JITs, designer of ORCs] > > On Tue, Mar 17, 2015 at 1:27 AM, Hayden Livingston > <halivingston at gmail.com <mailto:halivingston at gmail.com>> wrote: > > I've been playing with OrcJIT a bit, and from the looks of it > I can (like in the previous JIT I suppose?) ask for a function > to be re generated. > > If I've given the address of the function that LLVM gave me to > an external party, do "I" need to ensure thread-safety? > > Or is it safe to ask OrcJIT to re generate code at that > address and everything will work magically? > > > As I understand it, Orc won't regenerate the function at the same > location unless your memory manager returns the same memory twice > - so if you know you've successfully migrated all callers off a > certain chunk of allocated memory, you might be able to recycle it > back into Orc (but I think on MacOS, the way page permissions > work, this would be impossible - once a memory page is marked > executable, it's no longer writable and can't be set back - you > need a new page). > > I'm thinking it won't because it's quite possible some thread > might be executing code, and we'll be asking LLVM to write > bytes there. > > How does one generally go do such updates? I'm looking for > some guidance without adding a trampoline in front of it. Do > runtimes that support re-generation of code have an if check > or something before entering the method? > > > Without a trampoline you're probably going to have to be > constrained in some other ways - possibly (& I'm really out of my > depth at this point) the kind of safe/pause points used for GC - > but perhaps more constrained than that, such that you have safe > places where your JIT'd code (or at least the replaceable > functions) isn't running. > > But again, still depends on platform - writing to executable > memory isn't possible on MacOS so far as I know (as mentioned > above) so there would be no way to replace a function there > without a trampoline or at least a global variable to load/jump to. > > - David > > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu> > http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150324/94329693/attachment.html>