Does a HVM guest without PV drivers have any way to know it got resumed after a migration or save/restore cycle? Thanks, Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
No. Perhaps something could be added if it''s useful. -- Keir On 07/08/2009 09:34, "Jan Beulich" <JBeulich@novell.com> wrote:> Does a HVM guest without PV drivers have any way to know it got resumed > after a migration or save/restore cycle? > > Thanks, Jan > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Meanwhile I thought of a (theoretical) way, but to my surprise it wouldn''t work: The guest vCPU-s could check their registered runstate areas'' state field, which should be stale (i.e. not RUNSTATE_running), but the surprise (to me) was that this gets updated only on context switch in, not on switch out (which means that at present that field also cannot be used to determine whether a remote vCPU is currently running - another thing that I intended to exploit [accepting that the state maybe stale by the time it gets evaluated/used]). Fixing this is obviously trivial (under the premise that you''d take such a patch), but expected functionality then of course will depend upon this patch being present in hypervisors derived from older code bases. Jan>>> Keir Fraser <keir.fraser@eu.citrix.com> 07.08.09 11:01 >>>No. Perhaps something could be added if it''s useful. -- Keir On 07/08/2009 09:34, "Jan Beulich" <JBeulich@novell.com> wrote:> Does a HVM guest without PV drivers have any way to know it got resumed > after a migration or save/restore cycle? > > Thanks, Jan > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
So you''re interested in a scenario that is PV''ed-up enough to do hypercalls, but not enough to talk xenbus? If you only care about being able to poll for having migrated/saved, there must be some other bit of state you can push to Xen that doesn''t get migrated/saved. Like, create an IPI event channel and poll its status (EVTCHNOP_status). -- Keir On 07/08/2009 10:13, "Jan Beulich" <JBeulich@novell.com> wrote:> Meanwhile I thought of a (theoretical) way, but to my surprise it wouldn''t > work: The guest vCPU-s could check their registered runstate areas'' state > field, which should be stale (i.e. not RUNSTATE_running), but the surprise > (to me) was that this gets updated only on context switch in, not on > switch out (which means that at present that field also cannot be used to > determine whether a remote vCPU is currently running - another thing > that I intended to exploit [accepting that the state maybe stale by the > time it gets evaluated/used]). > > Fixing this is obviously trivial (under the premise that you''d take such a > patch), but expected functionality then of course will depend upon this > patch being present in hypervisors derived from older code bases. > > Jan > >>>> Keir Fraser <keir.fraser@eu.citrix.com> 07.08.09 11:01 >>> > No. Perhaps something could be added if it''s useful. > > -- Keir > > On 07/08/2009 09:34, "Jan Beulich" <JBeulich@novell.com> wrote: > >> Does a HVM guest without PV drivers have any way to know it got resumed >> after a migration or save/restore cycle? >> >> Thanks, Jan >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > Does a HVM guest without PV drivers have any way to know it gotresumed> after a migration or save/restore cycle? >I don''t think so, but why do you want to do this? Do you want to have a service or some other software that polls or something and says "a suspend or resume has happened since last poll"? James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> Keir Fraser <keir.fraser@eu.citrix.com> 07.08.09 11:44 >>> >So you''re interested in a scenario that is PV''ed-up enough to do hypercalls, >but not enough to talk xenbus?Yes - right now the main goal is to improve ticket lock behavior.>If you only care about being able to poll for having migrated/saved, there >must be some other bit of state you can push to Xen that doesn''t get >migrated/saved. Like, create an IPI event channel and poll its status >(EVTCHNOP_status).No, a hypercall of any sort is not a good option (namely if I wanted to use a hypercall page, which I''m not certain about yet, as that would create a chicken-and-egg problem). It must be some memory state that I can examine. And with the goal of improving lock behavior, I also definitely will need to have a way to know whether a remote vCPU is currently running, and here it is obviously preferable to not have to use a hypercall to find out (i.e. I''d want the runstate area change I''m looking at anyway). Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> "James Harper" <james.harper@bendigoit.com.au> 07.08.09 11:50 >>> >> >> Does a HVM guest without PV drivers have any way to know it gotresumed>> after a migration or save/restore cycle? >> > >I don''t think so, but why do you want to do this? > >Do you want to have a service or some other software that polls or >something and says "a suspend or resume has happened since last poll"?See my other reply - the goal is to have a way to yield the CPU from the spinlock contention path if I know (or better, can be reasonably sure) the lock owner isn''t currently executing. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 07/08/2009 10:53, "Jan Beulich" <JBeulich@novell.com> wrote:>> If you only care about being able to poll for having migrated/saved, there >> must be some other bit of state you can push to Xen that doesn''t get >> migrated/saved. Like, create an IPI event channel and poll its status >> (EVTCHNOP_status). > > No, a hypercall of any sort is not a good option (namely if I wanted to > use a hypercall page, which I''m not certain about yet, as that would > create a chicken-and-egg problem). It must be some memory state that > I can examine.So what does improving ticket lock behaviour have to do with save/restore? I''m a bit confused now.> And with the goal of improving lock behavior, I also definitely will need to > have a way to know whether a remote vCPU is currently running, and > here it is obviously preferable to not have to use a hypercall to find out > (i.e. I''d want the runstate area change I''m looking at anyway).I''ll certainly consider the patch. I''d also consider a CPUID flag to indicate its presence. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> Keir Fraser <keir.fraser@eu.citrix.com> 07.08.09 12:13 >>> >So what does improving ticket lock behaviour have to do with save/restore? >I''m a bit confused now.I need a hypercall to do the yield that I want to do when realizing that the lock owner is blocked. In order to find out whether the lock owner is (not) running, I need the runstate area (or else I''d need a second hypercall, which I dislike for the case the lock owner *is* running, and hence can expect that the local vCPU will be able to get the lock soon). The runstate area, however, needs to be re-registered after resume, and since I''m not not told that I''m being resumed, I need a way to figure this out from just memory state. Besides that, doing hypercalls here also requires that I''ll be able to re- setup the hypercall (or the hypercall page) in case I got migrated between VMX and SVM. While this could certainly be done in a fixup handler invoked from #UD (a little tricky for the hypercall page case), I find it preferable to synchronously fix things up beforehand.>I''ll certainly consider the patch. I''d also consider a CPUID flag to >indicate its presence.I thought about a feature flag briefly, but I''m not certain it''s worth it. Given the current state of things, the feature being absent will just result in all vCPU-s always being considered running, and hence the intended optimization just not taking any effect (but specifically also not having any adverse effect on the guest afaict). Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 07/08/2009 11:35, "Jan Beulich" <JBeulich@novell.com> wrote:> Besides that, doing hypercalls here also requires that I''ll be able to re- > setup the hypercall (or the hypercall page) in case I got migrated > between VMX and SVM. While this could certainly be done in a fixup > handler invoked from #UD (a little tricky for the hypercall page case), I > find it preferable to synchronously fix things up beforehand.Won''t you always have a race between checking for migration and then entering the cpu-specific hypercall stub? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> On 07/08/2009 11:35, "Jan Beulich" <JBeulich@novell.com> wrote: > > > Besides that, doing hypercalls here also requires that I''ll be ableto re-> > setup the hypercall (or the hypercall page) in case I got migrated > > between VMX and SVM. While this could certainly be done in a fixup > > handler invoked from #UD (a little tricky for the hypercall pagecase), I> > find it preferable to synchronously fix things up beforehand. > > Won''t you always have a race between checking for migration and then > entering the cpu-specific hypercall stub? >That''s what I was thinking too. The GPLPV drivers (and presumably the Linux PVonHVM drivers) are very careful about making sure that nothing else can touch any of the hypercall related stuff after a resume, keeping all CPU''s spinning at a high irql (interrupts disabled etc) until the hypercall page is restored. If you are using the hypercall page for anything then I think you need to have your driver manage the suspend operation too... I think that xen/xend becomes aware that you are running PV drivers on the basis that you have set a watch on the shutdown xenstore key, although I''m possibly wrong about that. James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> Keir Fraser <keir.fraser@eu.citrix.com> 07.08.09 12:53 >>> >On 07/08/2009 11:35, "Jan Beulich" <JBeulich@novell.com> wrote: > >> Besides that, doing hypercalls here also requires that I''ll be able to re- >> setup the hypercall (or the hypercall page) in case I got migrated >> between VMX and SVM. While this could certainly be done in a fixup >> handler invoked from #UD (a little tricky for the hypercall page case), I >> find it preferable to synchronously fix things up beforehand. > >Won''t you always have a race between checking for migration and then >entering the cpu-specific hypercall stub?Indeed. However, the fixup handler in this case can be trivial - it just needs to restart the whole operation (including the have-I-been-resumed check). Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 07/08/2009 12:23, "Jan Beulich" <JBeulich@novell.com> wrote:>> Won''t you always have a race between checking for migration and then >> entering the cpu-specific hypercall stub? > > Indeed. However, the fixup handler in this case can be trivial - it just > needs to restart the whole operation (including the have-I-been-resumed > check).Mmm fair enough, if you''re prepared to make your own VMCALL/VMMCALL stubs (which seems safe enough) that you know how to unroll from then that could work. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > >>> Keir Fraser <keir.fraser@eu.citrix.com> 07.08.09 12:53 >>> > >On 07/08/2009 11:35, "Jan Beulich" <JBeulich@novell.com> wrote: > > > >> Besides that, doing hypercalls here also requires that I''ll be ableto re-> >> setup the hypercall (or the hypercall page) in case I got migrated > >> between VMX and SVM. While this could certainly be done in a fixup > >> handler invoked from #UD (a little tricky for the hypercall pagecase), I> >> find it preferable to synchronously fix things up beforehand. > > > >Won''t you always have a race between checking for migration and then > >entering the cpu-specific hypercall stub? > > Indeed. However, the fixup handler in this case can be trivial - itjust> needs to restart the whole operation (including thehave-I-been-resumed> check). >What if it happens this way: if (just_been_resumed) restore_hypercall_page(); /* suspend/resume happens here */ do_hypercall(...); suddenly do_hypercall() jumps into empty space. The chance of it happening isn''t even that small if there is a lot of spinning going on. You could resolve it by having an MSR or something that you could write to to tell Xen you are in a critical section and it can''t do a suspend right now, but suddenly you''re doing more work than if you''d just put a xenbus driver in in the first place. James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 07/08/2009 12:34, "James Harper" <james.harper@bendigoit.com.au> wrote:>> Indeed. However, the fixup handler in this case can be trivial - it > just >> needs to restart the whole operation (including the > have-I-been-resumed >> check). >> > > What if it happens this way: > if (just_been_resumed) > restore_hypercall_page(); > /* suspend/resume happens here */ > do_hypercall(...); > > suddenly do_hypercall() jumps into empty space.Why would it jump into empty space? I''ll jump into a stale stub is all. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> "James Harper" <james.harper@bendigoit.com.au> 07.08.09 13:34 >>> >What if it happens this way: >if (just_been_resumed) > restore_hypercall_page(); >/* suspend/resume happens here */ >do_hypercall(...); > >suddenly do_hypercall() jumps into empty space.No, the old stubs would still be there. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > >>> "James Harper" <james.harper@bendigoit.com.au> 07.08.09 13:34 >>> > >What if it happens this way: > >if (just_been_resumed) > > restore_hypercall_page(); > >/* suspend/resume happens here */ > >do_hypercall(...); > > > >suddenly do_hypercall() jumps into empty space. > > No, the old stubs would still be there. >Yes. Sorry. My bad :( James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel