With Xen-4.1.2: I''m trying to change a register value in a paused vmx vcpu. The general process looks like this: 1. Some vmexit calls vcpu_sleep_nosync(v) on the vcpu 2. From dom0, I issue a domctl to change a register via v->arch.guest_context.user_reg, then vcpu_wake(v) However, the guest register does not seem to be changed when I do it this way. Is there something I need to do to mark the registers as "dirty" ? Is there a way to force the foreign vcpu to update the changed registers? Or maybe I just have to change the registers somewhere else? I''ve tried directly using vmcs_enter(v), __vmwrite(), vmcs_exit(v) also, but that doesn''t seem to make a change either. Thanks! _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 23/08/2012 18:25, "Cutter 409" <cutter409@gmail.com> wrote:> With Xen-4.1.2: > > I''m trying to change a register value in a paused vmx vcpu. The general > process looks like this: > > 1. Some vmexit calls vcpu_sleep_nosync(v) on the vcpu > 2. From dom0, I issue a domctl to change a register via > v->arch.guest_context.user_reg, then vcpu_wake(v)Which domctl? From dom0 userspace you can use the libxc functions xc_vcpu_getcontext() and xc_vcpu_setcontext() to read/modify register state. You can read the libxc sources to see what hypercall these map to, if you don''t want to use libxc for any reason. -- Keir> However, the guest register does not seem to be changed when I do it this way. > Is there something I need to do to mark the registers as "dirty" ? Is there a > way to force the foreign vcpu to update the changed registers? Or maybe I just > have to change the registers somewhere else? > > I''ve tried directly using vmcs_enter(v), __vmwrite(), vmcs_exit(v) also, but > that doesn''t seem to make a change either. > > Thanks! > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
I''m making the register change directly from the hypervisor, inside of the domctl code. It''s a custom domctl that I''ve added. I''ll look into what setcontext does after it modifies the register values, though. Thank you! On Thu, Aug 23, 2012 at 1:34 PM, Keir Fraser <keir.xen@gmail.com> wrote:> On 23/08/2012 18:25, "Cutter 409" <cutter409@gmail.com> wrote: > > > With Xen-4.1.2: > > > > I''m trying to change a register value in a paused vmx vcpu. The general > > process looks like this: > > > > 1. Some vmexit calls vcpu_sleep_nosync(v) on the vcpu > > 2. From dom0, I issue a domctl to change a register via > > v->arch.guest_context.user_reg, then vcpu_wake(v) > > Which domctl? From dom0 userspace you can use the libxc functions > xc_vcpu_getcontext() and xc_vcpu_setcontext() to read/modify register > state. > > You can read the libxc sources to see what hypercall these map to, if you > don''t want to use libxc for any reason. > > -- Keir > > > However, the guest register does not seem to be changed when I do it > this way. > > Is there something I need to do to mark the registers as "dirty" ? Is > there a > > way to force the foreign vcpu to update the changed registers? Or maybe > I just > > have to change the registers somewhere else? > > > > I''ve tried directly using vmcs_enter(v), __vmwrite(), vmcs_exit(v) also, > but > > that doesn''t seem to make a change either. > > > > Thanks! > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org > > http://lists.xen.org/xen-devel > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
FWIW I would expect your approach to basically work. Except... Does your domctl do a vcpu_pause()/vcpu_unpause() on the vcpu? This will ensure that the vcpu is both fully de-scheduled, and all of its register state is synced back into its vcpu structure. Otherwise you race the vcpu_sleep_nosync() -- and that¹s assuming you also have a reason for that vcpu to sleep (e.g., non-zero pause counter), else vcpu_sleep_*() operations do nothing! In short, your problems are almost certainly something to do with the subtleties of actually putting a vcpu properly to sleep. -- Keir On 23/08/2012 18:37, "Cutter 409" <cutter409@gmail.com> wrote:> I''m making the register change directly from the hypervisor, inside of the > domctl code. > > It''s a custom domctl that I''ve added. I''ll look into what setcontext does > after it modifies the register values, though. > > Thank you! > > On Thu, Aug 23, 2012 at 1:34 PM, Keir Fraser <keir.xen@gmail.com> wrote: >> On 23/08/2012 18:25, "Cutter 409" <cutter409@gmail.com> wrote: >> >>> > With Xen-4.1.2: >>> > >>> > I''m trying to change a register value in a paused vmx vcpu. The general >>> > process looks like this: >>> > >>> > 1. Some vmexit calls vcpu_sleep_nosync(v) on the vcpu >>> > 2. From dom0, I issue a domctl to change a register via >>> > v->arch.guest_context.user_reg, then vcpu_wake(v) >> >> Which domctl? From dom0 userspace you can use the libxc functions >> xc_vcpu_getcontext() and xc_vcpu_setcontext() to read/modify register state. >> >> You can read the libxc sources to see what hypercall these map to, if you >> don''t want to use libxc for any reason. >> >> -- Keir >> >>> > However, the guest register does not seem to be changed when I do it this >>> way. >>> > Is there something I need to do to mark the registers as "dirty" ? Is >>> there a >>> > way to force the foreign vcpu to update the changed registers? Or maybe I >>> just >>> > have to change the registers somewhere else? >>> > >>> > I''ve tried directly using vmcs_enter(v), __vmwrite(), vmcs_exit(v) also, >>> but >>> > that doesn''t seem to make a change either. >>> > >>> > Thanks! >>> > >>> > >>> > >>> > _______________________________________________ >>> > Xen-devel mailing list >>> > Xen-devel@lists.xen.org >>> > http://lists.xen.org/xen-devel >> >> > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
So, for example, one possibly-valid scheme would be: - vcpu_pause_nosync() from the vmexit handler - vcpu_sleep_sync() at the start of the domctl - vcpu_unpause() at the end of the domctl HTH, Keir On 23/08/2012 18:49, "Keir Fraser" <keir.xen@gmail.com> wrote:> FWIW I would expect your approach to basically work. > > Except... Does your domctl do a vcpu_pause()/vcpu_unpause() on the vcpu? This > will ensure that the vcpu is both fully de-scheduled, and all of its register > state is synced back into its vcpu structure. > > Otherwise you race the vcpu_sleep_nosync() -- and that¹s assuming you also > have a reason for that vcpu to sleep (e.g., non-zero pause counter), else > vcpu_sleep_*() operations do nothing! > > In short, your problems are almost certainly something to do with the > subtleties of actually putting a vcpu properly to sleep. > > -- Keir > > > On 23/08/2012 18:37, "Cutter 409" <cutter409@gmail.com> wrote: > >> I''m making the register change directly from the hypervisor, inside of the >> domctl code. >> >> It''s a custom domctl that I''ve added. I''ll look into what setcontext does >> after it modifies the register values, though. >> >> Thank you! >> >> On Thu, Aug 23, 2012 at 1:34 PM, Keir Fraser <keir.xen@gmail.com> wrote: >>> On 23/08/2012 18:25, "Cutter 409" <cutter409@gmail.com> wrote: >>> >>>> > With Xen-4.1.2: >>>> > >>>> > I''m trying to change a register value in a paused vmx vcpu. The general >>>> > process looks like this: >>>> > >>>> > 1. Some vmexit calls vcpu_sleep_nosync(v) on the vcpu >>>> > 2. From dom0, I issue a domctl to change a register via >>>> > v->arch.guest_context.user_reg, then vcpu_wake(v) >>> >>> Which domctl? From dom0 userspace you can use the libxc functions >>> xc_vcpu_getcontext() and xc_vcpu_setcontext() to read/modify register state. >>> >>> You can read the libxc sources to see what hypercall these map to, if you >>> don''t want to use libxc for any reason. >>> >>> -- Keir >>> >>>> > However, the guest register does not seem to be changed when I do it this >>>> way. >>>> > Is there something I need to do to mark the registers as "dirty" ? Is >>>> there a >>>> > way to force the foreign vcpu to update the changed registers? Or maybe I >>>> just >>>> > have to change the registers somewhere else? >>>> > >>>> > I''ve tried directly using vmcs_enter(v), __vmwrite(), vmcs_exit(v) also, >>>> but >>>> > that doesn''t seem to make a change either. >>>> > >>>> > Thanks! >>>> > >>>> > >>>> > >>>> > _______________________________________________ >>>> > Xen-devel mailing list >>>> > Xen-devel@lists.xen.org >>>> > http://lists.xen.org/xen-devel >>> >>> >> >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Thanks, Keir! I''ve spent so much time trying to track down this problem, even before I realized the registers weren''t actually changing. You have no idea how helpful that was. Before I tried your example, I just wrapped the code to change the register in vcpu_pause() and vcpu_unpause(), which worked. Everything seems fine at the moment, is there any reason I should still change the vcpu_sleep_nosync() to vcpu_pause_nosync()? It seems to actually work as is; I''m setting a bit in v->pause_flags before I call it, then clear the bit before I wake it. I also tried pause_nosync on vmexit, unpause after domctl, but that didn''t work. Thanks again! On Thu, Aug 23, 2012 at 1:54 PM, Keir Fraser <keir.xen@gmail.com> wrote:> So, for example, one possibly-valid scheme would be: > > - vcpu_pause_nosync() from the vmexit handler > > - vcpu_sleep_sync() at the start of the domctl > - vcpu_unpause() at the end of the domctl > > HTH, > Keir > > > > On 23/08/2012 18:49, "Keir Fraser" <keir.xen@gmail.com> wrote: > > FWIW I would expect your approach to basically work. > > Except... Does your domctl do a vcpu_pause()/vcpu_unpause() on the vcpu? > This will ensure that the vcpu is both fully de-scheduled, and all of its > register state is synced back into its vcpu structure. > > Otherwise you race the vcpu_sleep_nosync() -- and that’s assuming you also > have a reason for that vcpu to sleep (e.g., non-zero pause counter), else > vcpu_sleep_*() operations do nothing! > > In short, your problems are almost certainly something to do with the > subtleties of actually putting a vcpu properly to sleep. > > -- Keir > > > On 23/08/2012 18:37, "Cutter 409" <cutter409@gmail.com> wrote: > > I''m making the register change directly from the hypervisor, inside of the > domctl code. > > It''s a custom domctl that I''ve added. I''ll look into what setcontext does > after it modifies the register values, though. > > Thank you! > > On Thu, Aug 23, 2012 at 1:34 PM, Keir Fraser <keir.xen@gmail.com> wrote: > > On 23/08/2012 18:25, "Cutter 409" <cutter409@gmail.com> wrote: > > > With Xen-4.1.2: > > > > I''m trying to change a register value in a paused vmx vcpu. The general > > process looks like this: > > > > 1. Some vmexit calls vcpu_sleep_nosync(v) on the vcpu > > 2. From dom0, I issue a domctl to change a register via > > v->arch.guest_context.user_reg, then vcpu_wake(v) > > Which domctl? From dom0 userspace you can use the libxc functions > xc_vcpu_getcontext() and xc_vcpu_setcontext() to read/modify register > state. > > You can read the libxc sources to see what hypercall these map to, if you > don''t want to use libxc for any reason. > > -- Keir > > > However, the guest register does not seem to be changed when I do it > this way. > > Is there something I need to do to mark the registers as "dirty" ? Is > there a > > way to force the foreign vcpu to update the changed registers? Or maybe > I just > > have to change the registers somewhere else? > > > > I''ve tried directly using vmcs_enter(v), __vmwrite(), vmcs_exit(v) also, > but > > that doesn''t seem to make a change either. > > > > Thanks! > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org > > http://lists.xen.org/xen-devel > > > > > ------------------------------ > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
If you have your own flag in v->pause_flags then indeed you do not need to vcpu_pause_nosync() in the vmexit handler. The best sequence would be: - vmexit handler: set flag in v->pause_flags, then vcpu_sleep_nosync() - domctl entry: vcpu_sleep_sync() - domctl exit: clear flag in v->pause_flags, then vcpu_wake() So that¹s pretty much what you had in the first place, except for the extra vcpu_sleep_sync() on domctl entry. That¹s absolutely critical, and why your pause_nosync on vmexit, unpause after domctl doesn¹t work something is needed on domctl entry to be sure that the vcpu is descheduled and its state is synchronised. Of course the extra machinery of vcpu_pause/unpause is harmless enough, but it¹s not actually necessary here. -- Keir On 23/08/2012 19:11, "Cutter 409" <cutter409@gmail.com> wrote:> Thanks, Keir! > > I''ve spent so much time trying to track down this problem, even before I > realized the registers weren''t actually changing. You have no idea how helpful > that was. > > Before I tried your example, I just wrapped the code to change the register in > vcpu_pause() and vcpu_unpause(), which worked. > > Everything seems fine at the moment, is there any reason I should still change > the vcpu_sleep_nosync() to vcpu_pause_nosync()? It seems to actually work as > is; I''m setting a bit in v->pause_flags before I call it, then clear the bit > before I wake it. I also tried pause_nosync on vmexit, unpause after domctl, > but that didn''t work. > > Thanks again! > > > On Thu, Aug 23, 2012 at 1:54 PM, Keir Fraser <keir.xen@gmail.com> wrote: >> So, for example, one possibly-valid scheme would be: >> >> - vcpu_pause_nosync() from the vmexit handler >> >> - vcpu_sleep_sync() at the start of the domctl >> - vcpu_unpause() at the end of the domctl >> >> HTH, >> Keir >> >> >> >> On 23/08/2012 18:49, "Keir Fraser" <keir.xen@gmail.com >> <http://keir.xen@gmail.com> > wrote: >> >>> FWIW I would expect your approach to basically work. >>> >>> Except... Does your domctl do a vcpu_pause()/vcpu_unpause() on the vcpu? >>> This will ensure that the vcpu is both fully de-scheduled, and all of its >>> register state is synced back into its vcpu structure. >>> >>> Otherwise you race the vcpu_sleep_nosync() -- and that¹s assuming you also >>> have a reason for that vcpu to sleep (e.g., non-zero pause counter), else >>> vcpu_sleep_*() operations do nothing! >>> >>> In short, your problems are almost certainly something to do with the >>> subtleties of actually putting a vcpu properly to sleep. >>> >>> -- Keir >>> >>> >>> On 23/08/2012 18:37, "Cutter 409" <cutter409@gmail.com >>> <http://cutter409@gmail.com> > wrote: >>> >>>> I''m making the register change directly from the hypervisor, inside of the >>>> domctl code. >>>> >>>> It''s a custom domctl that I''ve added. I''ll look into what setcontext does >>>> after it modifies the register values, though. >>>> >>>> Thank you! >>>> >>>> On Thu, Aug 23, 2012 at 1:34 PM, Keir Fraser <keir.xen@gmail.com >>>> <http://keir.xen@gmail.com> > wrote: >>>>> On 23/08/2012 18:25, "Cutter 409" <cutter409@gmail.com >>>>> <http://cutter409@gmail.com> > wrote: >>>>> >>>>>> > With Xen-4.1.2: >>>>>> > >>>>>> > I''m trying to change a register value in a paused vmx vcpu. The general >>>>>> > process looks like this: >>>>>> > >>>>>> > 1. Some vmexit calls vcpu_sleep_nosync(v) on the vcpu >>>>>> > 2. From dom0, I issue a domctl to change a register via >>>>>> > v->arch.guest_context.user_reg, then vcpu_wake(v) >>>>> >>>>> Which domctl? From dom0 userspace you can use the libxc functions >>>>> xc_vcpu_getcontext() and xc_vcpu_setcontext() to read/modify register >>>>> state. >>>>> >>>>> You can read the libxc sources to see what hypercall these map to, if you >>>>> don''t want to use libxc for any reason. >>>>> >>>>> -- Keir >>>>> >>>>>> > However, the guest register does not seem to be changed when I do it >>>>>> this way. >>>>>> > Is there something I need to do to mark the registers as "dirty" ? Is >>>>>> there a >>>>>> > way to force the foreign vcpu to update the changed registers? Or maybe >>>>>> I just >>>>>> > have to change the registers somewhere else? >>>>>> > >>>>>> > I''ve tried directly using vmcs_enter(v), __vmwrite(), vmcs_exit(v) >>>>>> also, but >>>>>> > that doesn''t seem to make a change either. >>>>>> > >>>>>> > Thanks! >>>>>> > >>>>>> > >>>>>> > >>>>>> > _______________________________________________ >>>>>> > Xen-devel mailing list >>>>>> > Xen-devel@lists.xen.org <http://Xen-devel@lists.xen.org> >>>>>> > http://lists.xen.org/xen-devel >>>>> >>>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@lists.xen.org <http://Xen-devel@lists.xen.org> >>>> http://lists.xen.org/xen-devel >>> > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
That works perfectly, thank you for the help. On Thu, Aug 23, 2012 at 2:55 PM, Keir Fraser <keir.xen@gmail.com> wrote:> If you have your own flag in v->pause_flags then indeed you do not need > to vcpu_pause_nosync() in the vmexit handler. > > The best sequence would be: > - vmexit handler: set flag in v->pause_flags, then vcpu_sleep_nosync() > - domctl entry: vcpu_sleep_sync() > - domctl exit: clear flag in v->pause_flags, then vcpu_wake() > > So that’s pretty much what you had in the first place, except for the > extra vcpu_sleep_sync() on domctl entry. That’s absolutely critical, and > why your pause_nosync on vmexit, unpause after domctl doesn’t work — * > something* is needed on domctl entry to be sure that the vcpu is > descheduled and its state is synchronised. Of course the extra machinery of > vcpu_pause/unpause is harmless enough, but it’s not actually necessary here. > > -- Keir > > > > On 23/08/2012 19:11, "Cutter 409" <cutter409@gmail.com> wrote: > > Thanks, Keir! > > I''ve spent so much time trying to track down this problem, even before I > realized the registers weren''t actually changing. You have no idea how > helpful that was. > > Before I tried your example, I just wrapped the code to change the > register in vcpu_pause() and vcpu_unpause(), which worked. > > Everything seems fine at the moment, is there any reason I should still > change the vcpu_sleep_nosync() to vcpu_pause_nosync()? It seems to actually > work as is; I''m setting a bit in v->pause_flags before I call it, then > clear the bit before I wake it. I also tried pause_nosync on vmexit, > unpause after domctl, but that didn''t work. > > Thanks again! > > > On Thu, Aug 23, 2012 at 1:54 PM, Keir Fraser <keir.xen@gmail.com> wrote: > > So, for example, one possibly-valid scheme would be: > > - vcpu_pause_nosync() from the vmexit handler > > - vcpu_sleep_sync() at the start of the domctl > - vcpu_unpause() at the end of the domctl > > HTH, > Keir > > > > On 23/08/2012 18:49, "Keir Fraser" <keir.xen@gmail.com < > http://keir.xen@gmail.com> > wrote: > > FWIW I would expect your approach to basically work. > > Except... Does your domctl do a vcpu_pause()/vcpu_unpause() on the vcpu? > This will ensure that the vcpu is both fully de-scheduled, and all of its > register state is synced back into its vcpu structure. > > Otherwise you race the vcpu_sleep_nosync() -- and that’s assuming you also > have a reason for that vcpu to sleep (e.g., non-zero pause counter), else > vcpu_sleep_*() operations do nothing! > > In short, your problems are almost certainly something to do with the > subtleties of actually putting a vcpu properly to sleep. > > -- Keir > > > On 23/08/2012 18:37, "Cutter 409" <cutter409@gmail.com < > http://cutter409@gmail.com> > wrote: > > I''m making the register change directly from the hypervisor, inside of the > domctl code. > > It''s a custom domctl that I''ve added. I''ll look into what setcontext does > after it modifies the register values, though. > > Thank you! > > On Thu, Aug 23, 2012 at 1:34 PM, Keir Fraser <keir.xen@gmail.com < > http://keir.xen@gmail.com> > wrote: > > On 23/08/2012 18:25, "Cutter 409" <cutter409@gmail.com < > http://cutter409@gmail.com> > wrote: > > > With Xen-4.1.2: > > > > I''m trying to change a register value in a paused vmx vcpu. The general > > process looks like this: > > > > 1. Some vmexit calls vcpu_sleep_nosync(v) on the vcpu > > 2. From dom0, I issue a domctl to change a register via > > v->arch.guest_context.user_reg, then vcpu_wake(v) > > Which domctl? From dom0 userspace you can use the libxc functions > xc_vcpu_getcontext() and xc_vcpu_setcontext() to read/modify register > state. > > You can read the libxc sources to see what hypercall these map to, if you > don''t want to use libxc for any reason. > > -- Keir > > > However, the guest register does not seem to be changed when I do it > this way. > > Is there something I need to do to mark the registers as "dirty" ? Is > there a > > way to force the foreign vcpu to update the changed registers? Or maybe > I just > > have to change the registers somewhere else? > > > > I''ve tried directly using vmcs_enter(v), __vmwrite(), vmcs_exit(v) also, > but > > that doesn''t seem to make a change either. > > > > Thanks! > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org <http://Xen-devel@lists.xen.org> > > http://lists.xen.org/xen-devel > > > > > ------------------------------ > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org <http://Xen-devel@lists.xen.org> > http://lists.xen.org/xen-devel > > > > > ------------------------------ > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel