Xen devel - Oct 2013 - Questions regarding implementing mem_event for PV

I am looking in to implementing mem_event for PV guests that use PV mmu_ops.
Initially I am thinking of supporting 64-bit PV guests. I think only RW, RX and
R permissions would be possible. I am looking at the hypercall interfaces that
do the pagetable manipulations. In addition to the hypercalls I am also looking
at the pagetable write emulation code.

My plan is to set the page permissions in:
do_mmu_update()
do_mmuext_op(MMUEXT_PIN_L1_TABLE)
__do_update_va_mapping()
ptwr_emulated_update()

I would then hook in to do_page_fault() to perform the access check and send the
request to the mem_event handler. Please let me know if I am on the right track
and if this is a sane approach?

Questions...
I am not sure where to store the p2m_access value in the PTE. EPT uses bits
61:58. The spec says they are ignored for regular pagetables, so is it safe to
use these bits? If not, do you have any suggestions about where I can stash
these bits?
The default functions for p2m->set_entry() and p2m->get_entry() default to
the shadow PT variants which I am assuming cannot be used in the PV cases and I
would need to do something along the lines of mod_l1_entry(). Is this correct?

Thanks,
Aravindh



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

At 18:38 +0000 on 17 Oct (1382031502), Aravindh Puthiyaparambil (aravindp)
wrote:
> I am looking in to implementing mem_event for PV guests that use PV
mmu_ops. Initially I am thinking of supporting 64-bit PV guests. I think only
RW, RX and R permissions would be possible. I am looking at the hypercall
interfaces that do the pagetable manipulations. In addition to the hypercalls I
am also looking at the pagetable write emulation code.
> 
> My plan is to set the page permissions in:
> do_mmu_update()
> do_mmuext_op(MMUEXT_PIN_L1_TABLE)
> __do_update_va_mapping()
> ptwr_emulated_update()
> 
I''m afraid you can''t do that, and still support the PV API as
it
stands.  The guest had read access to its pagetables, and it does not
expect Xen to change them underfoot.  E.g. if you temporarily mark a
page as inaccessible and the guest walks its own pagetables to
diagnose a page fault, it could see your change and send SIGSEGV to
the current process.

So if you want to have permissions in the pagetables that aren''t the
ones the guest asked for, your choices are:

 (a) use shadow pagetables.  This is what the live migration code
     does in order to intercept writes for tracking memory dirtying.
     Slow, but should work for most things.

 (b) Wait for the PVH patch series to go in (and the equivalent
     series in Linux and FreeBSD kernels).  PVH guests are enough
     like HVM ones in the Xen code that mem_events will be much easier.

(b) is probably much eaier, if you have the choice.
> The default functions for p2m->set_entry() and p2m->get_entry()
> default to the shadow PT variants which I am assuming cannot be used
> in the PV cases and I would need to do something along the lines of
> mod_l1_entry(). Is this correct?
PV guests don''t have a p2m in Xen -- they put MFNs directly in their
pagetabels and Xen only checks the pagetables for safety.  The guest
itself will probably have an internal p2m for bookkeeping (and for
save/restore) but again Xen isn''t allowed to write to it.

Cheers,

Tim.

> I am looking in to implementing mem_event for PV guests that use PV
mmu_ops. Initially I am thinking of supporting 64-bit PV guests. I think only
RW, RX and R permissions would be possible. I am looking at the hypercall
interfaces that do the pagetable manipulations. In addition to the hypercalls I
am also looking at the pagetable write emulation code.
I''m assuming mem_access here, rather than generic mem_event…
> 
> My plan is to set the page permissions in:
> do_mmu_update()
> do_mmuext_op(MMUEXT_PIN_L1_TABLE)
> __do_update_va_mapping()
> ptwr_emulated_update()
> 
> I would then hook in to do_page_fault() to perform the access check and
send the request to the mem_event handler. Please let me know if I am on the
right track and if this is a sane approach?
> 
> Questions...
> I am not sure where to store the p2m_access value in the PTE. EPT uses bits
61:58. The spec says they are ignored for regular pagetables, so is it safe to
use these bits? If not, do you have any suggestions about where I can stash
these bits?
Depends on the PV guest. If using auto-translated physmap (i.e. PVH but not for
dom0), then you will have a proper EPT. You can probably just rely on the same
men-event implementation that exists for HVMs (hand-waving though).

On a traditional PV, you will need to turn on shadow. I don''t think you
will get to where you want by changing the PT entries the guest sees. Or you
will, but more painfully (hand waving, redux)

Once you turn on shadows, that uses the pt implementation of the p2m
(arch/x86/mm/p2m-pt.c), which AMD also uses. There is room in there for mem
access bits. From a previous thread on xen-devel the IOMMU bits can be
sacrificed to host mem access bits, and the hypervisor already implements XOR
for IOMMU and "fancy memory stuff"

HTH
Andres
> The default functions for p2m->set_entry() and p2m->get_entry()
default to the shadow PT variants which I am assuming cannot be used in the PV
cases and I would need to do something along the lines of mod_l1_entry(). Is
this correct?
> 
> Thanks,
> Aravindh

> > I am looking in to implementing mem_event for PV guests that use PV
> mmu_ops. Initially I am thinking of supporting 64-bit PV guests. I think
only
> RW, RX and R permissions would be possible. I am looking at the hypercall
> interfaces that do the pagetable manipulations. In addition to the
hypercalls I
> am also looking at the pagetable write emulation code.
> >
> > My plan is to set the page permissions in:
> > do_mmu_update()
> > do_mmuext_op(MMUEXT_PIN_L1_TABLE)
> > __do_update_va_mapping()
> > ptwr_emulated_update()
> >
> 
> I''m afraid you can''t do that, and still support the PV
API as it
> stands.  The guest had read access to its pagetables, and it does not
> expect Xen to change them underfoot.  E.g. if you temporarily mark a
> page as inaccessible and the guest walks its own pagetables to
> diagnose a page fault, it could see your change and send SIGSEGV to
> the current process.
> 
> So if you want to have permissions in the pagetables that aren''t
the
> ones the guest asked for, your choices are:
> 
>  (a) use shadow pagetables.  This is what the live migration code
>      does in order to intercept writes for tracking memory dirtying.
>      Slow, but should work for most things.
> 
>  (b) Wait for the PVH patch series to go in (and the equivalent
>      series in Linux and FreeBSD kernels).  PVH guests are enough
>      like HVM ones in the Xen code that mem_events will be much easier.
> 
> (b) is probably much eaier, if you have the choice.
I don''t have the choice of waiting for (b) so I guess I would have to
go down the harder (a) route. Let me explore that and get back to you with my
thoughts.

Thanks,
Aravindh
> > The default functions for p2m->set_entry() and p2m->get_entry()
> > default to the shadow PT variants which I am assuming cannot be used
> > in the PV cases and I would need to do something along the lines of
> > mod_l1_entry(). Is this correct?
> 
> PV guests don''t have a p2m in Xen -- they put MFNs directly in
their
> pagetabels and Xen only checks the pagetables for safety.  The guest
> itself will probably have an internal p2m for bookkeeping (and for
> save/restore) but again Xen isn''t allowed to write to it.
> 
> Cheers,
> 
> Tim.

> > I am looking in to implementing mem_event for PV guests that use PV
> mmu_ops. Initially I am thinking of supporting 64-bit PV guests. I think
only
> RW, RX and R permissions would be possible. I am looking at the hypercall
> interfaces that do the pagetable manipulations. In addition to the
hypercalls I
> am also looking at the pagetable write emulation code.
> I''m assuming mem_access here, rather than generic mem_event...
Ah yes, I should have said mem_access rather than mem_event :-)

 > >
> > My plan is to set the page permissions in:
> > do_mmu_update()
> > do_mmuext_op(MMUEXT_PIN_L1_TABLE)
> > __do_update_va_mapping()
> > ptwr_emulated_update()
> >
> > I would then hook in to do_page_fault() to perform the access check
and
> send the request to the mem_event handler. Please let me know if I am on
> the right track and if this is a sane approach?
> >
> > Questions...
> > I am not sure where to store the p2m_access value in the PTE. EPT uses
bits
> 61:58. The spec says they are ignored for regular pagetables, so is it safe
to
> use these bits? If not, do you have any suggestions about where I can stash
> these bits?
> Depends on the PV guest. If using auto-translated physmap (i.e. PVH but not
> for dom0), then you will have a proper EPT. You can probably just rely on
the
> same men-event implementation that exists for HVMs (hand-waving
> though).
> 
> On a traditional PV, you will need to turn on shadow. I don''t
think you will get
> to where you want by changing the PT entries the guest sees. Or you will,
but
> more painfully (hand waving, redux)
> 
> Once you turn on shadows, that uses the pt implementation of the p2m
> (arch/x86/mm/p2m-pt.c), which AMD also uses. There is room in there for
> mem access bits. From a previous thread on xen-devel the IOMMU bits can
> be sacrificed to host mem access bits, and the hypervisor already
implements
> XOR for IOMMU and "fancy memory stuff"
That was helpful. Looks like I would have to go down the PV + shadow path if I
want this working for PV guests.

Thanks,
Aravindh

> -----Original Message-----
> From: Aravindh Puthiyaparambil (aravindp)
> Sent: Thursday, October 17, 2013 12:00 PM
> To: ''Tim Deegan''
> Cc: xen-devel@lists.xenproject.org
> Subject: RE: Questions regarding implementing mem_event for PV
> 
> > > I am looking in to implementing mem_event for PV guests that use
PV
> > mmu_ops. Initially I am thinking of supporting 64-bit PV guests. I
think only
> > RW, RX and R permissions would be possible. I am looking at the
hypercall
> > interfaces that do the pagetable manipulations. In addition to the
> hypercalls I
> > am also looking at the pagetable write emulation code.
> > >
> > > My plan is to set the page permissions in:
> > > do_mmu_update()
> > > do_mmuext_op(MMUEXT_PIN_L1_TABLE)
> > > __do_update_va_mapping()
> > > ptwr_emulated_update()
> > >
> >
> > I''m afraid you can''t do that, and still support the
PV API as it
> > stands.  The guest had read access to its pagetables, and it does not
> > expect Xen to change them underfoot.  E.g. if you temporarily mark a
> > page as inaccessible and the guest walks its own pagetables to
> > diagnose a page fault, it could see your change and send SIGSEGV to
> > the current process.
> >
> > So if you want to have permissions in the pagetables that
aren''t the
> > ones the guest asked for, your choices are:
> >
> >  (a) use shadow pagetables.  This is what the live migration code
> >      does in order to intercept writes for tracking memory dirtying.
> >      Slow, but should work for most things.
> >
> >  (b) Wait for the PVH patch series to go in (and the equivalent
> >      series in Linux and FreeBSD kernels).  PVH guests are enough
> >      like HVM ones in the Xen code that mem_events will be much
easier.
> >
> > (b) is probably much eaier, if you have the choice.
> 
> I don''t have the choice of waiting for (b) so I guess I would have
to go down
> the harder (a) route. Let me explore that and get back to you with my
> thoughts.
Is there an xl config file option to bring up a PV guest with shadow? Or do I
have to call xc_shadow_control() from the mem_event handler to enable shadow
paging for a PV guest?

Thanks,
Aravindh

On Thu, Oct 17, 2013 at 7:59 PM, Aravindh Puthiyaparambil (aravindp)
<aravindp@cisco.com> wrote:
>> > I am looking in to implementing mem_event for PV guests that use
PV
>> mmu_ops. Initially I am thinking of supporting 64-bit PV guests. I
think only
>> RW, RX and R permissions would be possible. I am looking at the
hypercall
>> interfaces that do the pagetable manipulations. In addition to the
hypercalls I
>> am also looking at the pagetable write emulation code.
>> >
>> > My plan is to set the page permissions in:
>> > do_mmu_update()
>> > do_mmuext_op(MMUEXT_PIN_L1_TABLE)
>> > __do_update_va_mapping()
>> > ptwr_emulated_update()
>> >
>>
>> I''m afraid you can''t do that, and still support the
PV API as it
>> stands.  The guest had read access to its pagetables, and it does not
>> expect Xen to change them underfoot.  E.g. if you temporarily mark a
>> page as inaccessible and the guest walks its own pagetables to
>> diagnose a page fault, it could see your change and send SIGSEGV to
>> the current process.
>>
>> So if you want to have permissions in the pagetables that
aren''t the
>> ones the guest asked for, your choices are:
>>
>>  (a) use shadow pagetables.  This is what the live migration code
>>      does in order to intercept writes for tracking memory dirtying.
>>      Slow, but should work for most things.
>>
>>  (b) Wait for the PVH patch series to go in (and the equivalent
>>      series in Linux and FreeBSD kernels).  PVH guests are enough
>>      like HVM ones in the Xen code that mem_events will be much easier.
>>
>> (b) is probably much eaier, if you have the choice.
>
> I don''t have the choice of waiting for (b) so I guess I would have
to go down the harder (a) route. Let me explore that and get back to you with my
thoughts.
FWIW, I think a "tech preview" version should go in fairly soon.  At
the developer meeting we agreed that it would be best to check in what
we have once I get the bugs worked out, so that people can start
working on additional functionality (like the mem events) in parallel.
 I''ve just fixed up all the known bugs, so I should be posting v14
later today, if all goes well.

 -George

> On Thu, Oct 17, 2013 at 7:59 PM, Aravindh Puthiyaparambil (aravindp)
> <aravindp@cisco.com> wrote:
> >> > I am looking in to implementing mem_event for PV guests that
use PV
> >> mmu_ops. Initially I am thinking of supporting 64-bit PV guests. I
think
> only
> >> RW, RX and R permissions would be possible. I am looking at the
hypercall
> >> interfaces that do the pagetable manipulations. In addition to the
> hypercalls I
> >> am also looking at the pagetable write emulation code.
> >> >
> >> > My plan is to set the page permissions in:
> >> > do_mmu_update()
> >> > do_mmuext_op(MMUEXT_PIN_L1_TABLE)
> >> > __do_update_va_mapping()
> >> > ptwr_emulated_update()
> >> >
> >>
> >> I''m afraid you can''t do that, and still support
the PV API as it
> >> stands.  The guest had read access to its pagetables, and it does
not
> >> expect Xen to change them underfoot.  E.g. if you temporarily mark
a
> >> page as inaccessible and the guest walks its own pagetables to
> >> diagnose a page fault, it could see your change and send SIGSEGV
to
> >> the current process.
> >>
> >> So if you want to have permissions in the pagetables that
aren''t the
> >> ones the guest asked for, your choices are:
> >>
> >>  (a) use shadow pagetables.  This is what the live migration code
> >>      does in order to intercept writes for tracking memory
dirtying.
> >>      Slow, but should work for most things.
> >>
> >>  (b) Wait for the PVH patch series to go in (and the equivalent
> >>      series in Linux and FreeBSD kernels).  PVH guests are enough
> >>      like HVM ones in the Xen code that mem_events will be much
easier.
> >>
> >> (b) is probably much eaier, if you have the choice.
> >
> > I don''t have the choice of waiting for (b) so I guess I would
have to go down
> the harder (a) route. Let me explore that and get back to you with my
> thoughts.
> 
> FWIW, I think a "tech preview" version should go in fairly soon. 
At
> the developer meeting we agreed that it would be best to check in what
> we have once I get the bugs worked out, so that people can start
> working on additional functionality (like the mem events) in parallel.
>  I''ve just fixed up all the known bugs, so I should be posting v14
> later today, if all goes well.
I will definitely implement mem_access for PVH, however the platform I am
targeting in the short term does not have EPT so I will have to go the shadow
route.

Thanks,
Aravindh