Xen devel - Mar 2009 - Re: how to enable shadow page table? Do I have to run HVM guest systems for shadow paging mode?

Long Wang wrote:
> I am working on a research project based on Xen 3.3.1. I want to use the
> shadow page table for setting guest system memory pages as read-only and/or
> dirty, and perform a copy-on-write mechanism when these memory pages are
> updated. I heard that auto_translated_physmap is not supported any more. 
>   
Yes, auto_translated_physmap is not supported by Xen or by the pvops 
guest kernels.
> Then how can I enable shadow page table? Do I have to run HVM guest systems
> to have shadow paging mode? Can I enable shadow paging mode in PV?
>   
Well, Xen supports log-dirty mode for doing live migration of PV guests, 
which looks to be what you''re describing above.  You might want to look
into the existing live migration machinery to see if it meets your needs.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 16:30 +0000 on 16 Mar (1237221012), Jeremy Fitzhardinge
wrote:
> Long Wang wrote:
> > I am working on a research project based on Xen 3.3.1. I want to use
the
> > shadow page table for setting guest system memory pages as read-only
and/or
> > dirty, and perform a copy-on-write mechanism when these memory pages
are
> > updated.
Several people are already working on copy-on-write memory in Xen, or
variations on that theme (Mike Sun, Patrick Colp, John Byrne, and
others) as well as more exotic things like Difference Engine and
Transcendent Memory.  It would probably be better to talk to those
people and try to help their efforts than start from scratch.

Patrick Colp''s slides from the most recent Xen Summit give an idea of
one method of implementing it:
http://www.xen.org/files/xensummit_oracle09/VMSnapshots.pdf

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

For the record, Transcendent Memory is not a copy-on-write
mechanism, doesn''t deal with shadow tables or PTE read/write
bits, and is really not exotic at all.  It''s a simple (though
paravirtualized) mechanism for optimizing the usage of
unutilized or underutilized physical memory while ensuring
(most of) that memory is synchronously reclaimable for
unexpectedly urgent needs.  :-)

Dan
> -----Original Message-----
> From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
> Sent: Monday, March 16, 2009 11:41 AM
> To: Jeremy Fitzhardinge
> Cc: Xen-devel; Long Wang
> Subject: [Xen-devel] Re: how to enable shadow page table? Do I have to
> run HVM guest systems for shadow paging mode?
> 
> 
> At 16:30 +0000 on 16 Mar (1237221012), Jeremy Fitzhardinge wrote:
> > Long Wang wrote:
> > > I am working on a research project based on Xen 3.3.1. I 
> want to use the
> > > shadow page table for setting guest system memory pages 
> as read-only and/or
> > > dirty, and perform a copy-on-write mechanism when these 
> memory pages are
> > > updated.
> 
> Several people are already working on copy-on-write memory in Xen, or
> variations on that theme (Mike Sun, Patrick Colp, John Byrne, and
> others) as well as more exotic things like Difference Engine and
> Transcendent Memory.  It would probably be better to talk to those
> people and try to help their efforts than start from scratch.
> 
> Patrick Colp''s slides from the most recent Xen Summit give an idea
of
> one method of implementing it:
> http://www.xen.org/files/xensummit_oracle09/VMSnapshots.pdf
> 
> Cheers,
> 
> Tim.
> 
> -- 
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, Citrix Systems (R&D) Ltd.
> [Company #02300071, SL9 0DZ, UK.]
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 17:57 +0000 on 16 Mar (1237226223), Dan Magenheimer
wrote:
> For the record, Transcendent Memory is not a copy-on-write
> mechanism, doesn''t deal with shadow tables or PTE read/write
> bits, and is really not exotic at all.
Indeed.  I was just including it in the current bundle of "fun things
to do with memory management". :)  And since it helps with systems under
memory pressure it might have solved whatever the underlying problem was.
Almost nobody wants to write a copy-on-write mechanism just for the
beauty of the thing.

Cheers,

Tim.
>  It''s a simple (though
> paravirtualized) mechanism for optimizing the usage of
> unutilized or underutilized physical memory while ensuring
> (most of) that memory is synchronously reclaimable for
> unexpectedly urgent needs.  :-)
> 
> Dan
> 
> > -----Original Message-----
> > From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
> > Sent: Monday, March 16, 2009 11:41 AM
> > To: Jeremy Fitzhardinge
> > Cc: Xen-devel; Long Wang
> > Subject: [Xen-devel] Re: how to enable shadow page table? Do I have to
> > run HVM guest systems for shadow paging mode?
> > 
> > 
> > At 16:30 +0000 on 16 Mar (1237221012), Jeremy Fitzhardinge wrote:
> > > Long Wang wrote:
> > > > I am working on a research project based on Xen 3.3.1. I 
> > want to use the
> > > > shadow page table for setting guest system memory pages 
> > as read-only and/or
> > > > dirty, and perform a copy-on-write mechanism when these 
> > memory pages are
> > > > updated.
> > 
> > Several people are already working on copy-on-write memory in Xen, or
> > variations on that theme (Mike Sun, Patrick Colp, John Byrne, and
> > others) as well as more exotic things like Difference Engine and
> > Transcendent Memory.  It would probably be better to talk to those
> > people and try to help their efforts than start from scratch.
> > 
> > Patrick Colp''s slides from the most recent Xen Summit give an
idea of
> > one method of implementing it:
> > http://www.xen.org/files/xensummit_oracle09/VMSnapshots.pdf
> > 
> > Cheers,
> > 
> > Tim.
> > 
> > -- 
> > Tim Deegan <Tim.Deegan@citrix.com>
> > Principal Software Engineer, Citrix Systems (R&D) Ltd.
> > [Company #02300071, SL9 0DZ, UK.]
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >
-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 18:40 +0000 on 16 Mar (1237228854), longwang@crhc.illinois.edu
wrote:
> 2) When I call paging_mode_log_dirty(pdom) in hypervisor, the hypervisor
> crashes. I thought paging_mode_log_dirty() enables the shadow paging as
> well as dirty logging after I had read live migration code. But I was
> wrong. paging_mode_log_dirty() crashes because shadow paging is not
> enabled yet.
> 
> Now I need to enable shadow paging mode in my testbed (PV). 
You wrote all the code before figuring out how to turn on shadow pagetables?
> I tried the
> auto_translated_physmap in the domU config file. But "xm create"
failed
> with :
> Error: (1, ''Internal error'',
''xc_dom_boot_domU_map: failed to mmap domU
> pages 0x100+0x2da [mmap, errno=14 (Bad address)]\n'')
As Jeremy says, auto_translated_physmap doesn''t work on recent Xens or
recent Linuxes.  
> So how can I enable shadow paging? Do I have to run HVM for this to work?
If you want the shadow code to hide a layer of address translations
(i.e. have gfn != mfn) then yes, you need HVM.  

Otherwise, you can just copy what the live-migration tools do to turn on
shadow pagetables for PV guests.  It''s just one hypercall.  For
example,
in python:

 #!/usr/bin/env python
 import sys
 import xen.lowlevel.xc
 xen.lowlevel.xc.xc().shadow_control(dom=int(sys.argv[1]), op=1)

To turn on log-dirty mode, use op=2.  To turn off shadow mode, op=0.

Cheers,

Tim.
> thanks,
> long
> 
> 
> > At 16:30 +0000 on 16 Mar (1237221012), Jeremy Fitzhardinge wrote:
> >> Long Wang wrote:
> >> > I am working on a research project based on Xen 3.3.1. I want
to use
> >> the
> >> > shadow page table for setting guest system memory pages as
read-only
> >> and/or
> >> > dirty, and perform a copy-on-write mechanism when these
memory pages
> >> are
> >> > updated.
> >
> > Several people are already working on copy-on-write memory in Xen, or
> > variations on that theme (Mike Sun, Patrick Colp, John Byrne, and
> > others) as well as more exotic things like Difference Engine and
> > Transcendent Memory.  It would probably be better to talk to those
> > people and try to help their efforts than start from scratch.
> >
> > Patrick Colp''s slides from the most recent Xen Summit give an
idea of
> > one method of implementing it:
> > http://www.xen.org/files/xensummit_oracle09/VMSnapshots.pdf
> >
> > Cheers,
> >
> > Tim.
> >
> > --
> > Tim Deegan <Tim.Deegan@citrix.com>
> > Principal Software Engineer, Citrix Systems (R&D) Ltd.
> > [Company #02300071, SL9 0DZ, UK.]
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >
> 
> 
-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

At 05:03 +0000 on 19 Mar (1237439004), Long Wang wrote:
> Here is another question. In the log-dirty mode, who sets the
"dirty" bit of
> a level-1 entry?
Depends what you mean.  The dirty bit of a *guest* PTE is set either
directly by the guest or by guest_walk_tables().

The dirty bit of a *shadow* PTE is usually set by _sh_propagate (since
we don''t care about it) except for some optimizations to do with HVM
video RAM mappings, where it''s set by the MMU.

The dirty bit in the log-dirty bitmap is set by paging_mark_dirty(),
which is called from anywhere that writes to guest memory or allows the
guest a writeable mapping.  It has nothing to do with the dirty bits in
PTEs.
> The mmu, or _sh_propagate()? As far as I know about the x86
> hardware, the dirty bit is independent of the R/W bit. But it seems that
> _sh_propagate() sets all pages as read-only and then sets the dirty bit
when
> handling the page fault. 
In order to get the *guest* PTE''s dirty bit set correctly, we leave the
*shadow* PTE read-only in order to intercept the guest''s first write to
the page.  At that point guest_walk_tables() sets the guest-PTE dirty
bit (as an MMU would if shadows were not in use) and _sh_propagate()
makes the shadow PTE R/W.
> Will the mmu sets the dirty bit for a page when the page is written?
Only for the *shadow* PTE, since that''s the only one the MMU ever
sees. 

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

At 15:55 +0000 on 19 Mar (1237478115), Long Wang wrote:
> I am a little confused. You wrote: 
> 1) "The dirty bit of a *shadow* PTE is usually set by _sh_propagate
(since
> we don''t care about it) except for some optimizations to do with
HVM video
> RAM mappings, where it''s set by the MMU." 
> 
> 2) "Only for the *shadow* PTE, since that''s the only one the
MMU ever
> Sees". 
> 
> Do you mean that most of *shadow* PTE dirty bits (in PV) are set by
> _sh_propagate() (through a read-only-bit-triggered interception)?
No, I mean that when _sh_propagate() makes a shadow PTE that has the R/W
bit set, it usually sets the Dirty bit as well (to save the MMU the
extra write cycle later).

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 16:40 +0000 on 19 Mar (1237480807), Long Wang wrote:
> So most of shadow PTE dirty bits are still set by the MMU when the MMU
finds
> there is a write to a page and the PTE dirty bit is cleared?
The MMU will indeed do this if it finds this is the case.  However as I
said in almost all shadow PTEs we keep the Dirty bit set all the time so
the MMU has nothing to do.
> Is this
> behavior always performed, or shall I turn on a switch for the MMU to set
> the dirty bit?
Always-on.  The MMU''s behaviour is described in the Intel SDMs vol 3
sections 3.6 to 3.12, or the AMD APMs vol 2 chapter 5.

And, again, this has nothing to do with the dirty-page bitmap maintained
by log-dirty mode.

Cheers,

Tim.
> Thanks.
> 
> long
> 
> -----Original Message-----
> From: Tim Deegan [mailto:Tim.Deegan@citrix.com] 
> Sent: Thursday, March 19, 2009 11:16 AM
> To: Long Wang
> Cc: ''Jeremy Fitzhardinge''; ''Xen-devel''
> Subject: Re: [Xen-devel] Re: how to enable shadow page table? Do Ihavetorun
> HVM guest systems for shadow paging mode?
> 
> Hi,
> 
> At 15:55 +0000 on 19 Mar (1237478115), Long Wang wrote:
> > I am a little confused. You wrote: 
> > 1) "The dirty bit of a *shadow* PTE is usually set by
_sh_propagate (since
> > we don''t care about it) except for some optimizations to do
with HVM video
> > RAM mappings, where it''s set by the MMU." 
> > 
> > 2) "Only for the *shadow* PTE, since that''s the only one
the MMU ever
> > Sees". 
> > 
> > Do you mean that most of *shadow* PTE dirty bits (in PV) are set by
> > _sh_propagate() (through a read-only-bit-triggered interception)?
> 
> No, I mean that when _sh_propagate() makes a shadow PTE that has the R/W
> bit set, it usually sets the Dirty bit as well (to save the MMU the
> extra write cycle later).
> 
> Cheers,
> 
> Tim.
> 
> -- 
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, Citrix Systems (R&D) Ltd.
> [Company #02300071, SL9 0DZ, UK.]
> 
-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel