Hi, Can someone please tell me how is Shadow Page Table generated/maintained? What is the flow for the page fault handler of VMM? If possible, please tell me the flow related to memory management in XEN. Regards, Sameer _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 11/6/06, Sameer Ahuja <sameer.ahuja@nechclst.in> wrote:> > Hi, > If possible, please tell me the flow related to memory management in XEN. > >Although the website seems to be down right now, the student project here: http://www.cs.aau.dk/library/cgi-bin/detail.cgi?id=1136884892 actually has a pretty good explanation of xen memory management. It will get you started before looking at the code at least. -Tim _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Tim, The link you posted does not work. Do you have a fixed link? Thanks, Liang ----- Original Message ----- From: "Tim Wood" <twwood@gmail.com> To: "Sameer Ahuja" <sameer.ahuja@nechclst.in> Cc: <xen-devel@lists.xensource.com> Sent: Monday, November 06, 2006 8:36 PM Subject: Re: [Xen-devel] Need information related to shadowing> On 11/6/06, Sameer Ahuja <sameer.ahuja@nechclst.in> wrote: >> >> Hi, >> If possible, please tell me the flow related to memory management in XEN. >> >> > > Although the website seems to be down right now, the student project > here: http://www.cs.aau.dk/library/cgi-bin/detail.cgi?id=1136884892 > actually has a pretty good explanation of xen memory management. It > will get you started before looking at the code at least. > > -Tim > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
(forgot to reply-all) On 11/7/06, Liang Yang <multisyncfe991@hotmail.com> wrote:> Hi Tim, > > The link you posted does not work. Do you have a fixed link? > > Thanks, > > Liang >It''s not the best way to read it, but google seems to have an html version cached: http://72.14.209.104/search?q=cache:VS3Scgtdi7sJ:www.cs.aau.dk/library/files/rapbibfiles1/1136884892.pdf+%22efficient+memory+sharing+in+the+xen+virtual+machine+monitor%22&hl=en&gl=us&ct=clnk&cd=1&client=firefox-a The second part of their project also has a bit on shadow page tables: http://www.cs.aau.dk/library/files/rapbibfiles1/1150283144.pdf (currently dead) http://72.14.209.104/search?q=cache:fzWHQLFjP4gJ:www.cs.aau.dk/library/files/rapbibfiles1/1150283144.pdf+%22Jacob+Faber+Kloster%22&hl=en&gl=us&ct=clnk&cd=3&client=firefox-a I suspect they will fix their system and the full pdfs will come back online soon. The Potemkin paper also has a section on memory mgmt/shadow page tables: http://www.cs.ucsd.edu/~savage/papers/Sosp05.pdf The xen wiki has a little bit: http://wiki.xensource.com/xenwiki/XenMemoryManagement Might be more info here http://www.cs.dartmouth.edu/~nihal/research_summer06_xen-mem.html ? And maybe something of use here: http://www.cs.wisc.edu/~remzi/Classes/736/Spring2005/Projects/Muru-Selva/cs736-report.pdf Note that I''m not the author of any of these and cannot really vouch for their accuracy. Also, they may be about older versions of xen -- their was recently a major update to the shadow page table code, so that may not be reflected in these. However, they will at least get you started. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Tim, Thanks a lot. It''s really going to help me a lot. Regards, Sameer -----Original Message----- From: Tim Wood [mailto:twwood@gmail.com] Sent: Tuesday, November 07, 2006 8:42 PM To: xen-devel@lists.xensource.com Cc: Sameer Ahuja Subject: Re: [Xen-devel] Need information related to shadowing (forgot to reply-all) On 11/7/06, Liang Yang <multisyncfe991@hotmail.com> wrote:> Hi Tim, > > The link you posted does not work. Do you have a fixed link? > > Thanks, > > Liang >It''s not the best way to read it, but google seems to have an html version cached: http://72.14.209.104/search?q=cache:VS3Scgtdi7sJ:www.cs.aau.dk/library/f iles/rapbibfiles1/1136884892.pdf+%22efficient+memory+sharing+in+the+xen+ virtual+machine+monitor%22&hl=en&gl=us&ct=clnk&cd=1&client=firefox-a The second part of their project also has a bit on shadow page tables: http://www.cs.aau.dk/library/files/rapbibfiles1/1150283144.pdf (currently dead) http://72.14.209.104/search?q=cache:fzWHQLFjP4gJ:www.cs.aau.dk/library/f iles/rapbibfiles1/1150283144.pdf+%22Jacob+Faber+Kloster%22&hl=en&gl=us&c t=clnk&cd=3&client=firefox-a I suspect they will fix their system and the full pdfs will come back online soon. The Potemkin paper also has a section on memory mgmt/shadow page tables: http://www.cs.ucsd.edu/~savage/papers/Sosp05.pdf The xen wiki has a little bit: http://wiki.xensource.com/xenwiki/XenMemoryManagement Might be more info here http://www.cs.dartmouth.edu/~nihal/research_summer06_xen-mem.html ? And maybe something of use here: http://www.cs.wisc.edu/~remzi/Classes/736/Spring2005/Projects/Muru-Selva /cs736-report.pdf Note that I''m not the author of any of these and cannot really vouch for their accuracy. Also, they may be about older versions of xen -- their was recently a major update to the shadow page table code, so that may not be reflected in these. However, they will at least get you started. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 11/8/06, Sameer Ahuja <sameer.ahuja@nechclst.in> wrote:> > Hi Tim, > > Thanks a lot. > It''s really going to help me a lot. >Happy to help. And of course for browsing the actual code, this is very helpful: http://lxr.xensource.com/lxr/source _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi, Thanks a lot to all of you for your previous replies. That helped me a lot in understanding the code. I still have some queries. Can you please tell me how is Shadow Page Table generated/maintained? What is the flow for the page fault handler of VMM in context of x86_64 (Intel EM64T)? If possible, please tell me the code flow related to shadow page table handling in XEN. I have gone through the links you sent earlier as well as some of the code files especially through following: xen\arch\x86\mm\shadow\multi.c xen\arch\x86\mm\shadow\common.c xen\arch\x86\mm\mm.c But I have not been able to establish the code flow based on Intel EM64T. Regards, Sameer -----Original Message----- From: Tim Wood [mailto:twwood@gmail.com] Sent: Tuesday, November 07, 2006 9:06 AM To: Sameer Ahuja Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] Need information related to shadowing On 11/6/06, Sameer Ahuja <sameer.ahuja@nechclst.in> wrote:> > Hi, > If possible, please tell me the flow related to memory management inXEN.> >Although the website seems to be down right now, the student project here: http://www.cs.aau.dk/library/cgi-bin/detail.cgi?id=1136884892 actually has a pretty good explanation of xen memory management. It will get you started before looking at the code at least. -Tim _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
At 09:41 +0530 on 15 Nov (1163583694), Sameer Ahuja wrote:> Can you please tell me how is Shadow Page Table generated/maintained?Shadow pagetables are generated in two places in the shadow code: on a guest CR3 change a top-level shadow is generated (multi.c: sh_update_cr3() calls sh_set_toplevel_shadow()), and all other shadows are generated in the page fault handler (sh_page_fault() calls shadow_get_and_create_l1e(), which recursively builds the shadow tables). Shadow entries are filled in by the l*e_propagate_from_guest() functions, which are called directly from the page fault handler and when we see a write to a guest pagetable (via the shadow_validate_guest_entry() and shadow_validate_guest_pt_write() functions). The control flow is a bit tricky there because we need to track shadows of different paging modes at the same time: a single page can have up to eight different shadows. To deal with different paging modes, the file multi.c is compiled multiple times, and its functions renamed to include the paging mode they handle. We can then call the correct function by name (see the various dispatch tables in common.c), or call the functions for the paging mode each vcpu is currently in, via the v->arch.shadow.mode array of pointers. Shadows are destroyed when their reference count hits zero, typically because shadow memory is being reclaimed: shadow_prealloc() un-pins top-level shadows which causes them to recursively destroy all their children. Cheers, Tim. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks a lot Tim. Your reply was useful but I have one more query - how this handling related to shadow tables is different between IA 32(x86_32) and EM64T (x86_64)? The thing is that with EM64T Intel Virtualization Technology comes into the picture and I think that it shifts a lot of responsibilities from Xen to the H/W level. (This is what I have understood after going through a couple of articles on Intel site and other sites) I have not been able to find out this in the Xen code though I can clearly see that the directories are different for x86_32 and x86_64. Please send me any article or paper related to this. Regards, Sameer -----Original Message----- From: Tim Deegan [mailto:Tim.Deegan@xensource.com] Sent: Wednesday, November 15, 2006 3:44 PM To: Sameer Ahuja Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] Need information related to shadowing At 09:41 +0530 on 15 Nov (1163583694), Sameer Ahuja wrote:> Can you please tell me how is Shadow Page Table generated/maintained?Shadow pagetables are generated in two places in the shadow code: on a guest CR3 change a top-level shadow is generated (multi.c: sh_update_cr3() calls sh_set_toplevel_shadow()), and all other shadows are generated in the page fault handler (sh_page_fault() calls shadow_get_and_create_l1e(), which recursively builds the shadow tables). Shadow entries are filled in by the l*e_propagate_from_guest() functions, which are called directly from the page fault handler and when we see a write to a guest pagetable (via the shadow_validate_guest_entry() and shadow_validate_guest_pt_write() functions). The control flow is a bit tricky there because we need to track shadows of different paging modes at the same time: a single page can have up to eight different shadows. To deal with different paging modes, the file multi.c is compiled multiple times, and its functions renamed to include the paging mode they handle. We can then call the correct function by name (see the various dispatch tables in common.c), or call the functions for the paging mode each vcpu is currently in, via the v->arch.shadow.mode array of pointers. Shadows are destroyed when their reference count hits zero, typically because shadow memory is being reclaimed: shadow_prealloc() un-pins top-level shadows which causes them to recursively destroy all their children. Cheers, Tim. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 15/11/06 12:09, "Sameer Ahuja" <sameer.ahuja@nechclst.in> wrote:> Your reply was useful but I have one more query - how this handling > related to shadow tables is different between IA 32(x86_32) and EM64T > (x86_64)? > > The thing is that with EM64T Intel Virtualization Technology comes into > the picture and I think that it shifts a lot of responsibilities from > Xen to the H/W level.VT capability is orthogonal to whether Xen is built for x86_32 or x86_64. There are 32-bit-only chips that support VT (e.g., the Core Duo chips) so it''s not limited to only EM64T chips running in 64-bit mode. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Petersson, Mats
2006-Nov-15 12:36 UTC
RE: [Xen-devel] Need information related to shadowing
> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of > Sameer Ahuja > Sent: 15 November 2006 12:10 > To: Tim Deegan; xen-devel@lists.xensource.com > Cc: sameer.ahuja81@gmail.com > Subject: RE: [Xen-devel] Need information related to shadowing > > > Thanks a lot Tim. > > Your reply was useful but I have one more query - how this handling > related to shadow tables is different between IA 32(x86_32) and EM64T > (x86_64)?Not much, except that x86_64 has a greater number of page-table levels, but everything else is identical.> > The thing is that with EM64T Intel Virtualization Technology > comes into > the picture and I think that it shifts a lot of responsibilities from > Xen to the H/W level. (This is what I have understood after going > through a couple of articles on Intel site and other sites)Not really - the major part of page-table management is exactly the same, it''s only where and how you get the catch or intercept for page-faults that differs, but the management of the shadow pages is still based on marking the actual page table entries that the guest has read-only, and maintaining a "behind the scenes" copy (shadow) page-table that actually reflect the REAL memory layout. The difference is that the VT (and AMD''s AMD-V/SVM) technology allows the VMM to intercept the page-fault before it hits the interrupt descriptor table (IDT), and thus it''s independent of the implementation of the guest-OS. In the para-virtual case, Xen handles the IDT, and catches the page-fault via the exception vector. Likewise, para-virtual Xen uses a "ring compression" method to catch reads/writes involving CR3, so that Xen can track the value of CR3 and maintain a "hidden" value of CR3 for the processor to use, that is different from what the guest-OS thinks CR3 is. (The guest-OS will attempt set CR3 to some value, but the intercept code in Xen will set the ACTUAL CR3 register to point to the shadow page-table, rather than the page-table maintained by the guest-OS). In HVM (VT/AMD-V), the CR3 intercept is handled by the virtualization technology and the relevant bits in virtual machine control structurure/block, so there''s no need to modify the guest-OS. But the actual process needed to be done when the CR3 is written with a new value is still the same. In the future, AMD and Intel will introduce extensions that allow a "second page-table", aka "Nested page tables", which essenitally allows the processor to automatically translate a page-table entry "twice" by having one set of page-tables for the guest-OS, and a second set of page-tables for the hypervisor. This way, we can simply tell the geust-OS that it''s got memory from 0..256MB, and just remap that in the HV to some other place(s) in the memory without the guest even knowing that this is happening.> > I have not been able to find out this in the Xen code though I can > clearly see that the directories are different for x86_32 and x86_64. > Please send me any article or paper related to this.You should be able to see how page-tables are dealt with in the em64t documentation from Intel, or in the AMD64 Architecture Programmer''s Manual, Volume 2, chapter 5. In case of how to handle shadow paging, there''s little difference between 2-level, 3-level (PAE) and 4-level page-tables, except for the number of times you need to do something and the slight differences between which bits can and can''t be set at any particular level of page-table. Shadow page-table management is of course complicated and non-trivial, but the idea behind it is independent of the number of levels used. -- Mats> > Regards, > Sameer > > -----Original Message----- > From: Tim Deegan [mailto:Tim.Deegan@xensource.com] > Sent: Wednesday, November 15, 2006 3:44 PM > To: Sameer Ahuja > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] Need information related to shadowing > > At 09:41 +0530 on 15 Nov (1163583694), Sameer Ahuja wrote: > > Can you please tell me how is Shadow Page Table > generated/maintained? > > Shadow pagetables are generated in two places in the shadow code: on a > guest CR3 change a top-level shadow is generated (multi.c: > sh_update_cr3() calls sh_set_toplevel_shadow()), and all other shadows > are > generated in the page fault handler (sh_page_fault() calls > shadow_get_and_create_l1e(), which recursively builds the shadow > tables). > > Shadow entries are filled in by the l*e_propagate_from_guest() > functions, which are called directly from the page fault handler and > when we see a write to a guest pagetable (via the > shadow_validate_guest_entry() and shadow_validate_guest_pt_write() > functions). > > The control flow is a bit tricky there because we need to > track shadows > of different paging modes at the same time: a single page can have up > to eight different shadows. To deal with different paging modes, the > file multi.c is compiled multiple times, and its functions renamed to > include the paging mode they handle. We can then call the correct > function by name (see the various dispatch tables in > common.c), or call > the functions for the paging mode each vcpu is currently in, via the > v->arch.shadow.mode array of pointers. > > Shadows are destroyed when their reference count hits zero, typically > because shadow memory is being reclaimed: shadow_prealloc() un-pins > top-level shadows which causes them to recursively destroy all their > children. > > Cheers, > > Tim. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi, DO you have any idea what auditing functions like sh_audit_l3_table() defined in x86\mm\shadow\multi.c do Regards, Sameer -----Original Message----- From: Tim Deegan [mailto:Tim.Deegan@xensource.com] Sent: Wednesday, November 15, 2006 3:44 PM To: Sameer Ahuja Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] Need information related to shadowing At 09:41 +0530 on 15 Nov (1163583694), Sameer Ahuja wrote:> Can you please tell me how is Shadow Page Table generated/maintained?Shadow pagetables are generated in two places in the shadow code: on a guest CR3 change a top-level shadow is generated (multi.c: sh_update_cr3() calls sh_set_toplevel_shadow()), and all other shadows are generated in the page fault handler (sh_page_fault() calls shadow_get_and_create_l1e(), which recursively builds the shadow tables). Shadow entries are filled in by the l*e_propagate_from_guest() functions, which are called directly from the page fault handler and when we see a write to a guest pagetable (via the shadow_validate_guest_entry() and shadow_validate_guest_pt_write() functions). The control flow is a bit tricky there because we need to track shadows of different paging modes at the same time: a single page can have up to eight different shadows. To deal with different paging modes, the file multi.c is compiled multiple times, and its functions renamed to include the paging mode they handle. We can then call the correct function by name (see the various dispatch tables in common.c), or call the functions for the paging mode each vcpu is currently in, via the v->arch.shadow.mode array of pointers. Shadows are destroyed when their reference count hits zero, typically because shadow memory is being reclaimed: shadow_prealloc() un-pins top-level shadows which causes them to recursively destroy all their children. Cheers, Tim. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel