I''m trying to understand some of the changes that were made to the shadow page table code a couple weeks ago to add support for 64-bit modes: changesets f261f14b9781f949f95a328433eff187c677a70b and 0bcfd66a431ebfc70fc068a134e684568ac02966, committed on July 11. I''m trying to make sense of what seems to be some code duplication. In xen/arch/x86, there are three files related to shadow mode: shadow.c, shadow32.c, and shadow_public.c. In my x86 build, only shadow32.c is built. shadow.c seems to contain a lot of similar code. The comments at the top of the files are also confusing; shadow32.c claims that it is shadow.c, and shadow.c claims that it is shadow_64.c. The Makefile is not very illuminating; for an x86_64 build, shadow32.o is not built (not surprising), otherwise, shadow.o, shadow_public.o, and shadow_xxx.o are skipped (the first two names sound rather generic to me, and shadow_xxx.o looks like part of a not-quite-finished change). Can anyone explain how this code is supposed to be structured, what is supposed to end up in each of the files, and whether the files are still in need of some renaming (say, shadow.c -> shadow64.c)? --Michael Vrable _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> I''m trying to make sense of what seems to be some code > duplication. In xen/arch/x86, there are three files related > to shadow mode: shadow.c, shadow32.c, and shadow_public.c. > In my x86 build, only shadow32.c is built. shadow.c seems to > contain a lot of similar code. The comments at the top of > the files are also confusing; shadow32.c claims that it is > shadow.c, and shadow.c claims that it is shadow_64.c.The new file called shadow.c can be compiled for both 32 and 64 bit, but is currently only used for 64bit by default. The shadow32.c file (formerly shadow.c) still exists because it''s had more testing than the new file, and we want both in the tree until we''re convinced there''s been no regression between the two. We''ll then delete shadow32 and compile shadow.c in 32 bit mode. Base any new development on the new shadow.c Make sense? Best, Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
"Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk> writes:> The shadow32.c file (formerly shadow.c) still exists because it''s had > more testing than the new file, and we want both in the tree until we''re > convinced there''s been no regression between the two. We''ll then delete > shadow32 and compile shadow.c in 32 bit mode. Base any new development > on the new shadow.cWhile we are at it and I''ll go work on shadow mode as well soon some followup questions ;) * "normal" shadow mode is almost identical to shadow mode off, i.e. the guest does machine<=>phys translations. The guest hasn''t to mark pages containing page tables r/o because xen can do that in the shadow tables. * In "translated" shadow mode the guest kernel handles a linear physical address space starting at addr zero and xen does the machine<=>phys translations when copying guest tables into the shadow page tables. * "external" is translated shadow mode + separated address space, i.e. no hypervisor window in the address space, used for vt. Correct so far? Enabling/switching shadow modes works by dropping all shadow tables, start with a zero-ed toplevel page directory (almost, except hypervisor window) and update the shadow pagetable tree as faults are coming in. Correct? But it''s not clear to me how the transition between "normal" and "translated" shadow mode works. Does that need support by the guest os? Gerd -- panic("it works"); /* avoid being flooded with debug messages */ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> While we are at it and I''ll go work on shadow mode as well > soon some followup questions ;) > > * "normal" shadow mode is almost identical to shadow mode off, > i.e. the guest does machine<=>phys translations. The guest hasn''t > to mark pages containing page tables r/o because xen can do that in > the shadow tables.Depends what you mean by "normal". The mode that we need for migration of a paravirtualized guest is log_dity but with refcounts in the *guest* pages i.e. SHM_refcounts is not set. The guest still has to mark pagetable pages RO and use the normal paravirtualized interface.> * In "translated" shadow mode the guest kernel handles a linear > physical address space starting at addr zero and xen does the > machine<=>phys translations when copying guest tables into the > shadow page tables.Yes. SHM_refcounts is typically set.> > * "external" is translated shadow mode + separated address space, > i.e. no hypervisor window in the address space, used for vt.Yes, this is used by VT-x.> Enabling/switching shadow modes works by dropping all shadow > tables, start with a zero-ed toplevel page directory (almost, > except hypervisor window) and update the shadow pagetable > tree as faults are coming in. Correct?Only in modes where SHM_refcounts is set. For so-called "lightweight shadow mode" (as used by migration) we continue using the guest refcounts.> But it''s not clear to me how the transition between "normal" > and "translated" shadow mode works. Does that need support > by the guest os?Guests typically don''t make this transition -- it''s currently a guest compile time option whether to use the paravirtualized or translated shadow mode interface. This isn''t strictly necessary. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi,> Depends what you mean by "normal". The mode that we need for migration > of a paravirtualized guest is log_dity but with refcounts in the *guest* > pages i.e. SHM_refcounts is not set.Ah, *that* the log_dirty option is good for. Looked like some performance measurement counter on a first quick look ...> > But it''s not clear to me how the transition between "normal" > > and "translated" shadow mode works. Does that need support > > by the guest os? > > Guests typically don''t make this transition -- it''s currently a guest > compile time option whether to use the paravirtualized or translated > shadow mode interface. This isn''t strictly necessary.Ok, more general: How does xen and/or the guest os handle the changes in machine<=>phys mapping as they happen on suspend/resume and migration because the guest gets a different set of machine pages? Especially the guest page table updates? Gerd -- panic("it works"); /* avoid being flooded with debug messages */ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Ok, more general: How does xen and/or the guest os handle > the changes in machine<=>phys mapping as they happen on > suspend/resume and migration because the guest gets a > different set of machine pages? Especially the guest page > table updates?The guest is notified that it should suspend itself. It should then get its self into a ''safe'' state i.e. ensure that no VCPUs are in the midst of pagetable updates [*], create a suspend record, and then call the suspend sched op hypercall. [upon resume, the sched op hypercall will appear to return] xc_linux_save is then responsible for reading the state out of the domain, converting any pages containing pagetables (as indiciated by the page type field) into ''canonical form'' i.e. changing mfns into pfns. At restore time, xc_linux_restore will rewrite them to the new mfns. The suspend record contains a page containing the list of mfns that make up the p2m table for the guest. For PAE mode we''ll need to extend this table to cover larger phys memory sizes as it''s only 1024 entries at present. Ian [*] We''re currently mising code to get other VCPUs to safety. We could hotunplug them, but its probably easiest just to have the VCPU that executes the suspend thread count the others into a tasklet. The save/restore code can trivially be extended to save multiple contexts. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel