This patch enables support of hugepages in a pv Xen environment. It is against the latest xen unstable tree on http://xenbits.xensource.com. The patch assumes the guest is passing a physically aligned hugepage. It does reference counting on all the underlying pages. Dave McCracken Oracle Corp. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Some issues: * You need to check return value of get_page_from_pagenr() on every page of the superpage. Any one of them can fail, causing you to undo your work so far and then fail. * You need to get_page_type(PGT_writable) on every page if the superpage mapping asserts _PAGE_RW. Otherwise the guest is getting write access without that being asserted in the reference counts. * Look at get_page_from_l1e() for an example of how this is done for a single page. You need to do similar work for every page of the super-page. * This surely breaks save/restore, since the restore code is not superpage-aware. -- Keir On 3/10/08 00:26, "Dave McCracken" <dcm@mccr.org> wrote:> > This patch enables support of hugepages in a pv Xen environment. It is > against the latest xen unstable tree on http://xenbits.xensource.com. > > The patch assumes the guest is passing a physically aligned hugepage. It does > reference counting on all the underlying pages. > > Dave McCracken > Oracle Corp. > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Friday 03 October 2008, Keir Fraser wrote:> Some issues: > * You need to check return value of get_page_from_pagenr() on every page > of the superpage. Any one of them can fail, causing you to undo your work > so far and then fail. > * You need to get_page_type(PGT_writable) on every page if the superpage > mapping asserts _PAGE_RW. Otherwise the guest is getting write access > without that being asserted in the reference counts. > * Look at get_page_from_l1e() for an example of how this is done for a > single page. You need to do similar work for every page of the super-page.Ok, here''s a version of the patch with all these issues addressed.> * This surely breaks save/restore, since the restore code is not > superpage-aware.I don''t have this one solved yet. I''m working on it. Dave McCracken _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 8/10/08 18:05, "Dave McCracken" <dcm@mccr.org> wrote:> On Friday 03 October 2008, Keir Fraser wrote: >> Some issues: >> * You need to check return value of get_page_from_pagenr() on every page >> of the superpage. Any one of them can fail, causing you to undo your work >> so far and then fail. >> * You need to get_page_type(PGT_writable) on every page if the superpage >> mapping asserts _PAGE_RW. Otherwise the guest is getting write access >> without that being asserted in the reference counts. >> * Look at get_page_from_l1e() for an example of how this is done for a >> single page. You need to do similar work for every page of the super-page. > > Ok, here''s a version of the patch with all these issues addressed. > >> * This surely breaks save/restore, since the restore code is not >> superpage-aware. > > I don''t have this one solved yet. I''m working on it.Actually this is an interesting one. For a PV guest it may be in general unsolvable, since the target machine may not have allocatable 2MB extents. It may also screw live migration since 2MB is a very coarse granularity to do dirty-page tracking. One option: perhaps the PV kernel could shatter and then reconstruct (as best it can) superpage mappings across save/restore? I''m actually not sure what''s for the best here. Perhaps just make 2MB mappings and save/restore mutually exclusive for now? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wednesday 08 October 2008, Keir Fraser wrote:> On 8/10/08 18:05, "Dave McCracken" <dcm@mccr.org> wrote: > > On Friday 03 October 2008, Keir Fraser wrote: > >> * This surely breaks save/restore, since the restore code is not > >> superpage-aware. > > > > I don''t have this one solved yet. I''m working on it. > > Actually this is an interesting one. For a PV guest it may be in general > unsolvable, since the target machine may not have allocatable 2MB extents. > It may also screw live migration since 2MB is a very coarse granularity to > do dirty-page tracking. One option: perhaps the PV kernel could shatter and > then reconstruct (as best it can) superpage mappings across save/restore? > I''m actually not sure what''s for the best here. Perhaps just make 2MB > mappings and save/restore mutually exclusive for now?Yeah, that''s what I''m finding. I think it''s a good idea to document for now that hugepages don''t work with save/restore. I''ll continue to dig into it and try to figure out a scheme to make it work as a future enhancement. Dave McCracken _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 8/10/08 19:28, "Dave McCracken" <dcm@mccr.org> wrote:>> Actually this is an interesting one. For a PV guest it may be in general >> unsolvable, since the target machine may not have allocatable 2MB extents. >> It may also screw live migration since 2MB is a very coarse granularity to >> do dirty-page tracking. One option: perhaps the PV kernel could shatter and >> then reconstruct (as best it can) superpage mappings across save/restore? >> I''m actually not sure what''s for the best here. Perhaps just make 2MB >> mappings and save/restore mutually exclusive for now? > > Yeah, that''s what I''m finding. I think it''s a good idea to document for now > that hugepages don''t work with save/restore. I''ll continue to dig into it > and try to figure out a scheme to make it work as a future enhancement.Then PV superpage support must be a configuration option, and disabled by default. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wednesday 08 October 2008, Keir Fraser wrote:> Then PV superpage support must be a configuration option, and disabled by > default.I added a command line option to enable it. Is this what you had in mind? Dave McCracken _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2008-Oct-08 22:50 UTC
Re: [Xen-devel] [PATCH 1/2] PV hugepages - Xen patch
Keir Fraser wrote:> Actually this is an interesting one. For a PV guest it may be in general > unsolvable, since the target machine may not have allocatable 2MB extents. > It may also screw live migration since 2MB is a very coarse granularity to > do dirty-page tracking. One option: perhaps the PV kernel could shatter and > then reconstruct (as best it can) superpage mappings across save/restore?That means you need to notify the guest when you''re starting a live migration, rather than just springing it on them at the last moment as we do now. But shattering large pages all over the place is going to be pretty expensive, and possibly awkward if it suddenly needs to come up with a pile of pages for the new L1 entries. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 8/10/08 23:07, "Dave McCracken" <dcm@mccr.org> wrote:> On Wednesday 08 October 2008, Keir Fraser wrote: >> Then PV superpage support must be a configuration option, and disabled by >> default. > > I added a command line option to enable it. Is this what you had in mind?It''ll certainly do, albeit rather coarse-grained. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel P. Berrange
2008-Oct-09 08:38 UTC
Re: [Xen-devel] [PATCH 1/2] PV hugepages - Xen patch
On Wed, Oct 08, 2008 at 03:50:56PM -0700, Jeremy Fitzhardinge wrote:> Keir Fraser wrote: > >Actually this is an interesting one. For a PV guest it may be in general > >unsolvable, since the target machine may not have allocatable 2MB extents. > >It may also screw live migration since 2MB is a very coarse granularity to > >do dirty-page tracking. One option: perhaps the PV kernel could shatter and > >then reconstruct (as best it can) superpage mappings across save/restore? > > That means you need to notify the guest when you''re starting a live > migration, rather than just springing it on them at the last moment as > we do now. > > But shattering large pages all over the place is going to be pretty > expensive, and possibly awkward if it suddenly needs to come up with a > pile of pages for the new L1 entries.Or you could just take the view this is a pre-migration capability check, and that admin (or mgmt app) must ensure sufficient free hugepages on the destination before attempting migration. If this isn''t satisfied then XenD can just fail / abort the migration op and leave it running on original host. Dainel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 8/10/08 23:07, "Dave McCracken" <dcm@mccr.org> wrote:> On Wednesday 08 October 2008, Keir Fraser wrote: >> Then PV superpage support must be a configuration option, and disabled by >> default. > > I added a command line option to enable it. Is this what you had in mind?Please fix the coding style (brace positions; white space around if/while/for headers; etc) and resubmit this and the Linux patch with a signed-off-by attribution. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2008-Oct-10 00:05 UTC
Re: [Xen-devel] [PATCH 1/2] PV hugepages - Xen patch
Daniel P. Berrange wrote:> Or you could just take the view this is a pre-migration capability check, > and that admin (or mgmt app) must ensure sufficient free hugepages on the > destination before attempting migration. If this isn''t satisfied then > XenD can just fail / abort the migration op and leave it running on original > host. >And do something to prevent new hugepages from being allocated during the dirty logging phase? J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel