forgot my signed-off, thus resent... the following patch adds NUMA node migration based on live migration to xend. By adding another parameter to "xm migrate" the target NUMA node number gets propagated to the target host (can be both localhost or a remote host). The restore function then sets the VCPU affinity accordingly. Only changes Python code in xend. I hope that the patch doesn''t break XenAPI compatibility (adding a parameter seems fine?). # xm migrate --live --node=<nodenr> <domid> localhost <nodenr> is the number as shown with ''xm info'' under node_to_cpu I am aware that using live migration isn''t the best approach (takes twice the memory and quite some time), but it''s less intrusive and works fine (given localhost migration stability...) Feedback appreciated, especially since I speak Python since Monday... Regards, Andre. Signed-off-by: Andre Przywara <andre.przywara@amd.com> -- Andre Przywara AMD-Operating System Research Center (OSRC), Dresden, Germany Tel: +49 351 277-84917 ----to satisfy European Law for business letters: AMD Saxony Limited Liability Company & Co. KG, Wilschdorfer Landstr. 101, 01109 Dresden, Germany Register Court Dresden: HRA 4896, General Partner authorized to represent: AMD Saxony LLC (Wilmington, Delaware, US) General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel P. Berrange
2007-Dec-21 22:45 UTC
Re: [Xen-devel] [PATCH] (resent) NUMA node migration
On Fri, Dec 21, 2007 at 11:39:58PM +0100, Andre Przywara wrote:> forgot my signed-off, thus resent... > > the following patch adds NUMA node migration based on live migration to > xend. By adding another parameter to "xm migrate" the target NUMA node > number gets propagated to the target host (can be both localhost or a > remote host). The restore function then sets the VCPU affinity > accordingly. Only changes Python code in xend. I hope that the patch > doesn''t break XenAPI compatibility (adding a parameter seems fine?). > > # xm migrate --live --node=<nodenr> <domid> localhost > <nodenr> is the number as shown with ''xm info'' under node_to_cpu > > I am aware that using live migration isn''t the best approach (takes > twice the memory and quite some time), but it''s less intrusive and works > fine (given localhost migration stability...) > > Feedback appreciated, especially since I speak Python since Monday...Rather than using ''-1'' to indiciate no pinning, it is more common python practice to use None, which indicates no value. Other than that it looks reasonable Regards, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Dec 21, 2007 at 11:39:58PM +0100, Andre Przywara wrote:> accordingly. Only changes Python code in xend. I hope that the patch > doesn''t break XenAPI compatibility (adding a parameter seems fine?). > > # xm migrate --live --node=<nodenr> <domid> localhost > <nodenr> is the number as shown with ''xm info'' under node_to_cpu > > I am aware that using live migration isn''t the best approach (takes > twice the memory and quite some time), but it''s less intrusive and works > fine (given localhost migration stability...)Is this really using localhost live migration to move a domain from one NUMA node to another on the same host? Why isn''t there a simpler way? cheers john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > I am aware that using live migration isn''t the best approach (takes > > twice the memory and quite some time), but it''s less intrusive and > works > > fine (given localhost migration stability...) > > Is this really using localhost live migration to move a domain fromone> NUMA node to another on the same host?Yep, it really is.> Why isn''t there a simpler way?Well, you can''t beat using live migration for code simplicity :) Doing page migration for HVM guests is easy as you can just stop the guest, copy a bunch of pages and update the p2m table, flush the shadow page table cache and all VCPU''s TLBs, then resume. Doing it live is a little tricky as you have to go from MFN''s to PTE''s which currently requires a full shadow page table scan. [Though some of the experimental page sharing patches maintain linked lists of backpointers, and we could switch to a shadow mode that supports this while doing page migration] PV guests are a little more challenging as all references in the direct-mode page tables need to be updated. We also need to make sure that the guest isn''t holding MFNs outside of pagetables, so we need to get all the VCPUs into a known state. The best way of handling this is to use the PV fast checkpoint support to freeze the guest, copy the badly located pages, scan and update all pagetables, resume from checkpoint. This would make a nice little project for someone... Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
hi for PV, is there a fast checkpoint mechanism available ? how does it work , that is ,what paremeters in the common line"xm migration ",or how to invoke it? and how does it work ,say ,fast checkpoint for PV in an incremental way such as live migration, or how to fast checkpoint ? Thanks in advance Ian Pratt 写道:> > Doing page migration for HVM guests is easy as you can just stop the > guest, copy a bunch of pages and update the p2m table, flush the shadow > page table cache and all VCPU''s TLBs, then resume. Doing it live is a > little tricky as you have to go from MFN''s to PTE''s which currently > requires a full shadow page table scan. [Though some of the experimental > page sharing patches maintain linked lists of backpointers, and we could > switch to a shadow mode that supports this while doing page migration] > > PV guests are a little more challenging as all references in the > direct-mode page tables need to be updated. We also need to make sure > that the guest isn''t holding MFNs outside of pagetables, so we need to > get all the VCPUs into a known state. The best way of handling this is > to use the PV fast checkpoint support to freeze the guest, copy the >> badly located pages, scan and update all pagetables, resume from > checkpoint. This would make a nice little project for someone... > > Ian > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Mark Williamson
2007-Dec-23 21:12 UTC
Re: [Xen-devel] [PATCH] (resent) NUMA node migration
> for PV, is there a fast checkpoint mechanism available ? how does it > work , that is ,what paremeters in the common line"xm migration ",or how > to invoke it?xm save -c does a checkpoint The "Checkpoint" operation means that the guest is resumed immediately after the save has completed, rather than being stopped or migrated.> and how does it work ,say ,fast checkpoint for PV in an > incremental way such as live migration, or how to fast checkpoint ?It''s not an incremental process in the case of a checkpoint: at the moment it still stops the guests execution for the period of the save - just like normal xm save does. After the save continues, the guest is instructed to resume its own execution. Cheers, Mark> > Thanks in advance > > Ian Pratt 写道: > > Doing page migration for HVM guests is easy as you can just stop the > > guest, copy a bunch of pages and update the p2m table, flush the shadow > > page table cache and all VCPU''s TLBs, then resume. Doing it live is a > > little tricky as you have to go from MFN''s to PTE''s which currently > > requires a full shadow page table scan. [Though some of the experimental > > page sharing patches maintain linked lists of backpointers, and we could > > switch to a shadow mode that supports this while doing page migration] > > > > PV guests are a little more challenging as all references in the > > direct-mode page tables need to be updated. We also need to make sure > > that the guest isn''t holding MFNs outside of pagetables, so we need to > > get all the VCPUs into a known state. The best way of handling this is > > to use the PV fast checkpoint support to freeze the guest, copy the > > > > > > badly located pages, scan and update all pagetables, resume from > > checkpoint. This would make a nice little project for someone... > > > > Ian > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel-- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
John Levon wrote:> On Fri, Dec 21, 2007 at 11:39:58PM +0100, Andre Przywara wrote: > >> accordingly. Only changes Python code in xend. I hope that the patch >> doesn''t break XenAPI compatibility (adding a parameter seems fine?). >> >> # xm migrate --live --node=<nodenr> <domid> localhost >> <nodenr> is the number as shown with ''xm info'' under node_to_cpu >> >> I am aware that using live migration isn''t the best approach (takes >> twice the memory and quite some time),After thinking more about this, I came to the conclusion that this is not entirely true. If you want to transfer the guest from one node (with it''s own memory) to another node (with it''s own memory), then you would need to have this amount of memory available at the target anyway. The performance problem can maybe overcome by not using TCP/IP to loopback, but a more direct solution (or by reverting to a UNIX socket).>>but it''s less intrusive and works >> fine (given localhost migration stability...) > > Is this really using localhost live migration to move a domain from one > NUMA node to another on the same host? Why isn''t there a simpler way?Well, Ian described the "simpler way" (thanks for that), I would call it more elegant and efficient, but simpler is not the word that comes to mind... One advantage of this solution is that the guest stays the same and there is no interruption. Also one could think about migrating hot pages first and letting the rest be done in the background more slowly (this is how VMware does this). Regards, Andre. -- Andre Przywara AMD-Operating System Research Center (OSRC), Dresden, Germany Tel: +49 351 277-84917 ----to satisfy European Law for business letters: AMD Saxony Limited Liability Company & Co. KG, Wilschdorfer Landstr. 101, 01109 Dresden, Germany Register Court Dresden: HRA 4896, General Partner authorized to represent: AMD Saxony LLC (Wilmington, Delaware, US) General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel