Hi all, I''m brand new to Xen and this list. I''m planning on using Xen. Especialy I''m interested in the Live Migration Feature. Somewhere I read that the hardware must be identical. What does this mean? Will migration work from a P4 to a Xeon (Core2 Mircoarchitecure) ? Will migration work if they got different NIC? What are the limitations? Maybe someone does already know or has a good link. Bye and thanks Thorsten -- ... black holes are where god divided by zero. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Dominik Klein
2007-Jun-29 12:23 UTC
Re: [Xen-users] Live Migration with different hardware
> Will migration work from a P4 to a Xeon (Core2 Mircoarchitecure) ? Will > migration work if they got different NIC? What are the limitations?Afaik, you need identical CPU flags. Compare the output of this command on your machines: cat /proc/cpuinfo|grep flags A different NIC should not be a problem. Of course you might want both to be of the same speed. As long as you bridge them correctly, a different NIC should not be an issue. Regards Dominik _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Petersson, Mats
2007-Jun-29 13:02 UTC
RE: [Xen-users] Live Migration with different hardware
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > Dominik Klein > Sent: 29 June 2007 13:24 > To: xen-users@lists.xensource.com > Subject: Re: [Xen-users] Live Migration with different hardware > > > Will migration work from a P4 to a Xeon (Core2 > Mircoarchitecure) ? Will > > migration work if they got different NIC? What are the limitations? > > Afaik, you need identical CPU flags. > > Compare the output of this command on your machines: > cat /proc/cpuinfo|grep flagsThat may well be the case. The "correct" test should really be something like: flags_new_cpu & ~flags_old_cpu != 0 That means, no flags on the old cpu that doesn''t exist on the new CPU. I''m not sure how much, if any, flags are actually tested on the migration path. The good thing is that if the migration fails due to incompatible CPU''s (by testing flags) it should not "fail" the domain - that is, the domain should stay on the original machine and you should get an error message - at least that''s how I understand it.> > A different NIC should not be a problem. Of course you might > want both > to be of the same speed. As long as you bridge them correctly, a > different NIC should not be an issue.Yes, as long as you don''t use "pci-passtrhough" to make a PCI card (e.g. network card) available directly to the domain, the guest DOESN''T KNOW what the REAL network card is [this makes the domain a driver domain, and driver domains aren''t allowed to be migrated anyways - that would be TERRIBLY hard to implement, as there''s no real good way to transfer the internal state of the hardware owned on one domain to the next one]. -- Mats> > Regards > Dominik > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nico Kadel-Garcia
2007-Jun-29 13:37 UTC
Re: [Xen-users] Live Migration with different hardware
Kyrios wrote:> Hi all, > > I''m brand new to Xen and this list. I''m planning on using Xen. > Especialy I''m interested in the Live Migration Feature. Somewhere I > read that the hardware must be identical. What does this mean?It need not. It''s important that the Xen servers, the Dom0''s, have similar versions of Xen and have a shared network repository such as iscsi where the Xen guest domain resides for *both* systems.> Will migration work from a P4 to a Xeon (Core2 Mircoarchitecure) ? > Will migration work if they got different NIC? What are the limitations? > > Maybe someone does already know or has a good link.This is normal. Do you want to use para-virtualized, or fully virtualized? And do you have a good network capable fileserver such as iscsi? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mark Williamson
2007-Jun-29 15:09 UTC
Re: [Xen-users] Live Migration with different hardware
> I''m brand new to Xen and this list. I''m planning on using Xen. Especialy > I''m interested in the Live Migration Feature. Somewhere I read that the > hardware must be identical. What does this mean? > > Will migration work from a P4 to a Xeon (Core2 Mircoarchitecure) ? Will > migration work if they got different NIC? What are the limitations?The devices won''t make a difference (nb. you can only migrate a domain that doesn''t have direct access to physical devices, but that''s the normal case so it shouldn''t be a problem). So the network card doesn''t matter. The "identical hardware" stipulation arises from problems with the CPU and the instruction set it supports. When Linux starts up, it tests what instructions are available and may choose to use certain instructions that are only available on that platform. Userspace libraries may also do this in order to provide optimised implementations of functions. This is fine if you move to a more featureful processor so e.g. P4 -> Core2 migration will probably work. Moving back can break in some cases, however, since the kernel and libraries start off using instructions which disappear when they migrate back. It''s worth being a bit careful about this. Often, people can start a domain on the less featureful system and then move it to a more featureful system *and back* without a crash. This is because the domain started with the smaller featureset and assumes that it stays the same - having booted on the less featureful CPU, it moves back successfully. This may not always work safely, since userspace may start using the extra instructions in the meantime and then crash when migrated back. The other thing that bites some people is that, although they sometimes get away with migrating a domain back to a less featureful machine, if it reboots whilst at the newer machine the kernel will detect the new features and start using them. Then it''ll *definitely* crash when migrated back :-( I believe Core2 has some instructions that P4 doesn''t, so migration will probably only be reliable in the P4->Core2 direction. Note that in my above discussion I''ve mainly spoken about more featureful, or newer CPUs. In practice, the same issues can occur when migrating between Intel and AMD CPUs of the same general age and capability, since they might implement different instructions for certain functions. Hope this helps you some. The same issues are true for live migration, non-live migration and save/resume. Shutting down a guest and then booting it on a new host of suitable architecture should always work. Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mark Williamson
2007-Jun-29 15:15 UTC
Re: [Xen-users] Live Migration with different hardware
> > Compare the output of this command on your machines: > > cat /proc/cpuinfo|grep flags > > That may well be the case. The "correct" test should really be something > like: > > flags_new_cpu & ~flags_old_cpu != 0 > > That means, no flags on the old cpu that doesn''t exist on the new CPU. > > I''m not sure how much, if any, flags are actually tested on the > migration path.I don''t think they are.> The good thing is that if the migration fails due to incompatible CPU''s > (by testing flags) it should not "fail" the domain - that is, the domain > should stay on the original machine and you should get an error message > - at least that''s how I understand it.I don''t think you can rely on this happening :-( I''m not sure what the status is of not killing the origin domain if migration fails. But regardless of this, mismatched CPUs is probably still dangerous. If you migrate a domain to a machine missing instructions that the domain is using, the /migration/ will succeed just fine - the execution state will all be transferred and the domain will successfully start running. So the control tools will want to kill the origin domain so that they don''t conflict over disk / network access. Unfortunately, this doesn''t stop the migrated guest from crashing a few cycles after the migration completes due to an undefined operation :-( It''d be nice if more flag checking was done, but AFAIK at the moment it isn''t. There''s also not a generally easy way to see if a guest is using the extra features on a particular CPU or not, and thus it''s hard to tell if migration would break anything. Trapping the CPUID instruction and faking out a "least common denominator" has been discussed but not (as far as I know) actually implemented. If it were possible to do this then you could sacrifice some machine-local optimisations and trade them off for the ability to migrate safely and freely within a cluster.> Yes, as long as you don''t use "pci-passtrhough" to make a PCI card (e.g. > network card) available directly to the domain, the guest DOESN''T KNOW > what the REAL network card is [this makes the domain a driver domain, > and driver domains aren''t allowed to be migrated anyways - that would be > TERRIBLY hard to implement, as there''s no real good way to transfer the > internal state of the hardware owned on one domain to the next one].Domains with PCI access shouldn''t be migrated or saved/restored. I think the tools might allow this at the moment, but that''s a bug and should be fixed (note to self: consider fixing this bug!) Cheers, mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Petersson, Mats
2007-Jun-29 15:21 UTC
RE: [Xen-users] Live Migration with different hardware
> -----Original Message----- > From: M.A. Williamson [mailto:maw48@hermes.cam.ac.uk] On > Behalf Of Mark Williamson > Sent: 29 June 2007 16:15 > To: xen-users@lists.xensource.com > Cc: Petersson, Mats; Dominik Klein; Kyrios > Subject: Re: [Xen-users] Live Migration with different hardware > > > > Compare the output of this command on your machines: > > > cat /proc/cpuinfo|grep flags > > > > That may well be the case. The "correct" test should really > be something > > like: > > > > flags_new_cpu & ~flags_old_cpu != 0 > > > > That means, no flags on the old cpu that doesn''t exist on > the new CPU. > > > > I''m not sure how much, if any, flags are actually tested on the > > migration path. > > I don''t think they are. > > > The good thing is that if the migration fails due to > incompatible CPU''s > > (by testing flags) it should not "fail" the domain - that > is, the domain > > should stay on the original machine and you should get an > error message > > - at least that''s how I understand it. > > I don''t think you can rely on this happening :-(Yes, if there''s no test, then it''s not going to work. [snip long explanation of why it doesn''t work]> > Yes, as long as you don''t use "pci-passtrhough" to make a > PCI card (e.g. > > network card) available directly to the domain, the guest > DOESN''T KNOW > > what the REAL network card is [this makes the domain a > driver domain, > > and driver domains aren''t allowed to be migrated anyways - > that would be > > TERRIBLY hard to implement, as there''s no real good way to > transfer the > > internal state of the hardware owned on one domain to the next one]. > > Domains with PCI access shouldn''t be migrated or > saved/restored. I think the > tools might allow this at the moment, but that''s a bug and > should be fixed > (note to self: consider fixing this bug!)That would be a good bug to fix... And I thought that I said that this type of domain can''t be migrated. I don''t think it''s very hard to discover if this domain has a PCI device attached or not - but I haven''t looked at it... -- Mats> > Cheers, > mark >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mark Williamson
2007-Jun-29 15:27 UTC
Re: [Xen-users] Live Migration with different hardware
> > Domains with PCI access shouldn''t be migrated or > > saved/restored. I think the > > tools might allow this at the moment, but that''s a bug and > > should be fixed > > (note to self: consider fixing this bug!) > > That would be a good bug to fix... And I thought that I said that this > type of domain can''t be migrated.Yes, I was just agreeing with you ;-) Actually, quite possibly, the *can* be migrated, it just won''t work and will crash things instead ;-p> I don''t think it''s very hard to discover if this domain has a PCI device > attached or not - but I haven''t looked at it...Should just be a modification in Python code to check if pci devs are assigned. I might be able to get round to it soon. We''ve had one report of somebody''s system breaking because the xendomains init script suspended and resumed a driver domain on reboot :-( Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users