Xen users - Jun 2007 - Live Migration with different hardware

Hi all,

I''m brand new to Xen and this list. I''m planning on using Xen.
Especialy I''m
interested in the Live Migration Feature. Somewhere I read that the hardware
must be identical. What does this mean?

Will migration work from a P4 to a Xeon (Core2 Mircoarchitecure) ? Will
migration work if they got different NIC? What are the limitations?

Maybe someone does already know or has a good link.

Bye and thanks
Thorsten

-- 
... black holes are where god divided by zero.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Will migration work from a P4 to a Xeon (Core2 Mircoarchitecure) ? Will 
> migration work if they got different NIC? What are the limitations?
Afaik, you need identical CPU flags.

Compare the output of this command on your machines:
cat /proc/cpuinfo|grep flags

A different NIC should not be a problem. Of course you might want both 
to be of the same speed. As long as you bridge them correctly, a 
different NIC should not be an issue.

Regards
Dominik

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> -----Original Message-----
> From: xen-users-bounces@lists.xensource.com 
> [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of 
> Dominik Klein
> Sent: 29 June 2007 13:24
> To: xen-users@lists.xensource.com
> Subject: Re: [Xen-users] Live Migration with different hardware
> 
> > Will migration work from a P4 to a Xeon (Core2 
> Mircoarchitecure) ? Will 
> > migration work if they got different NIC? What are the limitations?
> 
> Afaik, you need identical CPU flags.
> 
> Compare the output of this command on your machines:
> cat /proc/cpuinfo|grep flags
That may well be the case. The "correct" test should really be
something
like:

flags_new_cpu & ~flags_old_cpu != 0

That means, no flags on the old cpu that doesn''t exist on the new CPU. 

I''m not sure how much, if any, flags are actually tested on the
migration path.

The good thing is that if the migration fails due to incompatible CPU''s
(by testing flags) it should not "fail" the domain - that is, the
domain
should stay on the original machine and you should get an error message
- at least that''s how I understand it. 
> 
> A different NIC should not be a problem. Of course you might 
> want both 
> to be of the same speed. As long as you bridge them correctly, a 
> different NIC should not be an issue.
Yes, as long as you don''t use "pci-passtrhough" to make a PCI
card (e.g.
network card) available directly to the domain, the guest DOESN''T KNOW
what the REAL network card is [this makes the domain a driver domain,
and driver domains aren''t allowed to be migrated anyways - that would
be
TERRIBLY hard to implement, as there''s no real good way to transfer the
internal state of the hardware owned on one domain to the next one]. 

--
Mats
> 
> Regards
> Dominik
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
> 
> 
> 


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Kyrios wrote:
> Hi all,
>
> I''m brand new to Xen and this list. I''m planning on using
Xen.
> Especialy I''m interested in the Live Migration Feature. Somewhere
I
> read that the hardware must be identical. What does this mean?
It need not. It''s important that the Xen servers, the Dom0''s,
have
similar versions of Xen and have a shared network repository such as 
iscsi where the Xen guest domain resides for *both* systems.

> Will migration work from a P4 to a Xeon (Core2 Mircoarchitecure) ? 
> Will migration work if they got different NIC? What are the limitations?
>
> Maybe someone does already know or has a good link.
This is normal. Do you want to use para-virtualized, or fully 
virtualized? And do you have a good network capable fileserver such as 
iscsi?

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> I''m brand new to Xen and this list. I''m planning on using
Xen. Especialy
> I''m interested in the Live Migration Feature. Somewhere I read
that the
> hardware must be identical. What does this mean?
>
> Will migration work from a P4 to a Xeon (Core2 Mircoarchitecure) ? Will
> migration work if they got different NIC? What are the limitations?
The devices won''t make a difference (nb. you can only migrate a domain
that
doesn''t have direct access to physical devices, but that''s the
normal case so
it shouldn''t be a problem).  So the network card doesn''t
matter.

The "identical hardware" stipulation arises from problems with the CPU
and the
instruction set it supports.  When Linux starts up, it tests what 
instructions are available and may choose to use certain instructions that 
are only available on that platform.  Userspace libraries may also do this in 
order to provide optimised implementations of functions.

This is fine if you move to a more featureful processor so e.g. P4 -> Core2 
migration will probably work.  Moving back can break in some cases, however, 
since the kernel and libraries start off using instructions which disappear 
when they migrate back.

It''s worth being a bit careful about this.  Often, people can start a
domain
on the less featureful system and then move it to a more featureful system 
*and back* without a crash.  This is because the domain started with the 
smaller featureset and assumes that it stays the same - having booted on the 
less featureful CPU, it moves back successfully.  This may not always work 
safely, since userspace may start using the extra instructions in the 
meantime and then crash when migrated back.

The other thing that bites some people is that, although they sometimes get 
away with migrating a domain back to a less featureful machine, if it reboots 
whilst at the newer machine the kernel will detect the new features and start 
using them.  Then it''ll *definitely* crash when migrated back :-(

I believe Core2 has some instructions that P4 doesn''t, so migration
will
probably only be reliable in the P4->Core2 direction.

Note that in my above discussion I''ve mainly spoken about more
featureful, or
newer CPUs.  In practice, the same issues can occur when migrating between 
Intel and AMD CPUs of the same general age and capability, since they might 
implement different instructions for certain functions.

Hope this helps you some.  The same issues are true for live migration, 
non-live migration and save/resume.  Shutting down a guest and then booting 
it on a new host of suitable architecture should always work.

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> > Compare the output of this command on your machines:
> > cat /proc/cpuinfo|grep flags
>
> That may well be the case. The "correct" test should really be
something
> like:
>
> flags_new_cpu & ~flags_old_cpu != 0
>
> That means, no flags on the old cpu that doesn''t exist on the new
CPU.
>
> I''m not sure how much, if any, flags are actually tested on the
> migration path.
I don''t think they are.
> The good thing is that if the migration fails due to incompatible
CPU''s
> (by testing flags) it should not "fail" the domain - that is, the
domain
> should stay on the original machine and you should get an error message
> - at least that''s how I understand it.
I don''t think you can rely on this happening :-(

I''m not sure what the status is of not killing the origin domain if
migration
fails.  But regardless of this, mismatched CPUs is probably still dangerous.  
If you migrate a domain to a machine missing instructions that the domain is 
using, the /migration/ will succeed just fine - the execution state will all 
be transferred and the domain will successfully start running.  So the 
control tools will want to kill the origin domain so that they don''t
conflict
over disk / network access.

Unfortunately, this doesn''t stop the migrated guest from crashing a few
cycles
after the migration completes due to an undefined operation :-(

It''d be nice if more flag checking was done, but AFAIK at the moment it
isn''t.
There''s also not a generally easy way to see if a guest is using the
extra
features on a particular CPU or not, and thus it''s hard to tell if
migration
would break anything.  Trapping the CPUID instruction and faking out a
"least
common denominator" has been discussed but not (as far as I know) actually 
implemented.  If it were possible to do this then you could sacrifice some 
machine-local optimisations and trade them off for the ability to migrate 
safely and freely within a cluster.
> Yes, as long as you don''t use "pci-passtrhough" to make
a PCI card (e.g.
> network card) available directly to the domain, the guest DOESN''T
KNOW
> what the REAL network card is [this makes the domain a driver domain,
> and driver domains aren''t allowed to be migrated anyways - that
would be
> TERRIBLY hard to implement, as there''s no real good way to
transfer the
> internal state of the hardware owned on one domain to the next one].
Domains with PCI access shouldn''t be migrated or saved/restored.  I
think the
tools might allow this at the moment, but that''s a bug and should be
fixed
(note to self: consider fixing this bug!)

Cheers,
mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> -----Original Message-----
> From: M.A. Williamson [mailto:maw48@hermes.cam.ac.uk] On 
> Behalf Of Mark Williamson
> Sent: 29 June 2007 16:15
> To: xen-users@lists.xensource.com
> Cc: Petersson, Mats; Dominik Klein; Kyrios
> Subject: Re: [Xen-users] Live Migration with different hardware
> 
> > > Compare the output of this command on your machines:
> > > cat /proc/cpuinfo|grep flags
> >
> > That may well be the case. The "correct" test should really 
> be something
> > like:
> >
> > flags_new_cpu & ~flags_old_cpu != 0
> >
> > That means, no flags on the old cpu that doesn''t exist on 
> the new CPU.
> >
> > I''m not sure how much, if any, flags are actually tested on
the
> > migration path.
> 
> I don''t think they are.
> 
> > The good thing is that if the migration fails due to 
> incompatible CPU''s
> > (by testing flags) it should not "fail" the domain - that 
> is, the domain
> > should stay on the original machine and you should get an 
> error message
> > - at least that''s how I understand it.
> 
> I don''t think you can rely on this happening :-(
Yes, if there''s no test, then it''s not going to work.

[snip long explanation of why it doesn''t work]
> > Yes, as long as you don''t use "pci-passtrhough" to
make a
> PCI card (e.g.
> > network card) available directly to the domain, the guest 
> DOESN''T KNOW
> > what the REAL network card is [this makes the domain a 
> driver domain,
> > and driver domains aren''t allowed to be migrated anyways - 
> that would be
> > TERRIBLY hard to implement, as there''s no real good way to 
> transfer the
> > internal state of the hardware owned on one domain to the next one].
> 
> Domains with PCI access shouldn''t be migrated or 
> saved/restored.  I think the 
> tools might allow this at the moment, but that''s a bug and 
> should be fixed 
> (note to self: consider fixing this bug!)
That would be a good bug to fix... And I thought that I said that this
type of domain can''t be migrated. 

I don''t think it''s very hard to discover if this domain has a
PCI device
attached or not - but I haven''t looked at it... 

--
Mats
> 
> Cheers,
> mark
> 


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> > Domains with PCI access shouldn''t be migrated or
> > saved/restored.  I think the
> > tools might allow this at the moment, but that''s a bug and
> > should be fixed
> > (note to self: consider fixing this bug!)
>
> That would be a good bug to fix... And I thought that I said that this
> type of domain can''t be migrated.
Yes, I was just agreeing with you ;-)

Actually, quite possibly, the *can* be migrated, it just won''t work and
will
crash things instead ;-p
> I don''t think it''s very hard to discover if this domain
has a PCI device
> attached or not - but I haven''t looked at it...
Should just be a modification in Python code to check if pci devs are 
assigned.  I might be able to get round to it soon.  We''ve had one
report of
somebody''s system breaking because the xendomains init script suspended
and
resumed a driver domain on reboot :-(

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users