Xen users - Mar 2007 - Migration only works one way (seemingly depending on machine''s boot time)

Hi

I have two completely identical machines running opensuse 10.2. domU 
(also opensuse 10.2) is running on Root NFS.

When trying to migrate this domU from machine 1 to machine 2, it works 
perfectly. When trying to migrate back to machine 1, I can see the domU 
in "xm list" on machine 1, but cannot connect to it with "xm
console"
nor ssh.
The network interface is created, but says "NO-CARRIER" and xentop
shows
100% CPU usage for this domU.

When trying to migrate this domU from machine 2 to machine 1 without 
having migrated it earlier, this fails in the same way. Migration also 
fails using xm save, copying the file manually and restoring the file on 
the other machine.

Now the (at least to me) really strange thing: After a reboot of machine 
2, the whole procedure is the other way around. I am able to migrate 
from machine 2 to machine 1, but not from machine 1 to machine 2 - 
independant of wether the vm has been migrated before or not.

This is reproducable. The machine with less uptime can migrate virtual 
machines to the other machine. The other way never works.

I compared /etc using diff -r and it showed no difference except for 
hostname, ip addresses and stuff you''d expect to be different.

Any ideas?

Regards
Dominik

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Here''s some more information:

Situation: machine 2 migrates vm to machine 1, success
then machine 1 migrates vm back to machine 2, fails as described ealier

Now sniffing xenbr0 on machine 1 (ONE!) after the seemingly successful 
(but yet unsuccessful) migration to machine 2 shows that packets for the 
vm''s ip and mac address still come in on machine 1 (yes, ONE).

Maybe this helps figure it out?

Regards
Dominik

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, 2007-03-13 at 11:09 +0100, Dominik Klein wrote:
> Here''s some more information:
> 
> Situation: machine 2 migrates vm to machine 1, success
> then machine 1 migrates vm back to machine 2, fails as described ealier
> 
> Now sniffing xenbr0 on machine 1 (ONE!) after the seemingly successful 
> (but yet unsuccessful) migration to machine 2 shows that packets for the 
> vm''s ip and mac address still come in on machine 1 (yes, ONE).
> 
> Maybe this helps figure it out?
> 
> Regards
> Dominik
Are you assigning static MACS to these guests in the vif [] container?
This could be router re-arp lag wreaking havoc on NFS.

A failed NFS hard mount [ or the locking needed therein ] (especially on
a nfsroot) would cause the behavior you''re seeing. Could be a simple
fix.

Just a suggestion :)

Best,
--Tim
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Are you assigning static MACS to these guests in the vif [] container?
> This could be router re-arp lag wreaking havoc on NFS.
I tried it with and without a static guest MAC.

The result is the same, because a xen-generated MAC only changes at 
creation time, not at migration time. So the switch "sees" one MAC
that
is moved to another switch port.

Regards
Dominik

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Situation: machine 2 migrates vm to machine 1, success
> then machine 1 migrates vm back to machine 2, fails as described
ealier

I''ve seen this phenomenology on an earlier version of xen than
I am using today. Upgrading to 3.0.4 made things work better, but
not yet 100% reliable. Suggest you try "service xend restart" on
machine 2 before the migration and see if you get different results.

FYI, the Xen developers are not yet advertising migration as a stable
feature. I.e., as a migration user you are inside their development
cycle ("helping them debug").

Caveat emptor,

Joe.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Do the machines have different CPU types?

Cheers,
Mark

On Tuesday 13 March 2007 09:50, Dominik Klein wrote:
> Hi
>
> I have two completely identical machines running opensuse 10.2. domU
> (also opensuse 10.2) is running on Root NFS.
>
> When trying to migrate this domU from machine 1 to machine 2, it works
> perfectly. When trying to migrate back to machine 1, I can see the domU
> in "xm list" on machine 1, but cannot connect to it with "xm
console"
> nor ssh.
> The network interface is created, but says "NO-CARRIER" and
xentop shows
> 100% CPU usage for this domU.
>
> When trying to migrate this domU from machine 2 to machine 1 without
> having migrated it earlier, this fails in the same way. Migration also
> fails using xm save, copying the file manually and restoring the file on
> the other machine.
>
> Now the (at least to me) really strange thing: After a reboot of machine
> 2, the whole procedure is the other way around. I am able to migrate
> from machine 2 to machine 1, but not from machine 1 to machine 2 -
> independant of wether the vm has been migrated before or not.
>
> This is reproducable. The machine with less uptime can migrate virtual
> machines to the other machine. The other way never works.
>
> I compared /etc using diff -r and it showed no difference except for
> hostname, ip addresses and stuff you''d expect to be different.
>
> Any ideas?
>
> Regards
> Dominik
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Mark Williamson schrieb:
> Do the machines have different CPU types?
No.

machine1:
processor       : 3
vendor_id       : GenuineIntel
cpu family      : 15
model           : 4
model name      : Intel(R) Xeon(TM) CPU 3.00GHz
stepping        : 3
cpu MHz         : 2992.710
cache size      : 2048 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36 
clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc up 
pni monitor ds_cpl cid cx16 xtpr
bogomips        : 7484.92

machine2:
processor       : 3
vendor_id       : GenuineIntel
cpu family      : 15
model           : 4
model name      : Intel(R) Xeon(TM) CPU 3.00GHz
stepping        : 3
cpu MHz         : 2992.708
cache size      : 2048 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36 
clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc up 
pni monitor ds_cpl cid cx16 xtpr
bogomips        : 7484.88

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> I''ve seen this phenomenology on an earlier version of xen than
> I am using today. Upgrading to 3.0.4 made things work better, but
> not yet 100% reliable. 
I''ll try to upgrade. Not too sure about how to do that yet, but
I''ll see
what I can find about this topic. If you have any link at hand, feel 
free to share it :)
> Suggest you try "service xend restart" on
> machine 2 before the migration and see if you get different results.
No difference at all.
> FYI, the Xen developers are not yet advertising migration as a stable
> feature. I.e., as a migration user you are inside their development
> cycle ("helping them debug").
At least that''s something this is good for ;)

Dominik

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Dominik Klein schrieb:
>> I''ve seen this phenomenology on an earlier version of xen than
>> I am using today. Upgrading to 3.0.4 made things work better, but
>> not yet 100% reliable. 
> 
> I''ll try to upgrade. Not too sure about how to do that yet, but
I''ll see
> what I can find about this topic. If you have any link at hand, feel 
> free to share it :)
So I did upgrade to 3.0.4 (and downgrade to 2.6.16.33 necessarily) and 
what I can say up to now: It seems to work.

Migrating from 1 to 2 and vice versa is possible now regardless of which 
machine booted last. So it does not seem to be a switch (arp) problem.

Regards
Dominik

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> > Suggest you try "service xend restart" on
> > machine 2 before the migration and see if you get different results.
> 
> No difference at all.
Yes, weird, eh? Somehow these migration problems are indicating that
they
are "stateful," but I have not been able to track down where that
state
might be recorded.

Joe.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users