Xen users - Apr 2008 - Migrating domUs behind a firewall backend domU

I have the following setup:

- standard Dom0, _without_ xen bridge
- netbackend domU as a firewall, i.e. nic is exclusively used by this domU via 
pciback mechanism. This domU hosts the xen bridge and does the network setup 
for the other domUs (/etc/xen/scripts/vif-bridge etc.)

- a bunch of domUs _behind_ this firewall domU

This setup is described in
http://lists.xensource.com/archives/html/xen-users/2005-07/msg00558.html

Everything is working just fine. So far so good.
My next goal is to migrate one of the domUs _behind_ the netbackend firewall 
to another machine with the same setup.  I know I cannot migrate the 
netbackend firewall domU.

The migration works fine but unfortunately networking stops completely on the 
migrated domain. Otherwise it works just fine but has no network. This is 
what happens:

On xenhost1 I do:
# xm list
# xen16                                     15      128     1 -b----     14.4

# xm migrate xen16 xenhost2 --live
#

This is what''s then shown on xenhost2:
# xm list
# xen16                                     14      128     1 -b----      0.0

The following is the kernel output on the console.  When I hit enter I see the 
login screen again:
# xm console xen16
vif vif-0: 2 reading other end details from device/vif/0
xenbus: resume (talk_to_otherend) vif-0 failed: -2

Both Xenhosts are 64bit Debian Etch (Xen 3.0.3).  The common disk is hosted on 
an iSCSI target. Kernelversion is 2.6.18-5-xen-amd64 on all of the dom0s and 
domUs. 

This is the xen16.cfg on both Xenhosts:
***snip***
kernel  = ''/boot/vmlinuz-2.6.18-5-xen-amd64''
ramdisk = ''/boot/initrd.img-2.6.18-5-xen-amd64''
memory  = ''128''
maxmem  = ''256''
root    = ''/dev/sda1 ro''
disk    = 
[
''phy:/dev/disk/by-uuid/10875585-5295-4b1c-9043-3c87d052a3e7,sda1,w'',
''phy:/dev/disk/by-uuid/b993558c-933c-4391-a7b8-c6904f934b0d,sda2,w''
]

name    = ''xen16''
vif  = [ ''mac=00:16:3E:40:53:23 , backend=netbackend'' ]
on_poweroff = ''destroy''
on_reboot   = ''restart''
on_crash    = ''restart''
***snip***

I have the feeling that the migrated domU just does not get connected to the 
netbackend domU.  Because there are no network packets caught on either fo 
the firewall backends, I believe that firewall rules are not the problem. (I 
even flushed them to no avail)

Anybody having thoughts on this?  Need more information?

Cheers Jan Behrend

-- 
Jan Behrend
Max-Planck-Institut fuer Radioastronomie
Abteilung fuer Infrarot-Technologie
Auf dem Huegel 69, D-53121 Bonn (Germany)
Tel: (+49) 228 525 319, Fax: (+49) 228 525 411
http://www.mpifr-bonn.mpg.de

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Monday 14 April 2008 13:54, Jan Behrend wrote:
> I have the following setup:
>
> - standard Dom0, _without_ xen bridge
> - netbackend domU as a firewall, i.e. nic is exclusively used by this domU
> via pciback mechanism. This domU hosts the xen bridge and does the network
> setup for the other domUs (/etc/xen/scripts/vif-bridge etc.)
>
> - a bunch of domUs _behind_ this firewall domU
>
> This setup is described in
> http://lists.xensource.com/archives/html/xen-users/2005-07/msg00558.html
>
> Everything is working just fine. So far so good.
> My next goal is to migrate one of the domUs _behind_ the netbackend
> firewall to another machine with the same setup.  I know I cannot migrate
> the netbackend firewall domU.
>
> The migration works fine but unfortunately networking stops completely on
> the migrated domain. Otherwise it works just fine but has no network. This
> is what happens:
>
> On xenhost1 I do:
> # xm list
> # xen16                                     15      128     1 -b----    
> 14.4
>
> # xm migrate xen16 xenhost2 --live
> #
>
> This is what''s then shown on xenhost2:
> # xm list
> # xen16                                     14      128     1 -b----     
> 0.0
>
> The following is the kernel output on the console.  When I hit enter I see
> the login screen again:
> # xm console xen16
> vif vif-0: 2 reading other end details from device/vif/0
> xenbus: resume (talk_to_otherend) vif-0 failed: -2
>
> Both Xenhosts are 64bit Debian Etch (Xen 3.0.3).  The common disk is hosted
> on an iSCSI target. Kernelversion is 2.6.18-5-xen-amd64 on all of the dom0s
> and domUs.
>
> This is the xen16.cfg on both Xenhosts:
> ***snip***
> kernel  = ''/boot/vmlinuz-2.6.18-5-xen-amd64''
> ramdisk = ''/boot/initrd.img-2.6.18-5-xen-amd64''
> memory  = ''128''
> maxmem  = ''256''
> root    = ''/dev/sda1 ro''
> disk    > [
''phy:/dev/disk/by-uuid/10875585-5295-4b1c-9043-3c87d052a3e7,sda1,w'',
>
''phy:/dev/disk/by-uuid/b993558c-933c-4391-a7b8-c6904f934b0d,sda2,w''
]
>
> name    = ''xen16''
> vif  = [ ''mac=00:16:3E:40:53:23 , backend=netbackend'' ]
> on_poweroff = ''destroy''
> on_reboot   = ''restart''
> on_crash    = ''restart''
> ***snip***
>
> I have the feeling that the migrated domU just does not get connected to
> the netbackend domU.  Because there are no network packets caught on either
> fo the firewall backends, I believe that firewall rules are not the
> problem. (I even flushed them to no avail)
>
> Anybody having thoughts on this?  Need more information?
>
> Cheers Jan Behrend
Anyone?  I just need a pointer in the right direction.  I figure out the rest 
and report.

-- 
Jan Behrend
Max-Planck-Institut fuer Radioastronomie
Abteilung fuer Infrarot-Technologie
Auf dem Huegel 69, D-53121 Bonn (Germany)
Tel: (+49) 228 525 319, Fax: (+49) 228 525 411
jbehrend@mpifr-bonn.mpg.de http://www.mpifr-bonn.mpg.de

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Whoops, forgot to post to the list:

I''m just taking a totally blind stab, but here it goes:

If I''m reading correctly, your Firewall DomU is providing NAT for the
other
DomUs.  Is the IP interface that faces the other DomUs the same IP on both
machines?  IE, do ALL DomUs, regardless of physical machine, use the same
gateway IP address?


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wednesday 23 April 2008 09:01, you wrote:
> Whoops, forgot to post to the list:
>
> I''m just taking a totally blind stab, but here it goes:
>
> If I''m reading correctly, your Firewall DomU is providing NAT for
the other
> DomUs.  Is the IP interface that faces the other DomUs the same IP on both
> machines?  IE, do ALL DomUs, regardless of physical machine, use the same
> gateway IP address?
Thanks for the answer.
The netbackend domUs are _not_ doing NAT.  They just keep unwanted traffic 
out.  If I flush all iptables rules and merely do forwarding on the 
netbackend domUs I get the same effect.
To answer your question: Yes, all domUs, regardless of their physical host, 
use the same gateway IP.  In fact all domUs and the physical machines are on 
the same subnet.

Jan

-- 
Jan Behrend
Max-Planck-Institut fuer Radioastronomie
Abteilung fuer Infrarot-Technologie
Auf dem Huegel 69, D-53121 Bonn (Germany)
Tel: (+49) 228 525 319, Fax: (+49) 228 525 411
jbehrend@mpifr-bonn.mpg.de http://www.mpifr-bonn.mpg.de

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Monday 14 April 2008 13:54, Jan Behrend wrote:
> I have the following setup:
>
> - standard Dom0, _without_ xen bridge
> - netbackend domU as a firewall, i.e. nic is exclusively used by this domU
> via pciback mechanism. This domU hosts the xen bridge and does the network
> setup for the other domUs (/etc/xen/scripts/vif-bridge etc.)
>
> - a bunch of domUs _behind_ this firewall domU
>
> This setup is described in
> http://lists.xensource.com/archives/html/xen-users/2005-07/msg00558.html
>
> Everything is working just fine. So far so good.
> My next goal is to migrate one of the domUs _behind_ the netbackend
> firewall to another machine with the same setup.  I know I cannot migrate
> the netbackend firewall domU.
>
> The migration works fine but unfortunately networking stops completely on
> the migrated domain. Otherwise it works just fine but has no network. This
> is what happens:
>
> On xenhost1 I do:
> # xm list
> # xen16                                     15      128     1 -b----    
> 14.4
>
> # xm migrate xen16 xenhost2 --live
> #
>
> This is what''s then shown on xenhost2:
> # xm list
> # xen16                                     14      128     1 -b----     
> 0.0
>
> The following is the kernel output on the console.  When I hit enter I see
> the login screen again:
> # xm console xen16
> vif vif-0: 2 reading other end details from device/vif/0
> xenbus: resume (talk_to_otherend) vif-0 failed: -2
>
> Both Xenhosts are 64bit Debian Etch (Xen 3.0.3).  The common disk is hosted
> on an iSCSI target. Kernelversion is 2.6.18-5-xen-amd64 on all of the dom0s
> and domUs.
>
> This is the xen16.cfg on both Xenhosts:
> ***snip***
> kernel  = ''/boot/vmlinuz-2.6.18-5-xen-amd64''
> ramdisk = ''/boot/initrd.img-2.6.18-5-xen-amd64''
> memory  = ''128''
> maxmem  = ''256''
> root    = ''/dev/sda1 ro''
> disk    > [
''phy:/dev/disk/by-uuid/10875585-5295-4b1c-9043-3c87d052a3e7,sda1,w'',
>
''phy:/dev/disk/by-uuid/b993558c-933c-4391-a7b8-c6904f934b0d,sda2,w''
]
>
> name    = ''xen16''
> vif  = [ ''mac=00:16:3E:40:53:23 , backend=netbackend'' ]
> on_poweroff = ''destroy''
> on_reboot   = ''restart''
> on_crash    = ''restart''
> ***snip***
>
> I have the feeling that the migrated domU just does not get connected to
> the netbackend domU.  Because there are no network packets caught on either
> fo the firewall backends, I believe that firewall rules are not the
> problem. (I even flushed them to no avail)
>
> Anybody having thoughts on this?  Need more information?
>
> Cheers Jan Behrend
Hot Diggity, I made it!

Finally migration in the scenario described above works.
First of all I upgraded to Xen 3.2.  The downtime during live migration with 
this method is not milliseconds but rather a couple of seconds.  This is 
because the network device needs to be reattached to the netbackend.
Other than this it works beautifully.   Here it goes:

This describes a live migration of the xen16 DomU from Dom0 Xenhost1 to Dom0 
Xenhost2

1. Detach the network device from the DomU on Xenhost1
2. Take down the network in the DomU
3. do the actual migration from Xenhost1 to Xenhost2
4. Attach the network device to the DomU on Xenhost2
5. Bring up the network device in the DomU

Done!
If you use a script, pings to the DomU are disrupted for 1 to 5 seconds 
depending on the DomU''s RAM size.

Since this is all done via network, taking down the network interface in the 
domU leaves you with the problem of bringing it up again via network.  The 
trick is to look for the "/sys/class/net/eth0" which disappears while 
detaching the network device in the Dom0 and reappears while attaching the 
network device in the other Dom0 once the migration has taken place.

Here is the script I wrote to do this fairly quickly.  This is not failsafe, 
so you might wat to add safety measures to the it.  Anyway, this is just 
meant as an inspiration. You''ll need passwordless ssh logins on all
machines.
I use Kerberos for this.

#!/bin/bash
### Fix the line breaks which occured while posting to the list ###

function usage {
   echo "usage: ${0##*/} <Domain> <Host>"
   exit 0
}

if [ $# -ne 2 ] ; then
    usage
fi

if [ ! $(xm domid $1) ] ; then
   echo "Dying ..."
   exit 1
fi

echo "Timing network wakeup for DomU \"$1\""
ssh $1 ''while [ -d /sys/class/net/eth0 ] ; do sleep 1 ; done ; ifdown
eth0 ;
until [ -d /sys/class/net/eth0 ] ; do sleep 1 ; done ; ifup eth0'' &
echo "Detaching network device from DomU \"$1\""
MAC_ADDR=$(xm network-list $1 | tail -n1 | awk ''{print $3}'')
for fn in $(xm network-list $1 | tail -n-1 | awk ''{print $1}'')
; do
   xm network-detach $1 $fn -f
done
echo "Actually migrating \"$1\" to \"$2\""
xm migrate $1 $2 -l
echo "Attaching network device to DomU \"$1\""
ssh $2 "xm network-attach $1 mac=$MAC_ADDR backend=netbackend"
while [ $(ssh $2 "xm network-list $1" | tail -n1 | awk
''{print $5}'') -ne 4 ] ;
do
   echo "Attaching network device to DomU \"$1\" FAILED!!!
Retrying ... (Kill
me with C-c if you wish!)"
   for fn in $(ssh $2 "xm network-list $1" | tail -n-1 | awk
''{print $1}'') ;
do
      ssh $2 "xm network-detach $1 $fn -f"
   done
   ssh $2 "xm network-attach $1 mac=$MAC_ADDR backend=netbackend"
done
echo "Done! (Disregard SIOCDELRT error message)"
echo

Cheers Jan
-- 
Jan Behrend
Max-Planck-Institut fuer Radioastronomie
Abteilung fuer Infrarot-Technologie
Auf dem Huegel 69, D-53121 Bonn (Germany)
Tel: (+49) 228 525 319, Fax: (+49) 228 525 411
jbehrend@mpifr-bonn.mpg.de http://www.mpifr-bonn.mpg.de

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users