Hi all, I want to do live migration of domains using DRBD block devices for synchronisation between the two Xen hosts with Xen 2.0.7. I''ve written a script ( block-drbd ) which binds / unbinds DRBD devices for specific domains (with DRBD, only one host can write on a drbd device at the same time). This script sets the current host as master (bind) or secondary (unbind) in a DRBD "Netword-RAID" Bringing up a domain works fine. But when I try to migrate it ( live with -l flag as well as non-live) I get an error from xfrd. The xfrd.log says: ==> xfrd.log <=2508 [INF] XFRD> Accepted connection from 127.0.0.1:1327 on 2 2790 [INF] XFRD> Xfr service for 127.0.0.1:1327 [DEBUG] Conn_init> flags=1 [DEBUG] Conn_init> write stream... [DEBUG] stream_init>mode=w flags=1 compress=0 [DEBUG] stream_init> unbuffer... [DEBUG] stream_init< err=0 [DEBUG] Conn_init> read stream... [DEBUG] stream_init>mode=r flags=1 compress=0 [DEBUG] stream_init> unbuffer... [DEBUG] stream_init< err=0 [DEBUG] Conn_sxpr> (xfr.hello 1 0)[DEBUG] Conn_sxpr< err=0 [DEBUG] Conn_sxpr> (xfr.migrate 8 "(domain (id 8) (name Xen0) (memory 511) (maxmem 512) (state -b---) (cpu 0) (cpu_time 7.548731537) (up_time 1953.19617701) (start_time 1130242370.96) (console (status listening) (id 12) (domain 8) (local_port 12) (remote_port 1) (console_port 9608)) (devices (vif (idx 0) (vif 0) (mac aa:00:00:00:42:00) (vifname vif8.0) (bridge xen-br0) (evtchn 14 4) (index 0)) (vbd (idx 0) (vdev 769) (device 37632) (mode w) (dev hda1) (uname drbd:drbd0) (node /dev/drbd0) (index 0)) (vbd (idx 1) (vdev 770) (device 37633) (mode w) (dev hda2) (uname drbd:drbd1) (node /dev/drbd1) (index 1))) (config (vm (name Xen0) (memory 512) (image (linux (kernel /boot/vmlinuz-2.6.11.10-xenU) (root ''/dev/hda1 ro''))) (device (vbd (uname drbd:drbd0) (dev hda1) (mode w))) (device (vbd (uname drbd:drbd1) (dev hda2) (mode w))) (device (vif (mac aa:00:00:00:42:00) (bridge xen-br0))))))" testpc-023 8002 1 0)[DEBUG] Conn_sxpr< err=0 [DEBUG] Conn_connect> addr=192.168.111.23:8002 [DEBUG] Conn_init> flags=1 [DEBUG] Conn_init> write stream... [DEBUG] stream_init>mode=w flags=1 compress=0 [DEBUG] stream_init> unbuffer... [DEBUG] stream_init< err=0 [DEBUG] Conn_init> read stream... [DEBUG] stream_init>mode=r flags=1 compress=0 [DEBUG] stream_init> unbuffer... [DEBUG] stream_init< err=0 [DEBUG] Conn_sxpr> (xfr.err 0)[DEBUG] Conn_sxpr< err=0 [1130244324.172538] xc_linux_save start 8 xc_linux_save start 8 [1130244324.174077] Saving memory pages: iter 1 0% Saving memory pages: iter 1 0%Error when writing to state file (5) Error when writing to state file (5) 2790 [WRN] XFRD> Transfer errors: 2790 [WRN] XFRD> state=XFR_STATE err=1 2790 [INF] XFRD> Xfr service err=1 Does anybody have an idea? I''d appreciate. Cheers, Mike _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> I want to do live migration of domains using DRBD block devices for > synchronisation between the two Xen hosts with Xen 2.0.7. > > I''ve written a script ( block-drbd ) which binds / unbinds DRBD devices for > specific domains (with DRBD, only one host can write on a drbd device at > the same time). This script sets the current host as master (bind) or > secondary (unbind) in a DRBD "Netword-RAID"Yep - btw this is fixed with DRBD 8 (still in development), which supports multiple writers. How are you handling the sequencing here, though? Move the device to the new host at *unbind* time from the origin? I''m thinking that the DRBD script will get run at the destination whilst the origin domain is still running and using its disk...> xc_linux_save start 8 > [1130244324.174077] Saving memory pages: iter 1 0% > Saving memory pages: iter 1 0%Error when writing to state file (5) > Error when writing to state file (5) > 2790 [WRN] XFRD> Transfer errors: > 2790 [WRN] XFRD> state=XFR_STATE err=1 > 2790 [INF] XFRD> Xfr service err=1 > > > Does anybody have an idea? I''d appreciate.Weird... Does this work without your script? Does migration work with a simpler domain? If you boot a domain from an initrd (removing disk dependencies), will that migrate correctly? If it''s something weird in the migration-related code, somebody else might have some further ideas as to what caused this... Cheers, Mark _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Mark Williamson wrote:>>I want to do live migration of domains using DRBD block devices for >>synchronisation between the two Xen hosts with Xen 2.0.7. >> >>I''ve written a script ( block-drbd ) which binds / unbinds DRBD devices for >>specific domains (with DRBD, only one host can write on a drbd device at >>the same time). This script sets the current host as master (bind) or >>secondary (unbind) in a DRBD "Netword-RAID" > > > Yep - btw this is fixed with DRBD 8 (still in development), which supports > multiple writers. > > How are you handling the sequencing here, though? Move the device to the new > host at *unbind* time from the origin? I''m thinking that the DRBD script > will get run at the destination whilst the origin domain is still running and > using its disk...Correct. And it cannot work with drbd 0.7 currently without hacking Xen to change block device setup. What would be needed is that Xen should open the new block device read-only (there is a hack in drbd 0.7 that would allow this) and only when the source domain is shutdown reopen the block device r/w. Xen want''s to open the block device rw as soon as it does the setup of the destination domain. This may work currectly with drbd 0.8 since it supports multi-master. I have done testing with the current development trunk of drbd some time ago. It did not work for me. Migration crashed the guest. Best Regards, Michael Paesold _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tuesday 25 October 2005 16:49, Mark Williamson wrote:> > I want to do live migration of domains using DRBD block devices for > > synchronisation between the two Xen hosts with Xen 2.0.7. > > > > I''ve written a script ( block-drbd ) which binds / unbinds DRBD devices > > for specific domains (with DRBD, only one host can write on a drbd device > > at the same time). This script sets the current host as master (bind) or > > secondary (unbind) in a DRBD "Netword-RAID" > > Yep - btw this is fixed with DRBD 8 (still in development), which supports > multiple writers.I''ll give it a try, although I''m expecting it being not that successful because of its development state and Michael''s experiences.> How are you handling the sequencing here, though? Move the device to the > new host at *unbind* time from the origin? I''m thinking that the DRBD > script will get run at the destination whilst the origin domain is still > running and using its disk...So the solution would be to get Xen mounting the device in the following order for migration as Michael Paesold suggested: 1) create domain on target host and mount the drbd device r/o 2) do the rest of the migration, e.g. iterative copy of memory ... 3) unmount or mount the drbd device on the source host r/o 4) mount drbd device r/w on target host 5) start domain on target host Is there any further developer''s documentation for Xen 2.0.7? Or maybe anybody can give me some hints at approx. which points I''d have to patch Xen to get that running? That would help me a lot saving some time for trying to implement that because I''m not so deep into Xen right now.> > xc_linux_save start 8 > > [1130244324.174077] Saving memory pages: iter 1 0% > > Saving memory pages: iter 1 0%Error when writing to state file (5) > > Error when writing to state file (5) > > 2790 [WRN] XFRD> Transfer errors: > > 2790 [WRN] XFRD> state=XFR_STATE err=1 > > 2790 [INF] XFRD> Xfr service err=1 > > > > > > Does anybody have an idea? I''d appreciate. > > Weird... Does this work without your script? Does migration work with a > simpler domain? If you boot a domain from an initrd (removing disk > dependencies), will that migrate correctly? > > If it''s something weird in the migration-related code, somebody else might > have some further ideas as to what caused this...Migration without my script works fine, so there shouldn''t be anything weird in the migration-related code. Regards, Mike _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel