Cole Robinson
2014-Apr-16 18:17 UTC
Re: [libvirt-users] [virt-tools-list] live-migrating a VM whose current host's storage is fubar'ed
On 04/15/2014 05:16 PM, Digimer wrote:> Hi all, > > For those on the linux-cluster list, this is a similar question to my > post[1] there. > > I recently had a client destroy their node's RAID array (they yanked hard > drives until the array was destroyed). The VMs running on the node continued > to operate for another hour, impressively. Their storage was based on DRBD, so > the VM's storage just started going to/from the other node's storage. > > The problem was when I tried to migrate them off of the damaged node to a > healthy node. Without being able to write to the logs (I assume), the > migration failed. In the end, I had to force-off the damaged host and reboot > the VMs on the backup node. This caused an outage which, I think, could have > been avoided. > > So my question is; Can VMs be migrated if/when the host's FS is > read-only/destroyed? If so, how? > > Thanks! > > 1. https://www.redhat.com/archives/linux-cluster/2014-April/msg00010.html >Probably a better question for libvirt lists, CCd, but I don't think anyone tests a scenario like that so there's bound to be problems. - Cole
Digimer
2014-Apr-16 18:39 UTC
Re: [libvirt-users] [virt-tools-list] live-migrating a VM whose current host's storage is fubar'ed
On 16/04/14 02:17 PM, Cole Robinson wrote:> On 04/15/2014 05:16 PM, Digimer wrote: >> Hi all, >> >> For those on the linux-cluster list, this is a similar question to my >> post[1] there. >> >> I recently had a client destroy their node's RAID array (they yanked hard >> drives until the array was destroyed). The VMs running on the node continued >> to operate for another hour, impressively. Their storage was based on DRBD, so >> the VM's storage just started going to/from the other node's storage. >> >> The problem was when I tried to migrate them off of the damaged node to a >> healthy node. Without being able to write to the logs (I assume), the >> migration failed. In the end, I had to force-off the damaged host and reboot >> the VMs on the backup node. This caused an outage which, I think, could have >> been avoided. >> >> So my question is; Can VMs be migrated if/when the host's FS is >> read-only/destroyed? If so, how? >> >> Thanks! >> >> 1. https://www.redhat.com/archives/linux-cluster/2014-April/msg00010.html >> > > Probably a better question for libvirt lists, CCd, but I don't think anyone > tests a scenario like that so there's bound to be problems. > > - ColeThanks for cc'ing. Ya, I'm not surprised that I had problems. I guess now that I've hit it though, and given that it's not impossible to have this happen again, it would be worth looking in to a way to handle it. I suspect something like "--no-logging" to disable attempts to write to local disk during a migration would be enough. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education?