libvirt users - May 2016 - migrate local storage to ceph | exchanging the storage system

TLDR: Why is virsh migrate --persistent --live domain
qemu+ssh://root@host/system --xml domain.ceph.xml
not persistent and what could i do about it?

Hi,
 after years of beeing pleased with local storage and migrating the
complete storage from one host to another, it was time for ceph.

After setting up a cluster and testing it, its time now for moving a lot
of VMs on that type of storage, without a disturbance if possible.

So I 'virsh dumpxml domain > domain.ceph.xml', edit the file, replace
// ---- old ---- //
<disk type='file' device='disk'>
  <driver name='qemu' type='qcow2' cache='none'/>
  <source file='domain.img'/>
  <backingStore/>
  <target dev='vda' bus='virtio'/>
  <alias name='virtio-disk0'/>
  <address type='pci' domain='0x0000' bus='0x00'
slot='0x04'
function='0x0'/>
</disk>

// ---- with new ---- //
<disk type='network' device='disk'>
  <driver name='qemu' type='raw'/>
  <auth username='libvirt'>
    <secret type='ceph'
uuid='53e4ba03-5ad0-4f1e-b69e-897afba272ff'/>
  </auth>
  <source protocol='rbd' name='libvirt/domain'>
    <host name='cephmon0' port='6789'/>
    <host name='cephmon1' port='6789'/>
    <host name='cephmon2' port='6789'/>
  </source>
  <backingStore/>
  <target dev='vda' bus='virtio'/>
  <boot order='1'/>
  <alias name='virtio-disk0'/>
  <address type='pci' domain='0x0000' bus='0x00'
slot='0x04'
function='0x0'/>
</disk>

Now i'm calling:

root@host_a:~# virsh migrate --verbose --p2p --copy-storage-all
--persistent --change-protection --abort-on-error --undefinesource
--live domain qemu+ssh://root@host_b/system --xml domain.ceph.xml

This takes its time to copy everything into ceph, but when its over, i
never have to migrate Terabytes of Storage again.

But its not over.
The new domain definition on host_b is not persistent and after a
shutdown, local storage is used again.

What do i have todo, to make this change persistent, or is there a
better way for doing this?

(i am using libvirt 1.2.9)

Thanks in advance
Björn Lässig
-- 
Pengutronix e.K.                         |                             |
Industrial Linux Solutions               | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim        | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686         | Fax:   +49-5121-206917-5555 |

> -----Oorspronkelijk bericht-----
> Van: libvirt-users-bounces@redhat.com [mailto:libvirt-users-
> bounces@redhat.com] Namens Björn Lässig
> Verzonden: vrijdag 27 mei 2016 10:10
> Aan: libvirt-users@redhat.com
> Onderwerp: [libvirt-users] migrate local storage to ceph | exchanging the
> storage system
> 
> TLDR: Why is virsh migrate --persistent --live domain
> qemu+ssh://root@host/system --xml domain.ceph.xml
> not persistent and what could i do about it?
> 
> Hi,
>  after years of beeing pleased with local storage and migrating the
complete
> storage from one host to another, it was time for ceph.
> 
> After setting up a cluster and testing it, its time now for moving a lot of
VMs on
> that type of storage, without a disturbance if possible.
> 
> So I 'virsh dumpxml domain > domain.ceph.xml', edit the file,
replace // ---- old
> ---- // <disk type='file' device='disk'>
>   <driver name='qemu' type='qcow2'
cache='none'/>
>   <source file='domain.img'/>
>   <backingStore/>
>   <target dev='vda' bus='virtio'/>
>   <alias name='virtio-disk0'/>
>   <address type='pci' domain='0x0000' bus='0x00'
slot='0x04'
> function='0x0'/>
> </disk>
> 
> // ---- with new ---- //
> <disk type='network' device='disk'>
>   <driver name='qemu' type='raw'/>
>   <auth username='libvirt'>
>     <secret type='ceph'
uuid='53e4ba03-5ad0-4f1e-b69e-897afba272ff'/>
>   </auth>
>   <source protocol='rbd' name='libvirt/domain'>
>     <host name='cephmon0' port='6789'/>
>     <host name='cephmon1' port='6789'/>
>     <host name='cephmon2' port='6789'/>
>   </source>
>   <backingStore/>
>   <target dev='vda' bus='virtio'/>
>   <boot order='1'/>
>   <alias name='virtio-disk0'/>
>   <address type='pci' domain='0x0000' bus='0x00'
slot='0x04'
> function='0x0'/>
> </disk>
> 
> Now i'm calling:
> 
> root@host_a:~# virsh migrate --verbose --p2p --copy-storage-all
--persistent --
> change-protection --abort-on-error --undefinesource --live domain
> qemu+ssh://root@host_b/system --xml domain.ceph.xml
Weird: The domain should be persistent

You can make an transient domain persistent by doing a 'virsh define
domain.ceph.xml'

> 
> This takes its time to copy everything into ceph, but when its over, i
never have
> to migrate Terabytes of Storage again.
> 
> But its not over.
> The new domain definition on host_b is not persistent and after a shutdown,
> local storage is used again.
> 
> What do i have todo, to make this change persistent, or is there a better
way
> for doing this?
> 
> (i am using libvirt 1.2.9)
> 
> Thanks in advance
> Björn Lässig
> --
> Pengutronix e.K.                         |                             |
> Industrial Linux Solutions               | http://www.pengutronix.de/  |
> Peiner Str. 6-8, 31137 Hildesheim        | Phone: +49-5121-206917-0    |
> Amtsgericht Hildesheim, HRA 2686         | Fax:   +49-5121-206917-5555 |
> 
> _______________________________________________
> libvirt-users mailing list
> libvirt-users@redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users

On 05/30/2016 09:07 AM, Dominique Ramaekers wrote:
>> root@host_a:~# virsh migrate --verbose --p2p --copy-storage-all
--persistent --
>> change-protection --abort-on-error --undefinesource --live domain
>> qemu+ssh://root@host_b/system --xml domain.ceph.xml
> 
> Weird: The domain should be persistent
Well, the domain is persistent. But the changes i did to domain.ceph.xml
to migrate the storage to ceph are not.
> You can make an transient domain persistent by doing a 'virsh define
domain.ceph.xml'
This works for one vm, but this needs a shutdown/reboot sequence.
It is really painful to coordinate this for every VM, so i'd like to
skip this.

Björn Lässig