libvirt users - Apr 2017 - Live migration with non-shared ZFS volume

Hi all,

I'm using ZFS on Linux block volumes as my VM storage and want to do live
migrations between hypervisors.

If I create ZFS snapshot of used volume on source host, send it do
destination host (zfs send/recv) and then run live migration with
VIR_MIGRATE_NON_SHARED_DISK
flag, the migration works OK.

But this procedure copies the whole disk twice which is a huge downside.

The best solution would be, if libvirt could send the incremental data
since last snapshot itself but this feature is not there (AFAIK).

So I am thinking about a workaround:
1. Create snapshot using: "virsh snapshot-create --xmlfile snap.xml
--disk-only --no-metadata test-domain" which will start writing snapshot
data into temporary qcow2 file

<domainsnapshot>
  <disks>
    <disk name='/dev/zstore/test-volume'>
      <source file='/tmp/test-volume.qcow2'/>
    </disk>
  </disks>
</domainsnapshot>


2. Create snapshot of backing ZFS volume and send it to destination host.
3. Migrate the domain

Currently, in step 3 I need to create empty qcow snapshot file on the
destination host, otherwise the migration fails with: "Operation not
supported: pre-creation of storage targets for incremental storage
migration is not supported"

My question is: Is it possible to do live migration with blockcommit
operation? If not, would it be hard to implement?

I imagine it like I would start migration with some special parameter (e.g.
VIR_MIGRATE_NON_SHARED_INC_COMMIT) which would only migrate data from qcow
snapshot to destination storage.

This would ensure the disk consistency and avoid useless whole disk copy.

Or do you have any other idea how to solve this?

BR.
Daniel Kucera.

On Tue, Apr 04, 2017 at 12:04:42PM +0200, Daniel Kučera
wrote:
>Hi all,
>
Hi,

I caught your mail in my Spam folder for some reason, maybe the same
happened for others.  I don't have that deep knowledge of the snapshots,
but I'm replying so that if someone else has it in Spam and they have
more insight, they can reply.
>I'm using ZFS on Linux block volumes as my VM storage and want to do
live
>migrations between hypervisors.
>
>If I create ZFS snapshot of used volume on source host, send it do
>destination host (zfs send/recv) and then run live migration with
>VIR_MIGRATE_NON_SHARED_DISK
>flag, the migration works OK.
>
>But this procedure copies the whole disk twice which is a huge downside.
>
>The best solution would be, if libvirt could send the incremental data
>since last snapshot itself but this feature is not there (AFAIK).
>
>So I am thinking about a workaround:
>1. Create snapshot using: "virsh snapshot-create --xmlfile snap.xml
>--disk-only --no-metadata test-domain" which will start writing
snapshot
>data into temporary qcow2 file
>
><domainsnapshot>
>  <disks>
>    <disk name='/dev/zstore/test-volume'>
>      <source file='/tmp/test-volume.qcow2'/>
>    </disk>
>  </disks>
></domainsnapshot>
>
>
>2. Create snapshot of backing ZFS volume and send it to destination host.
>3. Migrate the domain
>
>Currently, in step 3 I need to create empty qcow snapshot file on the
>destination host, otherwise the migration fails with: "Operation not
>supported: pre-creation of storage targets for incremental storage
>migration is not supported"
>
So apart from this it does exactly what you want, right?
>My question is: Is it possible to do live migration with blockcommit
>operation? If not, would it be hard to implement?
>
>I imagine it like I would start migration with some special parameter (e.g.
>VIR_MIGRATE_NON_SHARED_INC_COMMIT) which would only migrate data from qcow
>snapshot to destination storage.
>
>This would ensure the disk consistency and avoid useless whole disk copy.
>
What would be the difference compared to what you are doing now (plus a
block commit)?  VIR_MIGRATE_NON_SHARED_INC should already migrate only
the topmost disk of the whole chain.

Of course lot can be added to the project, but I, for one, am not *that*
welcoming with regards to things that are meant for pretty narrow use
case while at the same time they are doable already, using libvirt's API.
>Or do you have any other idea how to solve this?
>
>BR.
>Daniel Kucera.
>_______________________________________________
>libvirt-users mailing list
>libvirt-users@redhat.com
>https://www.redhat.com/mailman/listinfo/libvirt-users

Hi Martin,

in the meantime, I've found a solution which I consider at least acceptable:

1. create zfs snapshot of domain disk (/dev/zstore/test-volume)

2. save original XML domain definition

3. create snapshot in libvirt like this:
virsh snapshot-create --xmlfile snap.xml --disk-only --no-metadata
test-domain
snap.xml:
<domainsnapshot>
  <disks>
    <disk name='/dev/zstore/test-volume'>
      <source file='/tmp/test-volume.qcow2'/>
    </disk>
  </disks>
</domainsnapshot>

This creates overlay qcow snapshot on top of my base ZFS volume

3. Transfer ZFS volume (with last snapshot) to remote host

4. Run live migration with setting XML from step 1 as DestXML and PersistXML
parameter and with VIR_MIGRATE_NON_SHARED_INC flag

This creates domain without snapshot on the remote host and migrates only
data from qcow overlay and commits them to the ZFS base volume.

What I don't like about this, is that I need to create and care about qcow
image.

The best solution for me would be to only create dirty block map in qemu
(block-dirty-bitmap-add) in step 3 and then migrate just changed blocks
without creating qcow snapshot.

Do you think it would be possible to implement it like this?

S pozdravom / Best regards
Daniel Kucera

2017-04-28 11:03 GMT+02:00 Martin Kletzander <mkletzan@redhat.com>:
> On Tue, Apr 04, 2017 at 12:04:42PM +0200, Daniel Kučera wrote:
>
>> Hi all,
>>
>>
> Hi,
>
> I caught your mail in my Spam folder for some reason, maybe the same
> happened for others.  I don't have that deep knowledge of the
snapshots,
> but I'm replying so that if someone else has it in Spam and they have
> more insight, they can reply.
>
>
> I'm using ZFS on Linux block volumes as my VM storage and want to do
live
>> migrations between hypervisors.
>>
>> If I create ZFS snapshot of used volume on source host, send it do
>> destination host (zfs send/recv) and then run live migration with
>> VIR_MIGRATE_NON_SHARED_DISK
>> flag, the migration works OK.
>>
>> But this procedure copies the whole disk twice which is a huge
downside.
>>
>> The best solution would be, if libvirt could send the incremental data
>> since last snapshot itself but this feature is not there (AFAIK).
>>
>> So I am thinking about a workaround:
>> 1. Create snapshot using: "virsh snapshot-create --xmlfile
snap.xml
>> --disk-only --no-metadata test-domain" which will start writing
snapshot
>> data into temporary qcow2 file
>>
>> <domainsnapshot>
>>  <disks>
>>    <disk name='/dev/zstore/test-volume'>
>>      <source file='/tmp/test-volume.qcow2'/>
>>    </disk>
>>  </disks>
>> </domainsnapshot>
>>
>>
>> 2. Create snapshot of backing ZFS volume and send it to destination
host.
>> 3. Migrate the domain
>>
>> Currently, in step 3 I need to create empty qcow snapshot file on the
>> destination host, otherwise the migration fails with: "Operation
not
>> supported: pre-creation of storage targets for incremental storage
>> migration is not supported"
>>
>>
> So apart from this it does exactly what you want, right?
>
> My question is: Is it possible to do live migration with blockcommit
>> operation? If not, would it be hard to implement?
>>
>> I imagine it like I would start migration with some special parameter
>> (e.g.
>> VIR_MIGRATE_NON_SHARED_INC_COMMIT) which would only migrate data from
>> qcow
>> snapshot to destination storage.
>>
>> This would ensure the disk consistency and avoid useless whole disk
copy.
>>
>>
> What would be the difference compared to what you are doing now (plus a
> block commit)?  VIR_MIGRATE_NON_SHARED_INC should already migrate only
> the topmost disk of the whole chain.
>
> Of course lot can be added to the project, but I, for one, am not *that*
> welcoming with regards to things that are meant for pretty narrow use
> case while at the same time they are doable already, using libvirt's
API.
>
> Or do you have any other idea how to solve this?
>>
>> BR.
>> Daniel Kucera.
>>
>
> _______________________________________________
>> libvirt-users mailing list
>> libvirt-users@redhat.com
>> https://www.redhat.com/mailman/listinfo/libvirt-users
>>
>