libvirt users - Nov 2014 - Re: backup KVM qcow2 over btrfs or zfs

Hi everybody,

I'm trying to implement in the non profit organization where I work a
backup strategy for our VMs.

At the moment I weekly backup the machine (on sunday nights) by stopping
the vm, making a snapshot, exporting the xml descriptor file and syncing
these files to a remote backup server.

I would really like to make daily snapshot without shutting down the vms.
at the moment the vms qcow2 disks are stored over an ext4 partition but
I'm planning to move them over a das with a new generation filesystem.

I'm trying out zfs and btrfs looking for the best one (at the moment zfs
seems to be more stable and somehow easy, flexible, while btrfs seems to
have better performance on my linux box).

I'm choosing one of these two FS in order to use theire instant-snapshot
feature;
my project will be to  pause the vm, dump the memory and the xml
descriptor file, make a snapshot of the subvolume storing the qcow2 file
and the resuming the machine back (in I hope less than 10 seconds of
downtime).

what do you guys suggest me?

Am I going to experience bad performance having qcow2 over btrfs?
I was reading something like that but the posts were really backdated...

any hints is really appreciated

Thank you

Francesco

On 11/06/2014 03:45 PM, Francesco Morosinotto wrote:
> Hi everybody,
> 
> I'm trying to implement in the non profit organization where I work a
> backup strategy for our VMs.
> 
> At the moment I weekly backup the machine (on sunday nights) by stopping
> the vm, making a snapshot, exporting the xml descriptor file and syncing
> these files to a remote backup server.
You should really consider using libvirt live snapshots.  With new
enough libvirt and qemu, you can even get optimal behavior with no guest
downtime.  This topic comes up frequently on the list; for example, a
quick search found this in the archives:

https://www.redhat.com/archives/libvirt-users/2014-October/msg00055.html
> 
> I would really like to make daily snapshot without shutting down the vms.
Doable.  Next question - do you want the snapshot to merely be disk
contents (and if so, do you want to quiesce the file system before
taking the snapshot, to avoid the need to fsck the disks before
restoring from them), or the more space-intensive full system snapshot
(where you don't need to quiesce, because reverting involves reloading
the RAM of the guest at the time the snapshot was taken)?
> at the moment the vms qcow2 disks are stored over an ext4 partition but
> I'm planning to move them over a das with a new generation filesystem.
> 
> I'm trying out zfs and btrfs looking for the best one (at the moment
zfs
> seems to be more stable and somehow easy, flexible, while btrfs seems to
> have better performance on my linux box).
btrfs has lousy behavior when coupled with qcow2 images (combining two
different layers that are both trying to do copy-on-write tends to lead
to horrible fragmentation).  There is an ioctl to tell btrfs to not do
copy-on-write for files used by snapshots, so you can get performance
back, but it is not default and puts some burden on you to set up.

I don't know enough about zfs snapshot to have an opinion.

You may not need file system snapshot if you instead choose to go with
libvirt live snapshot.
> 
> I'm choosing one of these two FS in order to use theire
instant-snapshot
> feature;
> my project will be to  pause the vm, dump the memory and the xml
> descriptor file, make a snapshot of the subvolume storing the qcow2 file
> and the resuming the machine back (in I hope less than 10 seconds of
> downtime).
Not ideal.  This only takes a disk snapshot, but the state of the disk
is not consistent (you didn't quiesce), so you lose on any pending I/O
that the OS had not yet flushed to disk.  Furthermore, this requires
pausing the VM manually, which is more downtime than you will have if
you use libvirt's live snapshot mechanisms.
> 
> what do you guys suggest me?
> 
> Am I going to experience bad performance having qcow2 over btrfs?
> I was reading something like that but the posts were really backdated...
As far as I know, this is still a real problem unless you manually
configure btrfs to not do copy-on-write, at least when qcow2 images are
involved.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thank you a lot for your reply;
> You should really consider using libvirt live snapshots.  With new
> enough libvirt and qemu, you can even get optimal behavior with no guest
> downtime.  This topic comes up frequently on the list; for example, a
> quick search found this in the archives:
>
> https://www.redhat.com/archives/libvirt-users/2014-October/msg00055.html
last time I tried to take external atomic disk only snapshot I
encountered bugs
(https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1004606) and
gave up.

In my box I have virsh 1.2.2 and despite this
$ virsh blockcommit test hda --active --verbose --pivot
error: command 'blockcommit' doesn't support option --active and
--pivot
it seems to be a solution;
> Next question - do you want the snapshot to merely be disk
> contents (and if so, do you want to quiesce the file system before
> taking the snapshot, to avoid the need to fsck the disks before
> restoring from them), or the more space-intensive full system snapshot
I didn't know there were two options; I always thought that dumping disk
AND ram was the only solution to have a working backup. what do you
suggest me? I would probably feel more safe dumping also the memory (my
guests are ubuntu boxes and win2008)  but I'm not an expert at all...
> btrfs has lousy behavior when coupled with qcow2 images (combining two
> different layers that are both trying to do copy-on-write tends to lead
> to horrible fragmentation).
This idea came into my mind after reading this benchmark
(http://jrs-s.net/2013/05/17/kvm-io-benchmarking/) that suggests the use
of ZFS.

Summing up you suggest me to use libvirt live snapshot instead of
dealing with fs snapshot?
This will mean that instead of building a zfs/btrfs "raid" over my 8
drive with one subvolume for vm taking care of backing up the guest with
its snapshot feature I should build a "classic" raid with a
"classic"
filesystem (ext4?) and delegate to libvirt the snapshotting tasks?

Shouldn't I be worried that a bit rot could lead me to qcow2 degradation?


Thank you

Francesco


Il 06/11/2014 16:02, Eric Blake ha scritto:
> On 11/06/2014 03:45 PM, Francesco Morosinotto wrote:
>> Hi everybody,
>>
>> I'm trying to implement in the non profit organization where I work
a
>> backup strategy for our VMs.
>>
>> At the moment I weekly backup the machine (on sunday nights) by
stopping
>> the vm, making a snapshot, exporting the xml descriptor file and
syncing
>> these files to a remote backup server.
>
> You should really consider using libvirt live snapshots.  With new
> enough libvirt and qemu, you can even get optimal behavior with no guest
> downtime.  This topic comes up frequently on the list; for example, a
> quick search found this in the archives:
>
> https://www.redhat.com/archives/libvirt-users/2014-October/msg00055.html
>
>>
>> I would really like to make daily snapshot without shutting down the
vms.
>
> Doable.  Next question - do you want the snapshot to merely be disk
> contents (and if so, do you want to quiesce the file system before
> taking the snapshot, to avoid the need to fsck the disks before
> restoring from them), or the more space-intensive full system snapshot
> (where you don't need to quiesce, because reverting involves reloading
> the RAM of the guest at the time the snapshot was taken)?
>
>> at the moment the vms qcow2 disks are stored over an ext4 partition but
>> I'm planning to move them over a das with a new generation
filesystem.
>>
>> I'm trying out zfs and btrfs looking for the best one (at the
moment zfs
>> seems to be more stable and somehow easy, flexible, while btrfs seems
to
>> have better performance on my linux box).
>
> btrfs has lousy behavior when coupled with qcow2 images (combining two
> different layers that are both trying to do copy-on-write tends to lead
> to horrible fragmentation).  There is an ioctl to tell btrfs to not do
> copy-on-write for files used by snapshots, so you can get performance
> back, but it is not default and puts some burden on you to set up.
>
> I don't know enough about zfs snapshot to have an opinion.
>
> You may not need file system snapshot if you instead choose to go with
> libvirt live snapshot.
>
>>
>> I'm choosing one of these two FS in order to use theire
instant-snapshot
>> feature;
>> my project will be to  pause the vm, dump the memory and the xml
>> descriptor file, make a snapshot of the subvolume storing the qcow2
file
>> and the resuming the machine back (in I hope less than 10 seconds of
>> downtime).
>
> Not ideal.  This only takes a disk snapshot, but the state of the disk
> is not consistent (you didn't quiesce), so you lose on any pending I/O
> that the OS had not yet flushed to disk.  Furthermore, this requires
> pausing the VM manually, which is more downtime than you will have if
> you use libvirt's live snapshot mechanisms.
>
>>
>> what do you guys suggest me?
>>
>> Am I going to experience bad performance having qcow2 over btrfs?
>> I was reading something like that but the posts were really
backdated...
>
> As far as I know, this is still a real problem unless you manually
> configure btrfs to not do copy-on-write, at least when qcow2 images are
> involved.
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iEYEARECAAYFAlRbqxEACgkQWZ6iCY3lmBSOiQCeMG6E60y1CmnCiBA7Ls9DThIx
D4sAn28Z4q/beDagN/KyKirZRnVEjIMN
=tFx+
-----END PGP SIGNATURE-----