Francesco Morosinotto
2014-Nov-06 14:45 UTC
[libvirt-users] backup KVM qcow2 over btrfs or zfs
Hi everybody, I'm trying to implement in the non profit organization where I work a backup strategy for our VMs. At the moment I weekly backup the machine (on sunday nights) by stopping the vm, making a snapshot, exporting the xml descriptor file and syncing these files to a remote backup server. I would really like to make daily snapshot without shutting down the vms. at the moment the vms qcow2 disks are stored over an ext4 partition but I'm planning to move them over a das with a new generation filesystem. I'm trying out zfs and btrfs looking for the best one (at the moment zfs seems to be more stable and somehow easy, flexible, while btrfs seems to have better performance on my linux box). I'm choosing one of these two FS in order to use theire instant-snapshot feature; my project will be to pause the vm, dump the memory and the xml descriptor file, make a snapshot of the subvolume storing the qcow2 file and the resuming the machine back (in I hope less than 10 seconds of downtime). what do you guys suggest me? Am I going to experience bad performance having qcow2 over btrfs? I was reading something like that but the posts were really backdated... any hints is really appreciated Thank you Francesco
On 11/06/2014 03:45 PM, Francesco Morosinotto wrote:> Hi everybody, > > I'm trying to implement in the non profit organization where I work a > backup strategy for our VMs. > > At the moment I weekly backup the machine (on sunday nights) by stopping > the vm, making a snapshot, exporting the xml descriptor file and syncing > these files to a remote backup server.You should really consider using libvirt live snapshots. With new enough libvirt and qemu, you can even get optimal behavior with no guest downtime. This topic comes up frequently on the list; for example, a quick search found this in the archives: https://www.redhat.com/archives/libvirt-users/2014-October/msg00055.html> > I would really like to make daily snapshot without shutting down the vms.Doable. Next question - do you want the snapshot to merely be disk contents (and if so, do you want to quiesce the file system before taking the snapshot, to avoid the need to fsck the disks before restoring from them), or the more space-intensive full system snapshot (where you don't need to quiesce, because reverting involves reloading the RAM of the guest at the time the snapshot was taken)?> at the moment the vms qcow2 disks are stored over an ext4 partition but > I'm planning to move them over a das with a new generation filesystem. > > I'm trying out zfs and btrfs looking for the best one (at the moment zfs > seems to be more stable and somehow easy, flexible, while btrfs seems to > have better performance on my linux box).btrfs has lousy behavior when coupled with qcow2 images (combining two different layers that are both trying to do copy-on-write tends to lead to horrible fragmentation). There is an ioctl to tell btrfs to not do copy-on-write for files used by snapshots, so you can get performance back, but it is not default and puts some burden on you to set up. I don't know enough about zfs snapshot to have an opinion. You may not need file system snapshot if you instead choose to go with libvirt live snapshot.> > I'm choosing one of these two FS in order to use theire instant-snapshot > feature; > my project will be to pause the vm, dump the memory and the xml > descriptor file, make a snapshot of the subvolume storing the qcow2 file > and the resuming the machine back (in I hope less than 10 seconds of > downtime).Not ideal. This only takes a disk snapshot, but the state of the disk is not consistent (you didn't quiesce), so you lose on any pending I/O that the OS had not yet flushed to disk. Furthermore, this requires pausing the VM manually, which is more downtime than you will have if you use libvirt's live snapshot mechanisms.> > what do you guys suggest me? > > Am I going to experience bad performance having qcow2 over btrfs? > I was reading something like that but the posts were really backdated...As far as I know, this is still a real problem unless you manually configure btrfs to not do copy-on-write, at least when qcow2 images are involved. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Francesco Morosinotto
2014-Nov-06 17:08 UTC
Re: [libvirt-users] backup KVM qcow2 over btrfs or zfs
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Thank you a lot for your reply;> You should really consider using libvirt live snapshots. With new > enough libvirt and qemu, you can even get optimal behavior with no guest > downtime. This topic comes up frequently on the list; for example, a > quick search found this in the archives: > > https://www.redhat.com/archives/libvirt-users/2014-October/msg00055.htmllast time I tried to take external atomic disk only snapshot I encountered bugs (https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1004606) and gave up. In my box I have virsh 1.2.2 and despite this $ virsh blockcommit test hda --active --verbose --pivot error: command 'blockcommit' doesn't support option --active and --pivot it seems to be a solution;> Next question - do you want the snapshot to merely be disk > contents (and if so, do you want to quiesce the file system before > taking the snapshot, to avoid the need to fsck the disks before > restoring from them), or the more space-intensive full system snapshotI didn't know there were two options; I always thought that dumping disk AND ram was the only solution to have a working backup. what do you suggest me? I would probably feel more safe dumping also the memory (my guests are ubuntu boxes and win2008) but I'm not an expert at all...> btrfs has lousy behavior when coupled with qcow2 images (combining two > different layers that are both trying to do copy-on-write tends to lead > to horrible fragmentation).This idea came into my mind after reading this benchmark (http://jrs-s.net/2013/05/17/kvm-io-benchmarking/) that suggests the use of ZFS. Summing up you suggest me to use libvirt live snapshot instead of dealing with fs snapshot? This will mean that instead of building a zfs/btrfs "raid" over my 8 drive with one subvolume for vm taking care of backing up the guest with its snapshot feature I should build a "classic" raid with a "classic" filesystem (ext4?) and delegate to libvirt the snapshotting tasks? Shouldn't I be worried that a bit rot could lead me to qcow2 degradation? Thank you Francesco Il 06/11/2014 16:02, Eric Blake ha scritto:> On 11/06/2014 03:45 PM, Francesco Morosinotto wrote: >> Hi everybody, >> >> I'm trying to implement in the non profit organization where I work a >> backup strategy for our VMs. >> >> At the moment I weekly backup the machine (on sunday nights) by stopping >> the vm, making a snapshot, exporting the xml descriptor file and syncing >> these files to a remote backup server. > > You should really consider using libvirt live snapshots. With new > enough libvirt and qemu, you can even get optimal behavior with no guest > downtime. This topic comes up frequently on the list; for example, a > quick search found this in the archives: > > https://www.redhat.com/archives/libvirt-users/2014-October/msg00055.html > >> >> I would really like to make daily snapshot without shutting down the vms. > > Doable. Next question - do you want the snapshot to merely be disk > contents (and if so, do you want to quiesce the file system before > taking the snapshot, to avoid the need to fsck the disks before > restoring from them), or the more space-intensive full system snapshot > (where you don't need to quiesce, because reverting involves reloading > the RAM of the guest at the time the snapshot was taken)? > >> at the moment the vms qcow2 disks are stored over an ext4 partition but >> I'm planning to move them over a das with a new generation filesystem. >> >> I'm trying out zfs and btrfs looking for the best one (at the moment zfs >> seems to be more stable and somehow easy, flexible, while btrfs seems to >> have better performance on my linux box). > > btrfs has lousy behavior when coupled with qcow2 images (combining two > different layers that are both trying to do copy-on-write tends to lead > to horrible fragmentation). There is an ioctl to tell btrfs to not do > copy-on-write for files used by snapshots, so you can get performance > back, but it is not default and puts some burden on you to set up. > > I don't know enough about zfs snapshot to have an opinion. > > You may not need file system snapshot if you instead choose to go with > libvirt live snapshot. > >> >> I'm choosing one of these two FS in order to use theire instant-snapshot >> feature; >> my project will be to pause the vm, dump the memory and the xml >> descriptor file, make a snapshot of the subvolume storing the qcow2 file >> and the resuming the machine back (in I hope less than 10 seconds of >> downtime). > > Not ideal. This only takes a disk snapshot, but the state of the disk > is not consistent (you didn't quiesce), so you lose on any pending I/O > that the OS had not yet flushed to disk. Furthermore, this requires > pausing the VM manually, which is more downtime than you will have if > you use libvirt's live snapshot mechanisms. > >> >> what do you guys suggest me? >> >> Am I going to experience bad performance having qcow2 over btrfs? >> I was reading something like that but the posts were really backdated... > > As far as I know, this is still a real problem unless you manually > configure btrfs to not do copy-on-write, at least when qcow2 images are > involved. >-----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlRbqxEACgkQWZ6iCY3lmBSOiQCeMG6E60y1CmnCiBA7Ls9DThIx D4sAn28Z4q/beDagN/KyKirZRnVEjIMN =tFx+ -----END PGP SIGNATURE-----