-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi :) We''re considering using snapshots of LVM logical volumes as a backup feature, that is, having domU instances running on each of their own logical volume, and then in the dom0, create snapshots and take backups. Has anyone else tried this? Is it safe? I saw an old message from 2006 on this list, where it is said that "Common understanding is that LVM2 snapshots are unstable". Is that still the case? Thanks, Morten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFI9NN3mq0JiiIWC2oRAkXJAJ4/H05SrmQLSuWxgE0yFeIpRhBC3QCdE1SA C663jPhd8OxK1rv4TX+49zQ=iu4b -----END PGP SIGNATURE----- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
"Morten W. Petersen" <morten@nidelven-it.no> writes:> We''re considering using snapshots of LVM logical volumes as a backup > feature, that is, having domU instances running on each of their own > logical volume, and then in the dom0, create snapshots and take backups. > > Has anyone else tried this? Is it safe?If you snapshot an LV while it''s being used by a virtual machine, then your snapshot will be no better than what you''d get by powering off a server. That is, inconsistent data and dirty filesystems. That means your snapshot filesystem will have its journal replayed on mount, etc. It may or may not be good enough for your purposes.> I saw an old message from 2006 on this list, where it is said that > "Common understanding is that LVM2 snapshots are unstable".I don''t understand this statement. -- Feri. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
2008/10/14 Morten W. Petersen <morten@nidelven-it.no>:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi :) > > We''re considering using snapshots of LVM logical volumes as a backup > feature, that is, having domU instances running on each of their own > logical volume, and then in the dom0, create snapshots and take backups. > > Has anyone else tried this? Is it safe? > > I saw an old message from 2006 on this list, where it is said that > "Common understanding is that LVM2 snapshots are unstable". > > Is that still the case? > > Thanks, > > MortenThan can be done, but you''ll have to stop the domU, take snapshot and start the domU again... Regards, Ciro _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Ferenc Wagner schrieb: (...)>> I saw an old message from 2006 on this list, where it is said that >> "Common understanding is that LVM2 snapshots are unstable". > > I don''t understand this statement.Anything older than 2.6.18 (so, including 2.6.18) will use lots of memory for a snapshot. This means, you won''t be able to do more than a handful of snapshots even on a system with lots of free memory (you may expect trouble i.e. you have 512 MB RAM, only 30 MB used, and want to do 10-20 snapshots). You have 400 MB memory left, did 30 snapshots, and it still works for you? You may reconsider - when they fill, (kernel) memory usage will rise, until your machine is unresponsive. As it''s kernel memory being used, oom-killer won''t help much here; bigger swap won''t help either, because real RAM is needed. To make the things worse, you will probably be unable to boot the machine again. As the system starts, LVM volumes are detected, memory is full again... You''re out of luck, unable to even remove the snapshots in order to boot properly, unless you have some more RAM sticks at hand to fix the problem. Coincidentally, 2.6.18 kernel is the one Xen is based on (at least dom0), so if you want your Xen server to be a storage server at the same time, you might run into problems as your system grows. Some major distributions base on this kernel as well (i.e., Debian Etch). A workaround might be: - using bigger stripes with LVM (it should take less memory) - use a newer kernel - as a rule of thumb - the newer, the better - might be problematic with Xen dom0, - have storage on a separate machine (i.e., iSCSI SAN) with a newer kernel than your Xen dom0 machine Newer kernels (2.6.20, maybe even 2.6.19, and later) are not affected by this "phenomenon". Another LVM disadvantage is that every snapshot means additional writes. I.e. if you have a logical volume, and 4 snapshots, writing to the origin will mean 4 more writes are needed. This is a serious scalability problem. -- Tomasz Chmielewski http://wpkg.org _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Tomasz Chmielewski wrote:> Ferenc Wagner schrieb: > > (...) > >>> I saw an old message from 2006 on this list, where it is said that >>> "Common understanding is that LVM2 snapshots are unstable". >> >> I don''t understand this statement. > > Anything older than 2.6.18 (so, including 2.6.18) will use lots of > memory for a snapshot. This means, you won''t be able to do more than a > handful of snapshots even on a system with lots of free memory (you may > expect trouble i.e. you have 512 MB RAM, only 30 MB used, and want to do > 10-20 snapshots). > > You have 400 MB memory left, did 30 snapshots, and it still works for > you? You may reconsider - when they fill, (kernel) memory usage will > rise, until your machine is unresponsive. As it''s kernel memory being > used, oom-killer won''t help much here; bigger swap won''t help either, > because real RAM is needed. > > > To make the things worse, you will probably be unable to boot the > machine again. As the system starts, LVM volumes are detected, memory is > full again... You''re out of luck, unable to even remove the snapshots in > order to boot properly, unless you have some more RAM sticks at hand to > fix the problem. > > > Coincidentally, 2.6.18 kernel is the one Xen is based on (at least > dom0), so if you want your Xen server to be a storage server at the same > time, you might run into problems as your system grows. > Some major distributions base on this kernel as well (i.e., Debian Etch).Right. :)> A workaround might be: > - using bigger stripes with LVM (it should take less memory) > - use a newer kernel - as a rule of thumb - the newer, the better - > might be problematic with Xen dom0, > - have storage on a separate machine (i.e., iSCSI SAN) with a newer > kernel than your Xen dom0 machine > > > Newer kernels (2.6.20, maybe even 2.6.19, and later) are not affected by > this "phenomenon". > > Another LVM disadvantage is that every snapshot means additional writes. > I.e. if you have a logical volume, and 4 snapshots, writing to the > origin will mean 4 more writes are needed. This is a serious scalability > problem.Well, what I was thinking, was to do a sync on the VM, and flush the MySQL database and then immediately afterwards, taking a snapshot of the system, backup that snapshot and then after backup "releasing" (deleting) the snapshot. We''re stuck with .18 for now, as that''s what Debian comes with and we try to stick with off-the-shelf packages. Do you think that''s feasible? - -Morten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFI9QVxmq0JiiIWC2oRAg8DAJ48kRkq1NHVOfEPg5UIh5Pc1jRgCwCgxfcD v97XYGPHBPGJk3FdqSDVszY=dpsD -----END PGP SIGNATURE----- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
2008/10/14 Morten W. Petersen <morten@nidelven-it.no>:> > Well, what I was thinking, was to do a sync on the VM, and flush the > MySQL database and then immediately afterwards, taking a snapshot of the > system, backup that snapshot and then after backup "releasing" > (deleting) the snapshot. > > We''re stuck with .18 for now, as that''s what Debian comes with and we > try to stick with off-the-shelf packages. > > Do you think that''s feasible? > > - -MortenIf all you want is the Mysql data files, In our case, we stop mysql, create the snapshot and start it again...All this within the domU (mysql is in a separate volume)... Regards, Ciro _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Morten W. Petersen schrieb: (...)>> Newer kernels (2.6.20, maybe even 2.6.19, and later) are not affected by >> this "phenomenon". >> >> Another LVM disadvantage is that every snapshot means additional writes. >> I.e. if you have a logical volume, and 4 snapshots, writing to the >> origin will mean 4 more writes are needed. This is a serious scalability >> problem. > > Well, what I was thinking, was to do a sync on the VM, and flush the > MySQL database and then immediately afterwards, taking a snapshot of the > system, backup that snapshot and then after backup "releasing" > (deleting) the snapshot. > > We''re stuck with .18 for now, as that''s what Debian comes with and we > try to stick with off-the-shelf packages. > > Do you think that''s feasible?Personally, I don''t think taking snapshots is a good backup strategy, for several reasons: - they take a lot of space - the more snapshots you have, the more I/O load you will have (writes will go to every snapshot) - unless you''re running a bleeding edge kernel with experimental patches, there is no way to turn the snapshot into original (other than copying it with dd, which will fill the snapshot, which means the snapshot has to be 100% of the size of the origin; which takes a lot of space) - for most Linux systems, taking backup is as easy as copying all files even from a live system; for databases, dump is of course needed to guarantee consistence; restoring backup is as easy as copying the files back - done - you really should make backups to a different machine, preferably somewhere else That said, I use snapshots for: - Windows - it''s a pain in the a*** to restore a working Windows machine if you only have file backups; I usually keep snapshots for the last 3 days here (in addition to the usual file backup) - if I make any major changes, and don''t have time to do a full file backup to another machine For backup, I can recommend BackupPC. -- Tomasz Chmielewski http://wpkg.org _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Morten W. Petersen wrote:> > Well, what I was thinking, was to do a sync on the VM, and flush the > MySQL database and then immediately afterwards, taking a snapshot of the > system, backup that snapshot and then after backup "releasing" > (deleting) the snapshot. >Since now you''re being specific on what you want to backup, you''d be more likely get a better response ;-) General backup strategies with mysql : - using mysqldump : works with all table type/engines, lock tables during backup - using innodb hot backup : works with innodb only, online backup possible - offline backup : shutdown mysql - copy files - startup mysql Those three methods can get you clean, consistent backup. Using file system snapshot (whatever the method is) will get you "consistent", dirty backup. "consistent", as in it depends on what engine you use (MyIsam tables will possibly be corrupted). dirty, as in some recovery method (which can run automatically, BTW) is needed (either myisamchk or innodb log replay). Now, considering the above implications, fs snapshot may not be desirable on certain situations. For example, if you back up a 50GB MyIsam table with fs snapshot, you''ll most likely require several hours of myisamchk before it can be restored cleanly. On the other hand Innodb is ACID-compliant, so you can "safely" use fs snapshot with Innodb tables.> We''re stuck with .18 for now, as that''s what Debian comes with and we > try to stick with off-the-shelf packages. > > Do you think that''s feasible? >If you still want snapshot-based backups I''d actually prefer to use zfs snapshot instead of Linux LVM snapshot. It''s more reliable, more managable, and more efficient. If you use opensolaris dom0, you can snapshot dom0''s zfs volume which acts as domU''s storage. If you use opensolaris domU (whatever the dom0 is, including Debian), you can use zfs (either as root or just for data) on domU and create the snapshot there. If you use Linux domU, you can use zfs-fuse and have MySQL data files there. MySQL runs fine on zfs-fuse. Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
"Morten W. Petersen" <morten@nidelven-it.no> writes:> Hi :) > > We''re considering using snapshots of LVM logical volumes as a backup > feature, that is, having domU instances running on each of their own > logical volume, and then in the dom0, create snapshots and take backups. > > Has anyone else tried this? Is it safe?It is safe, but be aware that the mere fact that you have a snapshot active can slow down write speeds to the volume by a factor of 10. I got bitten by this, but I also have DRBD in play, so I don''t know for sure if its a xem, drbd or lvm issue, or the interaction between the three. -- To sto si frustriran, zavidan tko zna na cemu i sto ne vidis dalje od svoje guzice je tuzno. Da onda barem imas toliko samokontrole da sutis umjesto da pravis budalu od sebe... izgleda da si prestar da se promjenis na bolje. - Davor Pasaric, hr.comp.mac _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I asked the same question a few posts down and was told that taking a snapshot of a running system could lead to some corrupted files. I recommend you look for "[Xen-users] Xen backups using LVM Snapshots" and make your own decision from there. Best of luck. -- eco ----- Original Message ----- From: "Aleksandar Ivanisevic" <aleksandar@ivanisevic.de> To: xen-users@lists.xensource.com Sent: Thursday, October 16, 2008 12:48:47 PM GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna Subject: [Xen-users] Re: Xen, LVM and snapshots "Morten W. Petersen" <morten@nidelven-it.no> writes:> Hi :) > > We''re considering using snapshots of LVM logical volumes as a backup > feature, that is, having domU instances running on each of their own > logical volume, and then in the dom0, create snapshots and take backups. > > Has anyone else tried this? Is it safe?It is safe, but be aware that the mere fact that you have a snapshot active can slow down write speeds to the volume by a factor of 10. I got bitten by this, but I also have DRBD in play, so I don''t know for sure if its a xem, drbd or lvm issue, or the interaction between the three. -- To sto si frustriran, zavidan tko zna na cemu i sto ne vidis dalje od svoje guzice je tuzno. Da onda barem imas toliko samokontrole da sutis umjesto da pravis budalu od sebe... izgleda da si prestar da se promjenis na bolje. - Davor Pasaric, hr.comp.mac _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, Oct 16, 2008 at 11:35:48AM +0000, Lists wrote:> I asked the same question a few posts down and was told that taking a snapshot of a running system could lead to some corrupted files. > I recommend you look for "[Xen-users] Xen backups using LVM Snapshots" and make your own decision from there. > Best of luck.If you snapshot a running system without saving both the memory state and swap state you could have issues with inconsistent data. It has always been my understanding that xm-save, lv-snapshot (all partitions including swap), xm-restore, dd snapshotx imgfilex, delete snapshot volumes, backup xm save checkpoint. Would leave you with consistent disk state being that you have everything to power the machine back on at time of snapshot. There may be some corruption in specific applications due to processing something over the network that is no longer there, or that has changed since the backup. But as far as I can tell your base os should be intact. If you combine that will application level backups you will probably be ok. Worst case take img backup restore, then restore from latest application backup. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Morten W. Petersen wrote:> We''re considering using snapshots of LVM logical volumes as a backup > feature, that is, having domU instances running on each of their own > logical volume, and then in the dom0, create snapshots and take backups. > > Has anyone else tried this? Is it safe? > > I saw an old message from 2006 on this list, where it is said that > "Common understanding is that LVM2 snapshots are unstable". > > Is that still the case?I do something similar to this with a xen guest that I have. I currently have openSuSE 10.3 running as dom0. I have a linux guest that is running paravirtualized and is using an LVM volume as it filesystem. I have a script that I run in dom0 that creates a LVM snapshot of the guest volume, mounts the snapshot as read only, creates a tarball of the guest root filesystem, unmounts the snapshot, and then removes the lvm snapshot. So far this has worked great for backing up the root filesystem for my xen guest. I haven''t needed to try a restore yet (thankfully things haven''t died). Also, this hasn''t been tested under heavy load so I make no guarantees about performance. What I did for my set up is an adaptation of a tutorial on howtoforge.com (http://howtoforge.com/linux_lvm_snapshots). From my understanding, if you create an LVM snapshot, that snapshot stays consistent independent of what is happening on the non-snapshot volume. It has been a while since I tested that fact but I''m pretty sure it worked that way. So you should be able to do what you are asking about. If you would like a copy of the script that i use, i can clean it up and send it to you. Mike _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users