Hi all, I am running Xen 3.2-1 / 2.6.18 on Debian Etch (Dom0). I setup a Dom0 with several DomUs (Linux). Each DomU disk is a LV partition. I have been seeing a few mails about backups and LVM and just wanted to make sure that what I am doing is correct. Each DomU with a database has a cron that dumps the db with mysqldump. Later on, Dom0 will do an LVM Snapshot of all DomUs and then do a file system backup of the Snapshots. I do no actions on the DomUs. No shutdown or ''xm pause/save/...'' Does this sound about right or am I missing something important? Any pointers to ''Xen backup'' documentation are welcome! :) Thanks for your help. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
lists@precognet.com schrieb:> Hi all, > > I am running Xen 3.2-1 / 2.6.18 on Debian Etch (Dom0). > > I setup a Dom0 with several DomUs (Linux). Each DomU disk is a LV partition. I have been seeing a few mails about backups and LVM and just wanted to make sure that what I am doing is correct. > > Each DomU with a database has a cron that dumps the db with mysqldump. Later on, Dom0 will do an LVM Snapshot of all DomUs and then do a file system backup of the Snapshots. I do no actions on the DomUs. No shutdown or ''xm pause/save/...'' > > Does this sound about right or am I missing something important?Sounds reasonable.> Any pointers to ''Xen backup'' documentation are welcome! :)Nothing really Xen-specific here. Same backup rules apply to normal, non-virtualized machines. -- Tomasz Chmielewski http://wpkg.org _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thanks for the feedback Tomasz! ----- Original Message ----- From: "Tomasz Chmielewski" <mangoo@wpkg.org> To: lists@precognet.com Cc: xen-users@lists.xensource.com Sent: Wednesday, October 15, 2008 10:36:06 AM GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna Subject: Re: [Xen-users] Xen backups using LVM Snapshots lists@precognet.com schrieb:> Hi all, > > I am running Xen 3.2-1 / 2.6.18 on Debian Etch (Dom0). > > I setup a Dom0 with several DomUs (Linux). Each DomU disk is a LV partition. I have been seeing a few mails about backups and LVM and just wanted to make sure that what I am doing is correct. > > Each DomU with a database has a cron that dumps the db with mysqldump. Later on, Dom0 will do an LVM Snapshot of all DomUs and then do a file system backup of the Snapshots. I do no actions on the DomUs. No shutdown or ''xm pause/save/...'' > > Does this sound about right or am I missing something important?Sounds reasonable.> Any pointers to ''Xen backup'' documentation are welcome! :)Nothing really Xen-specific here. Same backup rules apply to normal, non-virtualized machines. -- Tomasz Chmielewski http://wpkg.org _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, Oct 15, 2008 at 2:17 AM, <lists@precognet.com> wrote:> Each DomU with a database has a cron that dumps the db with mysqldump. Later on, Dom0 will do an LVM Snapshot of all DomUs and then do a file system backup of the Snapshots. I do no actions on the DomUs. No shutdown or ''xm pause/save/...''as you must have seen already, those snapshots mount as ''dirty'' filesystems, since they''re exact copies of an already mounted filesystem. therefore, you have to run fsck on mount (usually automatic on journalling filesystems). depending on the journalling algorithm, you risk having non-written data. much safer, and easier IMHO, is to do a regular network backup. something as simple as pulling via rsync the files you need. (i really like rsnapshot) -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thanks Javier. I might end up installing an online backup client on each DomU. I''ll also have a good look at rsnapshot. I used rsync extensively a few years back to make online backups. Thanks for your advice. -eco ----- Original Message ----- From: "Javier Guerra" <javier@guerrag.com> To: lists@precognet.com Cc: xen-users@lists.xensource.com Sent: Wednesday, October 15, 2008 3:55:13 PM GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna Subject: Re: [Xen-users] Xen backups using LVM Snapshots On Wed, Oct 15, 2008 at 2:17 AM, <lists@precognet.com> wrote:> Each DomU with a database has a cron that dumps the db with mysqldump. Later on, Dom0 will do an LVM Snapshot of all DomUs and then do a file system backup of the Snapshots. I do no actions on the DomUs. No shutdown or ''xm pause/save/...''as you must have seen already, those snapshots mount as ''dirty'' filesystems, since they''re exact copies of an already mounted filesystem. therefore, you have to run fsck on mount (usually automatic on journalling filesystems). depending on the journalling algorithm, you risk having non-written data. much safer, and easier IMHO, is to do a regular network backup. something as simple as pulling via rsync the files you need. (i really like rsnapshot) -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I would concur! Rsnapshot is an excellent way to get a good backup of needed data directories and provides a lower risk of loss. Steven G. Spencer, Network Administrator KSC Corporate - The Kelly Supply Family of Companies Office 308-382-8764 Ext. 231 Mobile 308-380-7957 Javier Guerra wrote:> On Wed, Oct 15, 2008 at 2:17 AM, <lists@precognet.com> wrote: >> Each DomU with a database has a cron that dumps the db with mysqldump. Later on, Dom0 will do an LVM Snapshot of all DomUs and then do a file system backup of the Snapshots. I do no actions on the DomUs. No shutdown or ''xm pause/save/...'' > > > as you must have seen already, those snapshots mount as ''dirty'' > filesystems, since they''re exact copies of an already mounted > filesystem. therefore, you have to run fsck on mount (usually > automatic on journalling filesystems). depending on the journalling > algorithm, you risk having non-written data. > > much safer, and easier IMHO, is to do a regular network backup. > something as simple as pulling via rsync the files you need. (i > really like rsnapshot) >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Javier Guerra wrote:> On Wed, Oct 15, 2008 at 2:17 AM, <lists@precognet.com> wrote: > >> Each DomU with a database has a cron that dumps the db with mysqldump. Later on, Dom0 will do an LVM Snapshot of all DomUs and then do a file system backup of the Snapshots. I do no actions on the DomUs. No shutdown or ''xm pause/save/...'' >> > > > as you must have seen already, those snapshots mount as ''dirty'' > filesystems, since they''re exact copies of an already mounted > filesystem. therefore, you have to run fsck on mount (usually > automatic on journalling filesystems). depending on the journalling > algorithm, you risk having non-written data. >This isn''t sounding quite right to me. According to LVM Howto on tldp.org a snapshot is "exact copy of a logical volume, frozen at some point in time." (http://tldp.org/HOWTO/LVM-HOWTO/snapshotintro.html).>From my experience, creating an LVM snapshot creates a new LVM volumethat stays consistent despite changes on the LVM volume the snapshot was created from. I''m sure there is a performance decrease since LVM has to track a lot more information during this time. But it should work. I do something similar to what is asked about in this and the other email thread without a problem yet. Here are the tutorials I used for coming up with the way I do things. http://tldp.org/HOWTO/LVM-HOWTO/snapshots_backup.html http://howtoforge.com/linux_lvm_snapshots It seems to work for me. Mike _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Javier Guerra Giraldez
2008-Oct-17 02:12 UTC
Re: [Xen-users] Xen backups using LVM Snapshots
On Thursday 16 October 2008, Mike Lovell wrote:> Javier Guerra wrote: > > On Wed, Oct 15, 2008 at 2:17 AM, <lists@precognet.com> wrote: > >> Each DomU with a database has a cron that dumps the db with mysqldump. > >> Later on, Dom0 will do an LVM Snapshot of all DomUs and then do a file > >> system backup of the Snapshots. I do no actions on the DomUs. No > >> shutdown or ''xm pause/save/...'' > > > > as you must have seen already, those snapshots mount as ''dirty'' > > filesystems, since they''re exact copies of an already mounted > > filesystem. therefore, you have to run fsck on mount (usually > > automatic on journalling filesystems). depending on the journalling > > algorithm, you risk having non-written data. > > This isn''t sounding quite right to me. According to LVM Howto on > tldp.org a snapshot is "exact copy of a logical volume, frozen at some > point in time." (http://tldp.org/HOWTO/LVM-HOWTO/snapshotintro.html).right; but if that "point in time" was while it was mounted, the snapshot is exactly that: a non-unmounted volume. just the same you would''ve got if you had pulled the plug on a running machine. when you mount the snapshot, pay attention to warning messages, they''ll tell the volume was dirty and run fsck. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
----- "Javier Guerra Giraldez" <javier@guerrag.com> wrote:> On Thursday 16 October 2008, Mike Lovell wrote: > > This isn''t sounding quite right to me. According to LVM Howto on > > tldp.org a snapshot is "exact copy of a logical volume, frozen at > some > > point in time." > (http://tldp.org/HOWTO/LVM-HOWTO/snapshotintro.html). > > right; but if that "point in time" was while it was mounted, the > snapshot is > exactly that: a non-unmounted volume. just the same you would''ve got > if you > had pulled the plug on a running machine. when you mount the > snapshot, pay > attention to warning messages, they''ll tell the volume was dirty and > run > fsck. > > -- > Javier >It should not be too hard to test and get a definite answer. I don''t have access to my box right now but all we need to do I suspect, is create a snapshot. Stop the DomU. Point the cfg file to the snapshot and start the DomU. If we get FS warnings, Javier is right... -- eco _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> It should not be too hard to test and get a definite answer. I don''t > have access to my box right now but all we need to do I suspect, is > create a snapshot. Stop the DomU. Point the cfg file to the snapshot > and start the DomU. If we get FS warnings, Javier is right... > > -- > eco > > _______________________________________________I ran a very quick test... here is my result DomU # while [ 1 -ne 2 ]; do echo `date` >> /root/test; done & Dom0 # lvcreate -L +1G -s -n s_web.test.com-disk /dev/vm/web.test.com-disk DomU - stopped script and shutdown system. Dom0 - Edit cfg file replace disk = [ ''phy:/dev/vm/web.test.com-swap,sda1,w'', ''phy:/dev/vm/web.test.com-disk,sda2,w'', ] with disk = [ ''phy:/dev/vm/web.test.com-swap,sda1,w'', ''phy:/dev/vm/s_web.test.com-disk,sda2,w'', ] DomU - Boot [...] Begin: Running /scripts/local-premount ... Done. EXT3-fs: INFO: recovery required on readonly filesystem. EXT3-fs: write access will be enabled during recovery. kjournald starting. Commit interval 5 seconds EXT3-fs: recovery complete. EXT3-fs: mounted filesystem with ordered data mode. [...] Javier is right! -- eco _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
If you were running a single OS with lvm snapshots (IE, no xen), would the snapshots be consistent? IE, no fsck needed ever for the snapshots? If so, then there must be a mechanism for LVM snapshotting to tell the file system to coalesce to disk before the snapshot. If there is such a connection between the LVM snapshotter and the file system, I could see it not working properly if the LVM snapshotter is in dom0 and the file system in domU. Perhaps a xen extension could be added to support that? Kevin On Thu, 2008-10-16 at 19:12 -0700, Javier Guerra Giraldez wrote:> On Thursday 16 October 2008, Mike Lovell wrote: > > Javier Guerra wrote: > > > On Wed, Oct 15, 2008 at 2:17 AM, <lists@precognet.com> wrote: > > >> Each DomU with a database has a cron that dumps the db with > mysqldump. > > >> Later on, Dom0 will do an LVM Snapshot of all DomUs and then do a > file > > >> system backup of the Snapshots. I do no actions on the DomUs. > No > > >> shutdown or ''xm pause/save/...'' > > > > > > as you must have seen already, those snapshots mount as ''dirty'' > > > filesystems, since they''re exact copies of an already mounted > > > filesystem. therefore, you have to run fsck on mount (usually > > > automatic on journalling filesystems). depending on the > journalling > > > algorithm, you risk having non-written data. > > > > This isn''t sounding quite right to me. According to LVM Howto on > > tldp.org a snapshot is "exact copy of a logical volume, frozen at > some > > point in > time." (http://tldp.org/HOWTO/LVM-HOWTO/snapshotintro.html). > > right; but if that "point in time" was while it was mounted, the > snapshot is > exactly that: a non-unmounted volume. just the same you would''ve got > if you > had pulled the plug on a running machine. when you mount the > snapshot, pay > attention to warning messages, they''ll tell the volume was dirty and > run > fsck. > > -- > Javier > > > plain text document attachment (ATT566064.txt), "ATT566064.txt" > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Oct 17, 2008 at 12:29 PM, Kevin Fox <Kevin.Fox@pnl.gov> wrote:> If you were running a single OS with lvm snapshots (IE, no xen), would > the snapshots be consistent? IE, no fsck needed ever for the snapshots? > If so, then there must be a mechanism for LVM snapshotting to tell the > file system to coalesce to disk before the snapshot.No. the snapshots ARE consistent. consistent to a point in time where it was mounted and running. if do an LVM snapshot, and a VM snapshot, you could recreate the running state. the difference between a mounted and an unmounted volume is totally filesystem dependent; it''s not the job of a blockdevice layer. (this is one reason why some people like the "manage everything at filesystem" approach of ZFS). that ''coalescing'' you refer have two parts: a full sync(), and unmounting itself. you can do sync() just before doing the snapshot, so the fsck needed when mounting it would be ''almost'' guaranteed to find it clean. On journalling filesystems, the metadata, at least is fairly safe. And, on ext3 with data=journal, the files data is just as safe. Other FSs have similar settings. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Kevin Fox wrote:> If there is such a connection between the LVM snapshotter and the file > system, I could see it not working properly if the LVM snapshotter is in > dom0 and the file system in domU. Perhaps a xen extension could be added > to support that? >There is indeed such a connection within the same OS and yes, if you want to take a back up of the guest''s file system you need to enlist its cooperation to get the file system into a consistent state before you take the snapshot and, of course, to let it go afterwards. In general, you''ll also need the cooperation of any applications that are running as well. In the end, though you have to ask yourself if it''s worth it. For example, someone, somewhere mentioned running mysqlhotbackup in the guest and putting the files somewhere that dom0 could see them. Is it quicker, in the end, to recover the OS data from a backup or just run a kickstart installation to rebuild it? Chances are that if you''re backing up an entire system by, basically, photocopying the disk, you''re backing up a load of stuff you don''t need to. There are hardware configurations where dom0 driven backups are better, but to make them as good as bare-metal backups then you really need the cooperation and, in many cases, active help from the domU. jch _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Lists wrote:> I ran a very quick test... here is my result > > DomU # while [ 1 -ne 2 ]; do echo `date` >> /root/test; done & > > Dom0 # lvcreate -L +1G -s -n s_web.test.com-disk /dev/vm/web.test.com-disk > > DomU - stopped script and shutdown system. > > Dom0 - Edit cfg file > replace > disk = [ > ''phy:/dev/vm/web.test.com-swap,sda1,w'', > ''phy:/dev/vm/web.test.com-disk,sda2,w'', > ] > > with > disk = [ > ''phy:/dev/vm/web.test.com-swap,sda1,w'', > ''phy:/dev/vm/s_web.test.com-disk,sda2,w'', > ] > > DomU - Boot > [...] > Begin: Running /scripts/local-premount ... > Done. > EXT3-fs: INFO: recovery required on readonly filesystem. > EXT3-fs: write access will be enabled during recovery. > kjournald starting. Commit interval 5 seconds > EXT3-fs: recovery complete. > EXT3-fs: mounted filesystem with ordered data mode. > [...] > > Javier is right! > > -- > eco >I think we are talking about two similar but slightly different procedures. On my script, I do not stop and start the DomU at all. The DomU is running the whole time with the LVM volume that it was originally assigned. Here is what I run completely inside the Dom0 and I don''t get FS errors (names edited to generic stuff and very verbose): #!/bin/sh echo "creating lvm snapshot and mounting..." lvcreate -L50G -s -n DomU-snapshot /dev/VG0/DomU mount /dev/VG0/DomU-snapshot /mnt echo "taring filesystem...." cd /mnt pwd tar jcvf /data/DomU-backup-`date +%Y-%m-%d`.tar.bz2 * echo "unmounting and removing snapshot..." cd /root umount /mnt lvremove -f /dev/VG0/DomU-snapshot echo "all done!!!" My guest is using ext3 and the mount completed without giving an error on the terminal. But checking dmesg did show some errors. And, I am able to read the data off of the disk. So I think I see what you guys are saying about snapshotting making a snapshot of the still-mounted volume. Backups made using this method could not guarantee a good backup for files that were open at the time the snapshot was made. Doing a backup this way though would be equivalent to doing a backup right after a machine crashes. (I guess some data is better than no data). One way around this that results in minimal downtime for the guest is to do something like - Shutdown the guest VM - create an LVM snapshot - start the guest VM still using the original LVM volume - make a backup using the snapshot volume - remove the LVM snapshot The guest would only see a down time about equal to just a standard reboot. You guys were right on the filesystem issues. But I still think LVM snapshots can be a valid way of doing a backup if precautions are taken. So there are my thoughts on it. Mike _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, 2008-10-17 at 10:41 -0700, Javier Guerra wrote:> On Fri, Oct 17, 2008 at 12:29 PM, Kevin Fox <Kevin.Fox@pnl.gov> wrote: > > If you were running a single OS with lvm snapshots (IE, no xen), > would > > the snapshots be consistent? IE, no fsck needed ever for the > snapshots? > > If so, then there must be a mechanism for LVM snapshotting to tell > the > > file system to coalesce to disk before the snapshot. > > No.Strange that the file system and LVM have enough knowledge of each other to do online resizing, but not snapshotting. Guess it wouldn''t be easy to implement then. Kevin> the snapshots ARE consistent. consistent to a point in time where it > was mounted and running. if do an LVM snapshot, and a VM snapshot, > you could recreate the running state. > > the difference between a mounted and an unmounted volume is totally > filesystem dependent; it''s not the job of a blockdevice layer. (this > is one reason why some people like the "manage everything at > filesystem" approach of ZFS). > > that ''coalescing'' you refer have two parts: a full sync(), and > unmounting itself. you can do sync() just before doing the snapshot, > so the fsck needed when mounting it would be ''almost'' guaranteed to > find it clean. On journalling filesystems, the metadata, at least is > fairly safe. And, on ext3 with data=journal, the files data is just > as safe. Other FSs have similar settings. > > -- > Javier > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, 2008-10-17 at 10:50 -0700, John Haxby wrote:> Kevin Fox wrote: > > If there is such a connection between the LVM snapshotter and the > file > > system, I could see it not working properly if the LVM snapshotter > is in > > dom0 and the file system in domU. Perhaps a xen extension could be > added > > to support that? > > > There is indeed such a connection within the same OS and yes, if you > want to take a back up of the guest''s file system you need to enlist > its > cooperation to get the file system into a consistent state before you > take the snapshot and, of course, to let it go afterwards.Yeah. I wouldn''t think that would be too difficult to do. Though usually you see calls going from DomU->Xen, not the other way around. I''m not sure there is a mechanism in place to go the other way.> In general, > you''ll also need the cooperation of any applications that are running > as > well.Depending on what your trying to backup, true.> > In the end, though you have to ask yourself if it''s worth it. For > example, someone, somewhere mentioned running mysqlhotbackup in the > guest and putting the files somewhere that dom0 could see them. Is > it > quicker, in the end, to recover the OS data from a backup or just run > a > kickstart installation to rebuild it? Chances are that if you''re > backing up an entire system by, basically, photocopying the disk, > you''re > backing up a load of stuff you don''t need to.I think whether its worth it all depends what kind of backups your trying to do. If your trying to backup database servers, its probably not. One case where it could be worth it is if you had a virtual machine per user for doing something like a terminal server. Each user gets their own VM to do email, web surfing, office, etc. Being able to run one piece of backup software on the dom0 would be less maintenance then maintaining one copy per virtual machine. If your paying for backup licenses, it can be substantially cheaper too. Kevin> > There are hardware configurations where dom0 driven backups are > better, > but to make them as good as bare-metal backups then you really need > the > cooperation and, in many cases, active help from the domU. > > jch > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On October 17, 2008 10:51 am Kevin Fox wrote:> On Fri, 2008-10-17 at 10:41 -0700, Javier Guerra wrote: > > On Fri, Oct 17, 2008 at 12:29 PM, Kevin Fox <Kevin.Fox@pnl.gov> wrote: > > > If you were running a single OS with lvm snapshots (IE, no xen), > > > would the snapshots be consistent? IE, no fsck needed ever for the > > > snapshots? > > > > > If so, then there must be a mechanism for LVM snapshotting to tell > > > the file system to coalesce to disk before the snapshot. > > > > No.Not directly, but some FS, like XFS, have features that make it more amenable to LVM-style snapshots. xfs_freeze can be used to temporarily put all I/O on "pause", and flush data to the FS. Then you can take an LVM snapshot and get a consistent snapshot. After the snapshot is created, you unfreeze the filesystem. If you script it, you get an I/O interruption of only a couple of seconds at most. However, this wouldn''t help in the "snapshot LVM from dom0" situation, as you couldn''t run xfs_freeze directly from the dom0. But, a little SSH and sudo trickery could allow you to do it. In the domU, configure a backups user, add them to sudoers with the ability to run xfs_freeze without a password. Then, from the dom0, you could run something like: #!/bin/sh ssh -l backups <domU> "sudo xfs_freeze -f /mountpoint" lvcreate -s -n domU-snapshot /path/to/domU/lv ssh -l backups <domU> "sudo xfs_freeze -u /mountpoint" <do your backups using the domU-snapshot>> Strange that the file system and LVM have enough knowledge of each > other to do online resizing, but not snapshotting.That''s the nature of LVM-style snapshots. The snapshots are done below the filesystem layer, where the filesystem has no knowledge of what''s going on. As far as the FS is concerned, it''s running on a harddrive. What I really dislike about LVM-style snapshots is that the snapshot is outside of the FS, and you have to plan ahead for how much space you think you''ll need for each snapshot, and you can''t keep a bunch of them around for very long. This is one area where in-FS snapshots can work better, depending on how they are done. For example, UFS2 snapshots in FreeBSD can take up to a minute to create, and don''t scale beyond a handful of snapshots. But ZFS snapshots in FreeBSD (I don''t have access to Solaris) are virtually instantaneous, and (so far in our testing) scale above 64 simultaneous snapshots without any issues. Plus, you can stream snapshots between servers, making remote backups a breeze. But, that''s all kind of beside the point here. I find doing backups from within the domU (same as is done on physical servers) to be the easiest. -- Freddie Cash fjwcash@gmail.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users