Hi, What should I do for a right backup of guests and dom0? What is the effective way of backup on xen. thanks, _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
2009/4/2 Umut Arus <umuta@sabanciuniv.edu>:> Hi, > > What should I do for a right backup of guests and dom0? What is the > effective way of backup on xen.the most general answer, and with the least surprises, is to use any network backup solution on the DomU. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com [mailto:xen-users- > bounces@lists.xensource.com] On Behalf Of Javier Guerra > Sent: Thursday, April 02, 2009 11:07 AM > To: Umut Arus > Cc: xen-users@lists.xensource.com > Subject: Re: [Xen-users] Xen Backup Method > > 2009/4/2 Umut Arus <umuta@sabanciuniv.edu>: > > Hi, > > > > What should I do for a right backup of guests and dom0? What is the > > effective way of backup on xen. > > the most general answer, and with the least surprises, is to use any > network backup solution on the DomU. > > -- > Javier > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-usersYes, it seems the easiest way to backup a domU is to use the backup solution that you have been using for your physical servers. Same goes for dom0. You can experiment with other things, but to provide zero downtime and a clean backup those seem like the best solutions. I have been working on something that backs up files after degrading a DRBD array (where the vm files are hosted), but then if you restore a VM it thinks that the power cord has been pulled. This has zero downtime but a MUCH higher probability that the VM could become corrupt. As I've said, it's a work in progress. Tait _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Umut, You can do this; 1 - Create an LVM and put your dom0/domU(s) there. 2 - Create a snapshot volume of that LVM. 3 - Dump|Restore from the snapshot to secondary disk and/or network. I have custom dom0 settings like startup params and network bridge scripts fromage, etc... so I like having a dom0 backup. In my case, I have 2 disks, where I dump restore from disk 1 to disk 2 and then copy to the network. I''ve designed it so if disk 1 fails, I can boot from disk 2 immediately. And if the machine or its env fail (like the server room over heating, flooding, earthquake, etc...), I can restore from network backup. Your philosophy should be first and foremost; its not a question of if but when. Technology sux major major a#@ and will be the death of us all, however its too late to go back so prepare for the worst and pray for the best. - Brian On Apr 2, 2009, at 7:18 AM, Umut Arus wrote:> Hi, > > What should I do for a right backup of guests and dom0? What is the > effective way of backup on xen. > > thanks, > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> > Hi, > > What should I do for a right backup of guests and dom0? What is the > effective way of backup on xen. >I use bacula with the director running in its own DomU and the storage daemon running in Dom0 (for performance). Prior to that I have also run both daemons running in Dom0. The advantage of running the director in DomU is that you don''t have to have mysql etc in Dom0. James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Has anyone tried using 2.6.29 file system freezing with lvm snapshotting yet? That should solve the possible corrupt filesystem problem. Kevin On Thu, 2009-04-02 at 08:37 -0700, Tait Clarridge wrote:> > > > -----Original Message----- > > From: xen-users-bounces@lists.xensource.com [mailto:xen-users- > > bounces@lists.xensource.com] On Behalf Of Javier Guerra > > Sent: Thursday, April 02, 2009 11:07 AM > > To: Umut Arus > > Cc: xen-users@lists.xensource.com > > Subject: Re: [Xen-users] Xen Backup Method > > > > 2009/4/2 Umut Arus <umuta@sabanciuniv.edu>: > > > Hi, > > > > > > What should I do for a right backup of guests and dom0? What is > the > > > effective way of backup on xen. > > > > the most general answer, and with the least surprises, is to use any > > network backup solution on the DomU. > > > > -- > > Javier > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users > > Yes, it seems the easiest way to backup a domU is to use the backup > solution that you have been using for your physical servers. Same goes > for dom0. > > You can experiment with other things, but to provide zero downtime and > a clean backup those seem like the best solutions. I have been working > on something that backs up files after degrading a DRBD array (where > the vm files are hosted), but then if you restore a VM it thinks that > the power cord has been pulled. This has zero downtime but a MUCH > higher probability that the VM could become corrupt. As I''ve said, > it''s a work in progress. > > Tait > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Apr 7, 2009 at 1:46 PM, Kevin Fox <Kevin.Fox@pnl.gov> wrote:> Has anyone tried using 2.6.29 file system freezing with lvm snapshotting > yet? That should solve the possible corrupt filesystem problem.sounds promising. but: 1: its done from ''inside'' the VM, so it might allow better network backup systems but not replace them. 2: it does nothing about complex applications with their own caches and file structures: IOW, databases. if you get an ''in VM'' snapshot capability, you still have to flush caches and suspend operations for the snapshot. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello, I want to properly backup on virtual servers. Has anyone tried lvm backup with scripts or scheduled commands everyday? Could you shared with me? And should be guests and Dom0 shutoff state before snapshot? Another one, which directories should backup other than /var/lib/xen/images? thanks, Javier Guerra wrote:> On Tue, Apr 7, 2009 at 1:46 PM, Kevin Fox <Kevin.Fox@pnl.gov> wrote: > >> Has anyone tried using 2.6.29 file system freezing with lvm snapshotting >> yet? That should solve the possible corrupt filesystem problem. >> > > sounds promising. but: > > 1: its done from ''inside'' the VM, so it might allow better network > backup systems but not replace them. > > 2: it does nothing about complex applications with their own caches > and file structures: IOW, databases. if you get an ''in VM'' snapshot > capability, you still have to flush caches and suspend operations for > the snapshot. > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Umut, I cron a script running once every early morning that does; 1 - pause the domUs * optional, but state will be accurate if paused. 2 - create/recreate a snapshot of the volume where my dom0/domUs live. 3 - mount a separate drive and dump/restore the snapshot onto a separate drive. 4 - unmount the drive. 5 - send mail when all is done. - Brian On Apr 7, 2009, at 2:01 PM, Umut Arus wrote:> Hello, > > I want to properly backup on virtual servers. Has anyone tried lvm > backup with scripts or scheduled commands everyday? Could you shared > with me? And should be guests and Dom0 shutoff state before snapshot? > > Another one, which directories should backup other than /var/lib/xen/ > images? > > thanks, > > Javier Guerra wrote: >> >> On Tue, Apr 7, 2009 at 1:46 PM, Kevin Fox <Kevin.Fox@pnl.gov> wrote: >> >>> Has anyone tried using 2.6.29 file system freezing with lvm >>> snapshotting >>> yet? That should solve the possible corrupt filesystem problem. >>> >> sounds promising. but: >> >> 1: its done from ''inside'' the VM, so it might allow better network >> backup systems but not replace them. >> >> 2: it does nothing about complex applications with their own caches >> and file structures: IOW, databases. if you get an ''in VM'' snapshot >> capability, you still have to flush caches and suspend operations for >> the snapshot. >> >> >> > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, 2009-04-07 at 12:30 -0700, Javier Guerra wrote:> On Tue, Apr 7, 2009 at 1:46 PM, Kevin Fox <Kevin.Fox@pnl.gov> wrote: > > Has anyone tried using 2.6.29 file system freezing with lvm > snapshotting > > yet? That should solve the possible corrupt filesystem problem. > > sounds promising. but: > > 1: its done from ''inside'' the VM, so it might allow better network > backup systems but not replace them.So, in a single kernel (2.6.29+), lvm can tell the file system using the block device to freeze so it can take its snapshot, and then tell it to unfreeze. How hard would this be to extend this so that an LVM sitting in Dom0 could forward the freeze calls to DomU? Say, through the virtual block devices. You could then run your backups in Dom0 instead of every DomU. This would save you on per client licensing fees.> 2: it does nothing about complex applications with their own caches > and file structures: IOW, databases. if you get an ''in VM'' snapshot > capability, you still have to flush caches and suspend operations for > the snapshot.A lot of backup systems I''ve seen tough, largely just save the files, or ignore them if they are written while saving. They don''t bother to flush caches and suspend. It doesn''t work for everything (databases), but for most applications it is fine. One example use case would be each users desktop session gets a virtual machine. You could backup the users data without needing to have a backup client in the vm. Kevin> > -- > Javier > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I have a similar script but it does not perform the first step (pausing DomU) before running. Do you care to share the syntax of that step in your script? Tom Jensen | President Digital Toolbox Quoting Brian Krusic <brian@krusic.com>:> Hi Umut, > > I cron a script running once every early morning that does; > > 1 - pause the domUs * optional, but state will be accurate if paused. > 2 - create/recreate a snapshot of the volume where my dom0/domUs live. > 3 - mount a separate drive and dump/restore the snapshot onto a > separate drive. > 4 - unmount the drive. > 5 - send mail when all is done. > > - Brian > > > > > On Apr 7, 2009, at 2:01 PM, Umut Arus wrote: > >> Hello, >> >> I want to properly backup on virtual servers. Has anyone tried lvm >> backup with scripts or scheduled commands everyday? Could you >> shared with me? And should be guests and Dom0 shutoff state before >> snapshot? >> >> Another one, which directories should backup other than >> /var/lib/xen/ images? >> >> thanks, >> >> Javier Guerra wrote: >>> >>> On Tue, Apr 7, 2009 at 1:46 PM, Kevin Fox <Kevin.Fox@pnl.gov> wrote: >>> >>>> Has anyone tried using 2.6.29 file system freezing with lvm snapshotting >>>> yet? That should solve the possible corrupt filesystem problem. >>>> >>> sounds promising. but: >>> >>> 1: its done from ''inside'' the VM, so it might allow better network >>> backup systems but not replace them. >>> >>> 2: it does nothing about complex applications with their own caches >>> and file structures: IOW, databases. if you get an ''in VM'' snapshot >>> capability, you still have to flush caches and suspend operations for >>> the snapshot. >>> >>> >>> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users > >---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Apr 7, 2009 at 4:12 PM, Kevin Fox <Kevin.Fox@pnl.gov> wrote:> So, in a single kernel (2.6.29+), lvm can tell the file system using the > block device to freeze so it can take its snapshot, and then tell it to > unfreeze. How hard would this be to extend this so that an LVM sitting > in Dom0 could forward the freeze calls to DomU? Say, through the virtual > block devices. You could then run your backups in Dom0 instead of every > DomU. This would save you on per client licensing fees.AFAICT, it has nothing to do with the block device. it''s a new feature that some filesystems might choose to implement, to establish a point in time where the filesystem is safely snapshottable without preserving RAM state. XFS has had this feature (and some extra bells/whistles) for years; i don''t know if any non-SGI backup system uses it. in any case, if you want it to be initiated on Dom0, it has to signal the DomU to do all the freezing. also, it might not flush caches to the blockdevice, since it might assume that the snapshot is done over the cache layer (ie. by LVM in the same machine, not the Dom0 ''beneath it'').>> 2: it does nothing about complex applications with their own caches >> and file structures: IOW, databases. if you get an ''in VM'' snapshot >> capability, you still have to flush caches and suspend operations for >> the snapshot. > > A lot of backup systems I''ve seen tough, largely just save the files, or > ignore them if they are written while saving. They don''t bother to flush > caches and suspend. It doesn''t work for everything (databases), but for > most applications it is fine. One example use case would be each users > desktop session gets a virtual machine. You could backup the users data > without needing to have a backup client in the vm.if you''re willing to forgive databases and similar loads, a simple LVM snapshot is more than enough. any current journal FS will easily regain integrity. heck, for desktop users'' data just do rsync to a central location and be done! (extra points if using rsnapshot to keep several ''snapshots'') -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Thomas, I call a script from cron, that has the lines; xm pause domUa xm pause domUb xm pause domUc ..... ..... ..... xm unpause domUa xm unpause domUb xm unpause domUc Nothing fancy as I''m not artistic. xm pause takes 1 arg so I suppose one could create a var that reads all the files in /var/lib/images, strips out the .img and assumes the domUs are named consistently with there corresponding img files and rums the pause command for every instance of a file name w/o needing to name them individually in a script. This would automate things to the next level and be very cool in large envs. My client is fine with what I set up (for now). - Brian On Apr 7, 2009, at 2:25 PM, Thomas Jensen wrote:> I have a similar script but it does not perform the first step > (pausing DomU) before running. Do you care to share the syntax of > that step in your script? > > > Tom Jensen | President > Digital Toolbox > > Quoting Brian Krusic <brian@krusic.com>: > >> Hi Umut, >> >> I cron a script running once every early morning that does; >> >> 1 - pause the domUs * optional, but state will be accurate if paused. >> 2 - create/recreate a snapshot of the volume where my dom0/domUs >> live. >> 3 - mount a separate drive and dump/restore the snapshot onto a >> separate drive. >> 4 - unmount the drive. >> 5 - send mail when all is done. >> >> - Brian >> >> >> >> >> On Apr 7, 2009, at 2:01 PM, Umut Arus wrote: >> >>> Hello, >>> >>> I want to properly backup on virtual servers. Has anyone tried >>> lvm backup with scripts or scheduled commands everyday? Could >>> you shared with me? And should be guests and Dom0 shutoff state >>> before snapshot? >>> >>> Another one, which directories should backup other than /var/lib/ >>> xen/ images? >>> >>> thanks, >>> >>> Javier Guerra wrote: >>>> >>>> On Tue, Apr 7, 2009 at 1:46 PM, Kevin Fox <Kevin.Fox@pnl.gov> >>>> wrote: >>>> >>>>> Has anyone tried using 2.6.29 file system freezing with lvm >>>>> snapshotting >>>>> yet? That should solve the possible corrupt filesystem problem. >>>>> >>>> sounds promising. but: >>>> >>>> 1: its done from ''inside'' the VM, so it might allow better network >>>> backup systems but not replace them. >>>> >>>> 2: it does nothing about complex applications with their own caches >>>> and file structures: IOW, databases. if you get an ''in VM'' >>>> snapshot >>>> capability, you still have to flush caches and suspend operations >>>> for >>>> the snapshot. >>>> >>>> >>>> >>> _______________________________________________ >>> Xen-users mailing list >>> Xen-users@lists.xensource.com >>> http://lists.xensource.com/xen-users >> >> > > > > ---------------------------------------------------------------- > This message was sent using IMP, the Internet Messaging Program._______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, 2009-04-07 at 14:32 -0700, Javier Guerra wrote:> On Tue, Apr 7, 2009 at 4:12 PM, Kevin Fox <Kevin.Fox@pnl.gov> wrote: > > So, in a single kernel (2.6.29+), lvm can tell the file system using > the > > block device to freeze so it can take its snapshot, and then tell it > to > > unfreeze. How hard would this be to extend this so that an LVM > sitting > > in Dom0 could forward the freeze calls to DomU? Say, through the > virtual > > block devices. You could then run your backups in Dom0 instead of > every > > DomU. This would save you on per client licensing fees. > > AFAICT, it has nothing to do with the block device. it''s a new > feature that some filesystems might choose to implement, to establish > a point in time where the filesystem is safely snapshottable without > preserving RAM state.Ah, I see. I misread this part: "Essentially the patch just exports the freeze_bdev() kernel function in a user accessible way." It implements it as an ioctl to a fd for the mount point of the file system. I thought it was passed a block device. My bad. So it looks like its up to the lvm snapshot utility to find the filesystem in question if mounted, do the freeze ioctl, do the snapshot and then run the unfreeze ioctl? To support this, the lvm snapshot tool would have to know that the block device was used by a xen/qemu session and forward the freeze/unfreeze through.> XFS has had this feature (and some extra bells/whistles) for years; i > don''t know if any non-SGI backup system uses it. > > in any case, if you want it to be initiated on Dom0, it has to signal > the DomU to do all the freezing. also, it might not flush caches to > the blockdevice, since it might assume that the snapshot is done over > the cache layer (ie. by LVM in the same machine, not the Dom0 ''beneath > it'').I glanced at the code last week, and I think I remember it doing a sync to the block device, but don''t quote me on that. :) So it should flush the caches.> > >> 2: it does nothing about complex applications with their own caches > >> and file structures: IOW, databases. if you get an ''in VM'' > snapshot > >> capability, you still have to flush caches and suspend operations > for > >> the snapshot. > > > > A lot of backup systems I''ve seen tough, largely just save the > files, or > > ignore them if they are written while saving. They don''t bother to > flush > > caches and suspend. It doesn''t work for everything (databases), but > for > > most applications it is fine. One example use case would be each > users > > desktop session gets a virtual machine. You could backup the users > data > > without needing to have a backup client in the vm. > > if you''re willing to forgive databases and similar loads, a simple LVM > snapshot is more than enough. any current journal FS will easily > regain integrity.Not exactly. I think the whole ext4 uber thread on create/write/rename/no fsync thing showed that there are cases when that can have bad results. I think your pretty safe doing it with ext3 (but there was some mention of a 5 second window that may be a problem). Kevin> heck, for desktop users'' data just do rsync to a central location and > be done! (extra points if using rsnapshot to keep several ''snapshots'') > > > -- > Javier > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Apr 7, 2009 at 5:46 PM, Kevin Fox <Kevin.Fox@pnl.gov> wrote:> Not exactly. I think the whole ext4 uber thread on > create/write/rename/no fsync thing showed that there are cases when that > can have bad results. I think your pretty safe doing it with ext3 (but > there was some mention of a 5 second window that may be a problem).i haven''t tested ext4 yet; but some of those issues are shrugged off with a "XFS does that since forever". and in the end all the blame is to the application, and rsync is the best behaved copy i''ve seen. i''ve done several disaster tests with both ''cp -a'' and rsync, using ext3, XFS and JFS, both with local and AoE disks, pulling the plug on the initiator, the target and the switches. in short: ext3 is the most resilient, by far. the journal rollback (with data=ordered) goes a long way in the past to guarantee only correctly copied files are there. cp loses data on all three filesystems rsync never loses data on ext3, and only one file on ~3% cases on XFS jfs always loses thousands of files. so, i prefer ext3, and if need more than 8TB in one filesystem, i use XFS without real worries. always copying with rsync, of course. it seems that cp does the ''braindead'' (quoting Linus) method. i guess ext4 will be like XFS, but with data=ordered, it should be a little like ext3. in any case, rsync does it correctly. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I''ve been told that Rsnapshot is an excellent way to get a good backup. Would you agree? --------------------------------------------------------------------------------------- 2009/4/8 Javier Guerra <javier@guerrag.com>> On Tue, Apr 7, 2009 at 5:46 PM, Kevin Fox <Kevin.Fox@pnl.gov> wrote: > > Not exactly. I think the whole ext4 uber thread on > > create/write/rename/no fsync thing showed that there are cases when that > > can have bad results. I think your pretty safe doing it with ext3 (but > > there was some mention of a 5 second window that may be a problem). > > i haven''t tested ext4 yet; but some of those issues are shrugged off > with a "XFS does that since forever". and in the end all the blame is > to the application, and rsync is the best behaved copy i''ve seen. > > i''ve done several disaster tests with both ''cp -a'' and rsync, using > ext3, XFS and JFS, both with local and AoE disks, pulling the plug on > the initiator, the target and the switches. > > in short: > > ext3 is the most resilient, by far. the journal rollback (with > data=ordered) goes a long way in the past to guarantee only correctly > copied files are there. > cp loses data on all three filesystems > rsync never loses data on ext3, and only one file on ~3% cases on XFS > jfs always loses thousands of files. > > so, i prefer ext3, and if need more than 8TB in one filesystem, i use > XFS without real worries. always copying with rsync, of course. it > seems that cp does the ''braindead'' (quoting Linus) method. > > i guess ext4 will be like XFS, but with data=ordered, it should be a > little like ext3. in any case, rsync does it correctly. > > > -- > Javier > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users