I''ve been testing Xen for a few weeks now and am just about to put it into production use. (I wrote some scripts to create Debian images easily; the tools are included in Debian Sid as "xen-tools".) The only thing troubling me at the moment is backing up the virtual instances. I''m looking to run about four virtual systems upon a host. If the physical host dies then I''m going to be in trouble without backups! I realise I can setup traditional backups within the virtual instances using rsync, scp, or a backup agent. But I''m curious about backing up images live. (Primarily to avoid having to setup near identical scripts upon each instance and make sure they work correctly.) It strikes me that the live-migration support does a lot of the things that a backup of a live system should do - literally sending the contents of the systems image + memory to a new host. Would it be possible to write a simple "fake" migration server to receive the image, write it to disk, and then "abort" gracefully? That would allow a live system to be backed up without taking it offline. Any thoughts, or suggestions, appreciated. Steve -- Debian GNU/Linux System Administration http://www.debian-administration.org/ _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
If I understood your post, you actually want to have recoverable "system images" of the virtual systems as opposed to backing up the data on the virtual systems. Did I get that right? I am very interested in the solution as well. The way I see it, to do "live" back ups, you will have to do it from within the virtual system. You could potential pause, the virtual instance to avoid data integrity issues and migrate. This pause is like to interrupt production, but better than doing off-line I suppose. Thanks Ram On 1/4/06, Steve Kemp < steve@steve.org.uk> wrote:> > > > I''ve been testing Xen for a few weeks now and am just about to > put it into production use. (I wrote some scripts to create > Debian images easily; the tools are included in Debian Sid as > "xen-tools".) > > The only thing troubling me at the moment is backing up the > virtual instances. > > I''m looking to run about four virtual systems upon a host. If > the physical host dies then I''m going to be in trouble without > backups! > > I realise I can setup traditional backups within the virtual > instances using rsync, scp, or a backup agent. But I''m curious > about backing up images live. (Primarily to avoid having to > setup near identical scripts upon each instance and make sure > they work correctly.) > > It strikes me that the live-migration support does a lot of the > things that a backup of a live system should do - literally sending > the contents of the systems image + memory to a new host. > > Would it be possible to write a simple "fake" migration server > to receive the image, write it to disk, and then "abort" gracefully? > > That would allow a live system to be backed up without taking > it offline. > > Any thoughts, or suggestions, appreciated. > > Steve > -- > Debian GNU/Linux System Administration > http://www.debian-administration.org/ > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, Jan 04, 2006 at 09:15:37AM -0800, Ram wrote:> If I understood your post, you actually want to have recoverable > "system images" of the virtual systems as opposed to backing up > the data on the virtual systems. Did I get that right?That is correct. Although now I realise that my "solution" will not work, the migration doesn''t actually copy the image, just the memory. I guess I wasn''t remembering correctly.> The way I see it, to do "live" back ups, you will have to > do it from within the virtual system. You could potential pause, > the virtual instance to avoid data integrity issues and migrate. > This pause is like to interrupt production, but better than doing > off-line I suppose.I think that will be the thing to try working with next. I assume that pausing the image then doing a local rsync will be reasonably quick. But I will need to test it. Steve -- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 1/4/06, Steve Kemp <steve@steve.org.uk> wrote:> > On Wed, Jan 04, 2006 at 09:15:37AM -0800, Ram wrote: > > If I understood your post, you actually want to have recoverable > > "system images" of the virtual systems as opposed to backing up > > the data on the virtual systems. Did I get that right? > > That is correct. > > Although now I realise that my "solution" will not work, the > migration doesn''t actually copy the image, just the memory. I > guess I wasn''t remembering correctly. > > > The way I see it, to do "live" back ups, you will have to > > do it from within the virtual system. You could potential pause, > > the virtual instance to avoid data integrity issues and migrate. > > This pause is like to interrupt production, but better than doing > > off-line I suppose. > > I think that will be the thing to try working with next. I > assume that pausing the image then doing a local rsync will > be reasonably quick. But I will need to test it.Thinking a bit more, you might be better off doing the following: - Pause the virtual system - backup from the physical system - restart the virtual system This will give you the ability to back up "virtual system images" [I think]. Also, this will mean that you can replicate things easily for multiple virtual machines running on the same physical host [you said you had 4 running on one physical host]. Also, if you are already backing up your physical, extending it to do this above might be less work. Thanks Ram Steve> -- > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, Jan 04, 2006 at 09:39:52AM -0800, Ram wrote:> > I think that will be the thing to try working with next. I > > assume that pausing the image then doing a local rsync will > > be reasonably quick. But I will need to test it. > > Thinking a bit more, you might be better off doing the following: > - Pause the virtual system > - backup from the physical system > - restart the virtual systemThats pretty much how I took your suggestion :)> Also, if you are already backing up your physical, extending it to do > this above might be less work.Yes, indeed. The physical host will be backed up - so adding the "copied" virtual images is the obvious thing to do. I can''t backup the live systems directly is the problem. Steve -- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, 4 Jan 2006 17:50:30 +0000 Steve Kemp <steve@steve.org.uk> wrote:> On Wed, Jan 04, 2006 at 09:39:52AM -0800, Ram wrote: > > > > I think that will be the thing to try working with next. I > > > assume that pausing the image then doing a local rsync will > > > be reasonably quick. But I will need to test it. > > > > Thinking a bit more, you might be better off doing the following: > > - Pause the virtual system > > - backup from the physical system > > - restart the virtual system > > Thats pretty much how I took your suggestion :) > > > Also, if you are already backing up your physical, extending it to do > > this above might be less work. > > Yes, indeed. > > The physical host will be backed up - so adding the "copied" virtual > images is the obvious thing to do. I can''t backup the live systems > directly is the problem.Because the kernel buffers might not be flushed to disk, you may either want to save/restore (instead of pause/unpause, and include RAM as part of backup) or sync the disks from inside the VM before backing up. Also, instead of straight rsync, check out rdiff-backup for incremental backups based on rsync, I really like this program: http://www.nongnu.org/rdiff-backup/ Tim _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
If you can consider the data in the kernel buffer as an acceptable loss you can implement LVM backed guests and simply use the LVM snapshot feature. Then you are free to backup the snapshot. I''m running guests with iSCSI root and have configured my iSCSI target server to take LVM snapshots of all the guests every 10 minutes, keeping a few around just incase something goes seriously wrong at some point in the day. The backup server then logs into the iSCSI target snapshots and backs up the guest. In this scenario, the guest and even the dom0 it lives on isn''t involved at all in the backup process. Dan- Tim Freeman wrote:>On Wed, 4 Jan 2006 17:50:30 +0000 >Steve Kemp <steve@steve.org.uk> wrote: > > > >>On Wed, Jan 04, 2006 at 09:39:52AM -0800, Ram wrote: >> >> >> >>>> I think that will be the thing to try working with next. I >>>>assume that pausing the image then doing a local rsync will >>>>be reasonably quick. But I will need to test it. >>>> >>>> >>>Thinking a bit more, you might be better off doing the following: >>>- Pause the virtual system >>>- backup from the physical system >>>- restart the virtual system >>> >>> >> Thats pretty much how I took your suggestion :) >> >> >> >>>Also, if you are already backing up your physical, extending it to do >>>this above might be less work. >>> >>> >> Yes, indeed. >> >> The physical host will be backed up - so adding the "copied" virtual >> images is the obvious thing to do. I can''t backup the live systems >> directly is the problem. >> >> > >Because the kernel buffers might not be flushed to disk, you may either want to >save/restore (instead of pause/unpause, and include RAM as part of backup) or >sync the disks from inside the VM before backing up. > >Also, instead of straight rsync, check out rdiff-backup for incremental backups >based on rsync, I really like this program: >http://www.nongnu.org/rdiff-backup/ > >Tim > > >_______________________________________________ >Xen-users mailing list >Xen-users@lists.xensource.com >http://lists.xensource.com/xen-users > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Steve. Steve Kemp wrote:> I''ve been testing Xen for a few weeks now and am just about to > put it into production use. (I wrote some scripts to create > Debian images easily; the tools are included in Debian Sid as > "xen-tools".) >:-)> The only thing troubling me at the moment is backing up the > virtual instances. > > I''m looking to run about four virtual systems upon a host. If > the physical host dies then I''m going to be in trouble without > backups! > > I realise I can setup traditional backups within the virtual > instances using rsync, scp, or a backup agent. But I''m curious > about backing up images live. (Primarily to avoid having to > setup near identical scripts upon each instance and make sure > they work correctly.) >If you place your images on LVM you should be able to take a snapshot of the filesystem(s) and mount or dump this (you probably have to fsck the snapshot if you are to use dump). That should be rather painless.> It strikes me that the live-migration support does a lot of the > things that a backup of a live system should do - literally sending > the contents of the systems image + memory to a new host. >Live migration does not migrate the filesystem image - so using it for backup purposes is rather hard. :-) -- There are only 10 different kinds of people in the world, those who understand binary, and those who don''t. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, Jan 04, 2006 at 04:59:48PM +0000, Steve Kemp wrote:> The only thing troubling me at the moment is backing up the > virtual instances.[...]> Would it be possible to write a simple "fake" migration server > to receive the image, write it to disk, and then "abort" gracefully?As you''ve realised now, this will only get the memory. I used to put everything on LVM and take snapshots, mount the snapshot in dom0 and use my normal backup process to take care of that. However as the number of snapshots increased, I started experiencing many lockups of the block devices I was snapshotting (i.e. the domUs'' filesystems). If I was lucky I could clear them by restarting domUs, but on a couple of occasions it required reboot of dom0. On reporting this to Red Hat it became clear that I wasn''t the only one, and Red Hat have confirmed that LVM snapshots are not stable enough for production use. Since then I have treated each domU as a separate server for backup purposes.> That would allow a live system to be backed up without taking > it offline.I do not think your pause/unpause or save/restore idea will be acceptable as even an optimistic estimate would put the length of time the domain must be paused for into the "several seconds, possibly minutes" region. That''s enough to seriously affect network traffic. Note also that you have the general backup problem of not being able to guarantee capture of your data in a consistent state, e.g. databases. That''s always goin to be the case though, so rather outside the scope of Xen. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users