I''m running several xen servers (sles10sp1 right now). Most of the domU''s are disk img files. They are either linux para-virt''s or windows full-virt''s. I know of the performance arguments for not running them as image files, but right now these work well for us. How is everyone doing backups in a xen world? Are you running backup agents on individual domUs? Can you back up individual domUs (img files) while they are running? Do you need to pause, make backup, un-pause? As we get more and more xen servers and domU''s this is increasingly concerning to me. Right now our disk images are stored on a SAN and mirrored to another location. That''s great, but if an image file gets corrupted, then I''m probably hosed for that server. Yesterday I had an issue where my xen server stopped responding. I could scan ports, and ping it, but otherwise I couldn''t do anything. I had one domU not responding in the same fashion, while another domU seemed to be running ok, but I could not shut that working domU down. In the end, I had to hard boot the server. Thankfully everything came up ok, but it would have been a major headache if it hadn''t. Any thoughts or suggestions are appreciated. Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Pifer <jep@obrien-pifer.com> writes:> How is everyone doing backups in a xen world? > Are you running backup agents on individual domUs?Yes.> Can you back up individual domUs (img files) while they are running?Sure, but it will be an inconsistent backup.> Do you need to pause, make backup, un-pause?That helps, but the backup will still contain dirty filesystems. -- Feri. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Ferenc Wagner wrote:> James Pifer <jep@obrien-pifer.com> writes: > >> How is everyone doing backups in a xen world? >> Are you running backup agents on individual domUs? > > Yes. > >> Can you back up individual domUs (img files) while they are running? > > Sure, but it will be an inconsistent backup. > >> Do you need to pause, make backup, un-pause? > > That helps, but the backup will still contain dirty filesystems.Yeah we thought about that issue and tried to address it. Currently we shutdown our Windows domUs, take an LVM snapshot, mount it, power the domU back on again, then mount and backup the snapshot. It works fairly well, except that sometimes (too often, actually) when the domUs are powered on again they fail to boot up. Greg. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Pifer <jep@obrien-pifer.com> writes:> How is everyone doing backups in a xen world?...> Do you need to pause, make backup, un-pause?I''ve been thinking about this for my setup. I was thinking about doing an ''xm save'' copying the save file, then taking an lvm snapshot (my backing store is lvm) then doing an ''xm restore'' and backing up the lvm snapshot with an ionice''d dd, then removing the lvm snapshot before backing up the next DomU. Now I haven''t actually tried this in production yet, but it seems to work on my test box, and I think it would give me a consistant backup with only the downtime required by ''xm save'' Now, lvm snapshots are pretty expensive, but eh, backups are important, and ram is cheap. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Ferenc Wagner wrote:> James Pifer <jep@obrien-pifer.com> writes: > > >> Can you back up individual domUs (img files) while they are running? >> > > Sure, but it will be an inconsistent backup. > >"Inconsistent" is an understatement. The image file will be changing while you''re backing up; you''ll most likely end up with garbage in the parts of the guest file system that are even vaguely active while you''re doing the back up.>> Do you need to pause, make backup, un-pause? >> > > That helps, but the backup will still contain dirty filesystems. >Sometimes it helps. If you do a save-backup-restore that helps even more, but be aware that if you ever restore and start the backup then it will continue doing what it did at the time of the backup. In fact, everything it had in progress at the time of the backup will be done again. At the "mostly harmless" end of the spectrum you''ll get mail messages that were in a queue re-sent; at the other end of the spectrum ... Well, that depends on the applications. You need the cooperation of those applications because you can''t just blindly restore (and restart) the backed up domU image and hope that everything will be fine. jch _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Sometimes it helps. If you do a save-backup-restore that helps even > more, but be aware that if you ever restore and start the backup then it > will continue doing what it did at the time of the backup. In fact, > everything it had in progress at the time of the backup will be done > again. At the "mostly harmless" end of the spectrum you''ll get mail > messages that were in a queue re-sent; at the other end of the spectrum > ... Well, that depends on the applications. > > You need the cooperation of those applications because you can''t just > blindly restore (and restart) the backed up domU image and hope that > everything will be fine.Thanks for the replies. So, right now, the only way to make a backup of a disk based image file, is to shutdown the domU, copy the img file, and restart the domU. Correct? I suppose for linux pv''s I could at least do this from a script during the night, ie: xm stop [domU-name] cp --sparse=always blah blah xm start [domU-name] Windows I would have to stop manually, otherwise they are shut down hard. I have a somewhat recent version of the GPLPV drivers installed, but I have yet to see my Windows domU''s shutdown nicely. I have not spent much time troubleshooting, but I believe I have the shutdown monitor service install. Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, Oct 15, 2008 at 08:17:10AM -0400, James Pifer wrote:> Thanks for the replies. > So, right now, the only way to make a backup of a disk based image file, > is to shutdown the domU, copy the img file, and restart the domU. > Correct?If the image files are sitting on a LV then you can save, snapshot, copy disk, xm save checkpoint, and domU config. But you still have to have LV somewhere. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users