All, I''ve got one Xen server running CentOS 5, with 3 CentOS virtual machines on it. I have what I hope is a simple question: I''d like to backup the virtual machines. I don''t care about anything but the ability to restore a couple of files and be able to be back up and running quickly. At some point I''ll have a spare server ready to migrate running Xen vm''s to, but at the moment it''s a simple backup I''m looking for. I googled and found this: http://blog.xen.org/index.php/2008/03/13/now-where-are-my-domus/ My question is, if I just want to backup the currently running vm''s can I just backup the disk images and config files or do I need to do as the article suggests and also backup machine states and memory images? Thanks, -- Steven G. Spencer, Network Administrator KSC Corporate - The Kelly Supply Family of Companies Office 308-382-8764 Ext. 231 Mobile 308-380-7957 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Steve Spencer wrote:> My question is, if I just want to backup the currently running vm''s can > I just backup the disk images and config files or do I need to do as the > article suggests and also backup machine states and memory images?If you only need to be able to replace a few files then you dont need the machine states and memory image. If you want a "warm" backup of the full OS then you would want that memory state. So if you want warm backup (think full point in time restore ... running point in time) and if you are using lvm backed domU. xm save lvcreate snapshot (of both disk and swap) xm restore dd the snapshots you created out to files grab the xm save checkpoint, the images you created by dding the disk and swap, and grab the domU config tar it up Then you can if you needed to dd the image back to the lvms, put the domU config in place and run xm restore on the checkpoint, It would put your machine back to the way it was when you first ran xm save. Does that make sense? -- Nick Anderson <nick@anders0n.net> http://www.cmdln.org http://www.anders0n.net _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Steve Spencer wrote:> My question is, if I just want to backup the currently running vm''s can > I just backup the disk images and config files or do I need to do as the > article suggests and also backup machine states and memory images?If you only need a few files it probably easiest to just back them up out of the os. -- Nick Anderson <nick@anders0n.net> http://www.cmdln.org http://www.anders0n.net _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, 2008-05-07 at 16:11 -0500, Steve Spencer wrote:> I''d like to backup the virtual machines. I don''t care about anything > but the ability to restore a couple of files and be able to be back up > and running quickly.I''m using this script: http://www.everlinux.com/blog/2008/04/03/fazendo-backup-das-suas-vms-do-xen/ It takes snapshot of all vm''s, and does one .tar.gz from "/". It''s useful for me. -- Tiago Cruz http://everlinux.com Linux User #282636 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi there, I wrote the blog post originally mentioned.> > My question is, if I just want to backup the currently running vm''s can > > I just backup the disk images and config files or do I need to do as the > > article suggests and also backup machine states and memory images? > > If you only need to be able to replace a few files then you dont need > the machine states and memory image. If you want a "warm" backup of the > full OS then you would want that memory state.I''d like to just clarify this with a little more detail. In general, it''s impossible to guarantee that a disk snapshot taken - even atomically using a LVM snapshot - at an arbitrary point in time from outside the domU will be cleanly mountable. It''s at likely to require some kind of recovery operation; it''s possible (though less likely) that such a snapshot could fail to mount at all. I''d not be surprised if this approach often - even usually - works. I''d just like to point out that it''s not certain. When you take an atomic snapshot of a domU''s state, it''s much the same kind of backup image as you''d get by yanking the power from a server, removing its hard drive and copying it byte-for-byte - often readable but definitely not clean. That consistency problem is what makes it impossible to guarantee that such backups are definitely readable. The advantage of backing up the memory state simultaneously is that the filesystem data that was currently in volatile RAM instead of on-disk is retained. This means that you have everything you need to restore the guest to that point-in-time, then log in and access your backed up files. This is not really ideal, since it''s not always desirable to have to save a domain''s memory state. It would be nicer if you could somehow guarantee a consistent filesystem state on your disk snapshot. I''ve been looking into how to do this and I believe it''s quite possible to modify the code to support this properly. Cheers, Mark PS. Note that the above comments about disk backup all assume you''re doing some kind of atomic backup operation, for instance snapshotting the domain''s VBD from dom0, using LVM. A simplistic approach like directly dd-ing or cp-ing a running domain''s VBD is (as you''d expect) very unlikely to work reliably. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mark Williamson wrote:> This is not really ideal, since it''s not always desirable to have to save a > domain''s memory state. It would be nicer if you could somehow guarantee a > consistent filesystem state on your disk snapshot. I''ve been looking into > how to do this and I believe it''s quite possible to modify the code to > support this properly.Mark, thanks for the reply. My thought is this. If I xm save it saves the memory state and pauses the VM. While the vm is paused I can then snapshot the block device from dom0 of the guest vm. Since the domain is paused there is no disk activity during the snapshot, and anything that was possibly half-written is in the memory state. So I should be able to reliably dd the snapshot since when I try to restore I could try restoring the block device state, and use xm restore on the checkpoint to restore the memory at the same time. Is this not essentially what xm migrate does? Can we not be assured that since we have memory state and block state at that point in time that the data will be consistent? .... When I say assured I mean just as assured as any other backup. There is always the possibility of flipped bits, I am just thinking that having memory state and block state when attempting a restore would be very very close if not the same as having turned the domain off and copied the block device then turn it back on. hope that makes sense and there wasnt too much rambling. -- Nick Anderson <nick@anders0n.net> http://www.cmdln.org http://www.anders0n.net _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nick Anderson wrote:> Mark Williamson wrote: >> This is not really ideal, since it''s not always desirable to have to >> save a domain''s memory state. It would be nicer if you could somehow >> guarantee a consistent filesystem state on your disk snapshot. I''ve >> been looking into how to do this and I believe it''s quite possible to >> modify the code to support this properly. > > Mark, thanks for the reply. > My thought is this. If I xm save it saves the memory state and pauses > the VM. While the vm is paused I can then snapshot the block device from > dom0 of the guest vm. Since the domain is paused there is no disk > activity during the snapshot, and anything that was possibly > half-written is in the memory state. So I should be able to reliably dd > the snapshot since when I try to restore I could try restoring the block > device state, and use xm restore on the checkpoint to restore the > memory at the same time. Is this not essentially what xm migrate does? > Can we not be assured that since we have memory state and block state at > that point in time that the data will be consistent? > > .... When I say assured I mean just as assured as any other backup. > There is always the possibility of flipped bits, I am just thinking that > having memory state and block state when attempting a restore would be > very very close if not the same as having turned the domain off and > copied the block device then turn it back on. > > hope that makes sense and there wasnt too much rambling.Does that sound right? I guess I am really interested how lvm snapshots are thought of as not safe when a domain is paused and you have the memory state. -- Nick Anderson <nick@anders0n.net> http://www.cmdln.org http://www.anders0n.net _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mark Williamson wrote:> When you take an atomic snapshot of a domU''s state, it''s much the same kind of > backup image as you''d get by yanking the power from a server, removing its > hard drive and copying it byte-for-byte - often readable but definitely not > clean. That consistency problem is what makes it impossible to guarantee > that such backups are definitely readable. >Hi Mark, Some thoughts about your comment. In our environment, one of the requirements for the OS and applications is that it must be able to recover quickly and cleanly from a power failure. This is actually one of the reasons why we''re starting to migrate some stuff to zfs : because, when used properly, it always guarantee filesystem consistency in the event of power failure. IMHO, even the best commercial SAN-based backup solution uses that assumption, so for practical purposes, atomic backup for domU''s storage should be sufficient.> It would be nicer if you could somehow guarantee a > consistent filesystem state on your disk snapshot. I''ve been looking into > how to do this and I believe it''s quite possible to modify the code to > support this properly. > >It would be interesting if you can make it happen. Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha wrote:> IMHO, even the best commercial SAN-based backup solution uses that > assumption, so for practical purposes, atomic backup for domU''s storage > should be sufficient.So are local logical volume snapshots atomic? I could see a network block device not being quite atomic with lv snapshot, but it seems that local storage on lv should be atomic. I keep reading conflicting data about weather it is or is not atomic. -- Nick Anderson <nick@anders0n.net> http://www.cmdln.org http://www.anders0n.net _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Friday 09 May 2008, Nick Anderson wrote:> Fajar A. Nugraha wrote: > > IMHO, even the best commercial SAN-based backup solution uses that > > assumption, so for practical purposes, atomic backup for domU''s storage > > should be sufficient. > > So are local logical volume snapshots atomic? I could see a network > block device not being quite atomic with lv snapshot, but it seems that > local storage on lv should be atomic. I keep reading conflicting data > about weather it is or is not atomic.yes they are. Fajar''s point is that if you take a snapshot of a running VM''s storage, it reproduces that storage at a point in time where it was mounted and being used. if later you use that storage without the rest of that VM''s state (the RAM and CPU state), it''s just like you had a hard reset, and have to do some fsck. some filesystems are more resilient to that kind of ''failure''; i think ext3 is among the best on that regard, I think some BSD''s can be even better. (i have no idea how ZFS does, but Fajar implies that it''s really good). curiously, one of the worse is XFS. even if it''s one of the more stable filesystems out there, it heavily assumes that it won''t _ever_ get a ''yank of the cord''. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> >> This is not really ideal, since it''s not always desirable to have to > >> save a domain''s memory state. It would be nicer if you could somehow > >> guarantee a consistent filesystem state on your disk snapshot. I''ve > >> been looking into how to do this and I believe it''s quite possible to > >> modify the code to support this properly. > > > > Mark, thanks for the reply. > > My thought is this. If I xm save it saves the memory state and pauses > > the VM. While the vm is paused I can then snapshot the block device from > > dom0 of the guest vm. Since the domain is paused there is no disk > > activity during the snapshot, and anything that was possibly > > half-written is in the memory state. So I should be able to reliably dd > > the snapshot since when I try to restore I could try restoring the block > > device state, and use xm restore on the checkpoint to restore the > > memory at the same time. Is this not essentially what xm migrate does? > > Can we not be assured that since we have memory state and block state at > > that point in time that the data will be consistent? > > > > .... When I say assured I mean just as assured as any other backup. > > There is always the possibility of flipped bits, I am just thinking that > > having memory state and block state when attempting a restore would be > > very very close if not the same as having turned the domain off and > > copied the block device then turn it back on. > > > > hope that makes sense and there wasnt too much rambling. > > Does that sound right? I guess I am really interested how lvm snapshots > are thought of as not safe when a domain is paused and you have the > memory state.You''re correct; it''s safe to take a snapshot of the guest''s filesystem whilst the guest is suspended and take a snapshot of the suspend image at the same time. That gives you all the data that you need to restore the guest by reverting the filesystem to the backup and restoring the save image; I meant to express this in my original e-mail. My intention was just to remind people that merely snapshotting the guest''s storage but not taking the memory save image does not guarantee you the ability to restore, although usually it will probably work OK through filesystem recovery mechanisms. LVM snapshotting a domU is a different case to LVM snapshotting a filesystem in dom0; many filesystems will automatically put themselves into a consistent state if they *know* they''re being snapshotted. This means that OS local LVM snapshots should be mountable. The difference with Xen is that a filesystem within a domU doesn''t *know* that it''s being snapshotted by dom0 so it can''t put itself into a consistent, mountable state. This is the thing that I''ve been looking into fixing. Saving a memory image to create a "warm" backup should work fine in the meantime. Cheers, Mark _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Some thoughts about your comment. > In our environment, one of the requirements for the OS and applications > is that it must be able to recover quickly and cleanly from a power > failure. This is actually one of the reasons why we''re starting to > migrate some stuff to zfs : because, when used properly, it always > guarantee filesystem consistency in the event of power failure. > > IMHO, even the best commercial SAN-based backup solution uses that > assumption, so for practical purposes, atomic backup for domU''s storage > should be sufficient. > > > It would be nicer if you could somehow guarantee a > > consistent filesystem state on your disk snapshot. I''ve been looking > > into how to do this and I believe it''s quite possible to modify the code > > to support this properly. > > It would be interesting if you can make it happen.Some investigation prompted by observations by users on this mailing list revealed that LVM can notify a filesystem *within the same VM* that it''s going to be snapshotted so that it can put itself into a consistent state. I''d just like to add the plumbing so that you can tell a filesystem in another VM "You''re going to get snapshotted by dom0. Make your filesystem consistent so the snapshot will be mountable." Cheers, Mark _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> > IMHO, even the best commercial SAN-based backup solution uses that > > assumption, so for practical purposes, atomic backup for domU''s storage > > should be sufficient. > > So are local logical volume snapshots atomic? I could see a network > block device not being quite atomic with lv snapshot, but it seems that > local storage on lv should be atomic. I keep reading conflicting data > about weather it is or is not atomic.Local LVM snapshots are atomic, I think. It''s just that atomicity of a snapshot is not sufficient to guarantee that the snapshot is *consistent*, due to caching in the guest. Yanking the power cord out of the machine creates a consistent snapshot of its current disk state (by atomically preventing the machine updating its disk ;-)) but that will also need a filesystem recovery before the FS is actually readable. Cheers, Mark _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mark Williamson wrote:> Some investigation prompted by observations by users on this mailing list > revealed that LVM can notify a filesystem *within the same VM* that it''s > going to be snapshotted so that it can put itself into a consistent state. > I''d just like to add the plumbing so that you can tell a filesystem in > another VM "You''re going to get snapshotted by dom0. Make your filesystem > consistent so the snapshot will be mountable." >You really need to go a little further than that. The file system may be self-consistent but the data stored there may not be. For example, in my former job we used to run "omsuspend" to put the (mail) message store into a self-consistent state, that is, to commit all outstanding transactions on the message store so that a backup would come up cleanly. Of course, the message store in question is resiliant enough to cope with what amounts to a sudden power failure, but the you still have to run a consistency check to get everything OK. (That particular message store would have been much better with transaction logs, but that''s not germane to the topic in question.) The point is that it''s not just the file system you need to worry about: applications need to get their house in order to prepare for the backup. This problem is largely solved if you do the backup from the guest in domU, but would clearly need some work to be done outside domU. Taking a snapshot of both memory and disk image does work to a large extent, but in the case of this message store simply bringing up the old memory and disk image suddenly leaves the guest OS wondering what to do with all these network connections it used to have -- it will generally recover and clean-up its aborted transactions, but, for example, messages that were being sent _out_ at the time of the snapshot will be resent because the outgoing connection didn''t acknowledge the message. Depending on what the message is this will vary from the merely annoying all the way through to the downright weird. I would imagine that there are other application domains where restarting a transaction from the restored domU would have rather unpleasant side effects. jch _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nick Anderson wrote:> Nick Anderson wrote: >> Mark Williamson wrote: >>> This is not really ideal, since it''s not always desirable to have to >>> save a domain''s memory state. It would be nicer if you could >>> somehow guarantee a consistent filesystem state on your disk >>> snapshot. I''ve been looking into how to do this and I believe it''s >>> quite possible to modify the code to support this properly. >> >> Mark, thanks for the reply. >> My thought is this. If I xm save it saves the memory state and pauses >> the VM. While the vm is paused I can then snapshot the block device >> from dom0 of the guest vm. Since the domain is paused there is no >> disk activity during the snapshot, and anything that was possibly >> half-written is in the memory state. So I should be able to reliably >> dd the snapshot since when I try to restore I could try restoring the >> block device state, and use xm restore on the checkpoint to restore >> the memory at the same time. Is this not essentially what xm migrate >> does? >> Can we not be assured that since we have memory state and block state >> at that point in time that the data will be consistent? >> >> .... When I say assured I mean just as assured as any other backup. >> There is always the possibility of flipped bits, I am just thinking >> that having memory state and block state when attempting a restore >> would be very very close if not the same as having turned the domain >> off and copied the block device then turn it back on. >> >> hope that makes sense and there wasnt too much rambling. > > Does that sound right? I guess I am really interested how lvm > snapshots are thought of as not safe when a domain is paused and you > have the memory state.Any operation that has not paged disk operations out to the disk image have their changes stashed in RAM: this makes databases It''s bad enough when it''s a text file in the middle of being edited, but when it''s a database like Oracle or MySQL or even db4, you''re vulnerable to some real trouble because their ''atomic operations'' won''t be. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Freitag, 9. Mai 2008 15:07 schrieb John Haxby: [..]> Taking a snapshot of both memory and disk image does work to a large > extent, but in the case of this message store simply bringing up the old > memory and disk image suddenly leaves the guest OS wondering what to do > with all these network connections it used to have -- it will generally > recover and clean-up its aborted transactions, but, for example, > messages that were being sent _out_ at the time of the snapshot will be > resent because the outgoing connection didn''t acknowledge the message. > Depending on what the message is this will vary from the merely annoying > all the way through to the downright weird. > I would imagine that there are other application domains where > restarting a transaction from the restored domU would have rather > unpleasant side effects.and i think this is the point. You have to consider what you want. Fast recovery to a given time with the loss of all computed data since there you are fine with saved memory, state and lvm snapshot. But normally you won''t loose any bit, e.g. xen host dies and takes all domUs down. You won''t recover from a state hours ago, instead you would start the domUs. And if you copy the snapshots somewhere it would take a long time to put the disk data in place. If you you leave the lvm snapshot, perhaps making multiple snapshots it will highly decrease disk io performance. My normal backup strategy is as follows: * Let the domU make consistent backups of important data such as databases. A lot of (big) applications have commands to let them prepare for backup (let them make the data consistent on disk). Or just to write-lock the application and sync to disk. This state must only consist a shot period of time till the snapshot is created. * Make lvm snapshots of the disks outside domU while domUs are running (it''s like turning off the computer/harddisks without stopping the domUs). * Inside domUs release locks, say application that they can continue read/write. * Mount these snapshots to let the fs make the fs consistent (not the data). * Backup the files to somewhere (i use rsync with hardlink copies on a logical volume on the same volume group). * Umount snapshots and release snapshots. With this i have the following options: * Recover single files from backup without interrupting domU. * Recover databases with database dumps without interrupting domU. * If a domU dies unexpectly just start it, the fs should play back journal and so the fs is consistent. If a database doesn''t come up, take the dumps from the backup. * If domU gets badly destroyed like fs error or a lot of real harddisk failures i only have to make new fs for the domU, copy files from backup on it and start the domU (this is a desaster recovery). This is very similiar to the here discussed backup strategy. But in my experience it is a lot faster than handling big dd images or having a lot of snapshots active. The only thing i don''t have is a running application. But as mentioned here already, it could be very useless to have a running cpu managing a connection which is discarded long ago. I do this for about two years and i''ve made about five desaster recoveries (bacause of user failures) and normally i''m asked to bring back single files or databases without interrupting the whole domU. Doing a full desaster recovery is only an option to me if nothing is left (like deleted/overwritten filesystem). PS: And very new as bonus for my users i''ve managed to include /backup/YYYY-MM-DD/fullfilesystem over sshfs in all domUs so that users can easily get files out of /backup without consulting the backup operators (well, this is linux only for now). -- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users