Hi all. I''m using sles11 for my xen servers. I would really like to be able to do hot snapshots of disk AND memory, and then have these snapshots backed up to tape or off site for disaster recovery. I''m thinking this would be done weekly or monthly, not nightly. Two questions: 1) Is this even possible? 2) I could probably have the systems shutdown to make a clean backups, but it would need to be fast. My current file system would take too long. 3) What cluster file system is supported? I have sles11 with a Xiotech SAN. I thought I could do this with CLVM, but just found out I can''t. So how would I do it? (even if I have to shut the domU down first) Right now I''m using ocfs2 and growable storage, which definitely doesn''t support snapshots and copies take a pretty long time. Suggestions are appreciated. Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Apr 09, 2010 at 08:57:16PM -0400, James Pifer wrote:> Hi all. I''m using sles11 for my xen servers. I would really like to be > able to do hot snapshots of disk AND memory, and then have these > snapshots backed up to tape or off site for disaster recovery. I''m > thinking this would be done weekly or monthly, not nightly. > > Two questions: > > 1) Is this even possible? >1) xm save <guest> <guest>.save 2) save a copy of the guest disks 3) save a copy of the <guest>.save file 4) xm restore <guest>.save Remember that for PV guests the guest kernel needs to have proper/working save/restore/migration support. SLES11 kernels should have.> 2) I could probably have the systems shutdown to make a clean backups, > but it would need to be fast. My current file system would take too > long. > > 3) What cluster file system is supported? I have sles11 with a Xiotech > SAN. I thought I could do this with CLVM, but just found out I can''t. So > how would I do it? (even if I have to shut the domU down first) > > Right now I''m using ocfs2 and growable storage, which definitely doesn''t > support snapshots and copies take a pretty long time. > > Suggestions are appreciated. >Hopefully that helps. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
1) xm save <guest>> <guest>.save2) save a copy of the guest disks 3) save a copy of the> <guest>.save file4) xm restore <guest>.save I have put a feature request in for 4.1 to allow a domain to be saved but left in a paused state, so it can be simply unpaused after the storage is snapshotted. That will save time as the memory will not need to be reloaded. I hope it is developed. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sat, 2010-04-10 at 13:55 +0300, Pasi Kärkkäinen wrote:> On Fri, Apr 09, 2010 at 08:57:16PM -0400, James Pifer wrote: > > Hi all. I''m using sles11 for my xen servers. I would really like to be > > able to do hot snapshots of disk AND memory, and then have these > > snapshots backed up to tape or off site for disaster recovery. I''m > > thinking this would be done weekly or monthly, not nightly. > > > > Two questions: > > > > 1) Is this even possible? > > > > 1) xm save <guest> <guest>.save > 2) save a copy of the guest disks > 3) save a copy of the <guest>.save file > 4) xm restore <guest>.save >Pasi, Thanks for the reply. Questions based on your response: 1) Is this dependent on the file system at all? 2) When you do an xm save, does it automatically pause the domU and then basically make a copy of the disk? 3) I assume I would expect the process to take the same amount of time as a normal copy of the disk? Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sat, Apr 10, 2010 at 08:58:59AM -0400, James Pifer wrote:> On Sat, 2010-04-10 at 13:55 +0300, Pasi Kärkkäinen wrote: > > On Fri, Apr 09, 2010 at 08:57:16PM -0400, James Pifer wrote: > > > Hi all. I''m using sles11 for my xen servers. I would really like to be > > > able to do hot snapshots of disk AND memory, and then have these > > > snapshots backed up to tape or off site for disaster recovery. I''m > > > thinking this would be done weekly or monthly, not nightly. > > > > > > Two questions: > > > > > > 1) Is this even possible? > > > > > > > 1) xm save <guest> <guest>.save > > 2) save a copy of the guest disks > > 3) save a copy of the <guest>.save file > > 4) xm restore <guest>.save > > > > Pasi, > > Thanks for the reply. Questions based on your response: > > 1) Is this dependent on the file system at all? >No, it''s not.> 2) When you do an xm save, does it automatically pause the domU and then > basically make a copy of the disk? >No, it doesn''t. Just try it. "xm save" will pause the guest, and then save the cpu and memory state to a file, and then stop the domain. Later you can resume (restore) the guest from the save-file. It doesn''t do anything with the storage/disks.> 3) I assume I would expect the process to take the same amount of time > as a normal copy of the disk? >Yes. When the guest is saved/stopped, you can take a backup of the disks, and store the backup with the state/save-file. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Pasi, Am Montag, den 12.04.2010, 09:20 +0300 schrieb Pasi Kärkkäinen:> On Sat, Apr 10, 2010 at 08:58:59AM -0400, James Pifer wrote: > > On Sat, 2010-04-10 at 13:55 +0300, Pasi Kärkkäinen wrote: > > > On Fri, Apr 09, 2010 at 08:57:16PM -0400, James Pifer wrote: > > > > Hi all. I''m using sles11 for my xen servers. I would really like to be > > > > able to do hot snapshots of disk AND memory, and then have these > > > > snapshots backed up to tape or off site for disaster recovery. I''m > > > > thinking this would be done weekly or monthly, not nightly. > > > > > > > > Two questions: > > > > > > > > 1) Is this even possible? > > > > > > > > > > 1) xm save <guest> <guest>.save > > > 2) save a copy of the guest disks > > > 3) save a copy of the <guest>.save file > > > 4) xm restore <guest>.save > > > > >..........> When the guest is saved/stopped, you can take a backup of the disks, > and store the backup with the state/save-file.how does this behave with blktap2? i read that blktap2 passes all disk I/O-requests from VMs to the userspace deamon through a character device. As i read the release-notes correctly we should get a consistent FS, without "xm save" just through using blktap2/vhd?> > -- Pasibest regards, thomas _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Mon, Apr 12, 2010 at 09:31:59AM +0200, Thomas Halinka wrote:> Hi Pasi, > > Am Montag, den 12.04.2010, 09:20 +0300 schrieb Pasi Kärkkäinen: > > On Sat, Apr 10, 2010 at 08:58:59AM -0400, James Pifer wrote: > > > On Sat, 2010-04-10 at 13:55 +0300, Pasi Kärkkäinen wrote: > > > > On Fri, Apr 09, 2010 at 08:57:16PM -0400, James Pifer wrote: > > > > > Hi all. I''m using sles11 for my xen servers. I would really like to be > > > > > able to do hot snapshots of disk AND memory, and then have these > > > > > snapshots backed up to tape or off site for disaster recovery. I''m > > > > > thinking this would be done weekly or monthly, not nightly. > > > > > > > > > > Two questions: > > > > > > > > > > 1) Is this even possible? > > > > > > > > > > > > > 1) xm save <guest> <guest>.save > > > > 2) save a copy of the guest disks > > > > 3) save a copy of the <guest>.save file > > > > 4) xm restore <guest>.save > > > > > > > > .......... > > When the guest is saved/stopped, you can take a backup of the disks, > > and store the backup with the state/save-file. > > how does this behave with blktap2? i read that blktap2 passes all disk > I/O-requests from VMs to the userspace deamon through a character > device. > > As i read the release-notes correctly we should get a consistent FS, > without "xm save" just through using blktap2/vhd? >Depends what you mean with ''consistent''. If you want to do a disk snapshot online then you always need to coordinate the snapshot with the guest OS/kernel/apps - the guest needs to have the apps in a consistent state and all the buffers flushed when you take the disk snapshot. Windows provides VSS framework for this, but there''s nothing general in Linux for this. And also you need to coordinate that stuff with the snapshot, have the timing correct. So, even if you used blktap2/vhd, you''d have to trigger and coordinate the ''prepare apps and flush caches'' in the guest to happen at the correct time for the disk snapshot to be consistent. XenServer/XCP has method for this, through the Citrix windows PV drivers. So yeah.. blktap2 is just a part of the solution. You need more to actually do it properly. So "xm save" method is easier.. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Pasi, Am Montag, den 12.04.2010, 10:39 +0300 schrieb Pasi Kärkkäinen:> On Mon, Apr 12, 2010 at 09:31:59AM +0200, Thomas Halinka wrote: > > Hi Pasi, > > > > Am Montag, den 12.04.2010, 09:20 +0300 schrieb Pasi Kärkkäinen: > > > On Sat, Apr 10, 2010 at 08:58:59AM -0400, James Pifer wrote: > > > > On Sat, 2010-04-10 at 13:55 +0300, Pasi Kärkkäinen wrote: > > > > > On Fri, Apr 09, 2010 at 08:57:16PM -0400, James Pifer wrote: > > > > > > Hi all. I''m using sles11 for my xen servers. I would really like to be > > > > > > able to do hot snapshots of disk AND memory, and then have these > > > > > > snapshots backed up to tape or off site for disaster recovery. I''m > > > > > > thinking this would be done weekly or monthly, not nightly. > > > > > > > > > > > > Two questions: > > > > > > > > > > > > 1) Is this even possible? > > > > > > > > > > > > > > > > 1) xm save <guest> <guest>.save > > > > > 2) save a copy of the guest disks > > > > > 3) save a copy of the <guest>.save file > > > > > 4) xm restore <guest>.save > > > > > > > > > > > .......... > > > When the guest is saved/stopped, you can take a backup of the disks, > > > and store the backup with the state/save-file. > > > > how does this behave with blktap2? i read that blktap2 passes all disk > > I/O-requests from VMs to the userspace deamon through a character > > device. > > > > As i read the release-notes correctly we should get a consistent FS, > > without "xm save" just through using blktap2/vhd? > > > > Depends what you mean with ''consistent''.consistent means, that all buffer-caches and IO-queues were written/flushed to disks> > If you want to do a disk snapshot online then you always need to coordinate > the snapshot with the guest OS/kernel/apps - the guest needs to have > the apps in a consistent state and all the buffers flushed when you take the disk snapshot.Yeah, but i understood, that xen 4.0 implemented "some magic" around this topic.> > Windows provides VSS framework for this, but there''s nothing general in Linux for this. > And also you need to coordinate that stuff with the snapshot, have the timing correct. > > So, even if you used blktap2/vhd, you''d have to trigger and coordinate the ''prepare apps and flush caches'' > in the guest to happen at the correct time for the disk snapshot to be consistent.The XEN-Datasheet (http://www.xen.org/files/Xen_4_0_Datasheet.pdf) says: .... Blktap2 A new virtual hard disk (VHD) implementation delivers high performance VM snapshots and cloning features as well as the ability to do live virtual disk snapshots without stopping a VM process. So i thought it just works, eg through some kernel-hacking in pvops, or whatever.> XenServer/XCP has method for this, through the Citrix windows PV drivers. > So yeah.. blktap2 is just a part of the solution. You need more to actually do it properly.Hmm, ok - just found it in the docs, too .. 114 Snapshots: 115 116 Pausing a guest will also plug the corresponding IO queue for blktap2 117 devices and stop blktap2 drivers. This can be used to implement a 118 safe live snapshot of qcow and vhd disks. An example script "xmsnap" 119 is shown in the tools/blktap2/drivers directory. This script will 120 perform a live snapshot of a qcow disk. VHD files can use the 121 "vhd-util snapshot" tool discussed above. If this snapshot command is 122 applied to a raw file mounted with tap:tapdisk:AIO, include the -m 123 flag and the driver will be reloaded as VHD. If applied to an already 124 mounted VHD file, omit the -m flag. 125 So my next question is: blktap2/vhd seems great to do snapshots and clones and will have future-support for thin-provisionig (like pre-allocation), but are there any advantages over lvm at the moment?> So "xm save" method is easier.. > > -- Pasi >thanks, thomas _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Mon, Apr 12, 2010 at 10:40:01AM +0200, Thomas Halinka wrote:> Hi Pasi, > > Am Montag, den 12.04.2010, 10:39 +0300 schrieb Pasi Kärkkäinen: > > On Mon, Apr 12, 2010 at 09:31:59AM +0200, Thomas Halinka wrote: > > > Hi Pasi, > > > > > > Am Montag, den 12.04.2010, 09:20 +0300 schrieb Pasi Kärkkäinen: > > > > On Sat, Apr 10, 2010 at 08:58:59AM -0400, James Pifer wrote: > > > > > On Sat, 2010-04-10 at 13:55 +0300, Pasi Kärkkäinen wrote: > > > > > > On Fri, Apr 09, 2010 at 08:57:16PM -0400, James Pifer wrote: > > > > > > > Hi all. I''m using sles11 for my xen servers. I would really like to be > > > > > > > able to do hot snapshots of disk AND memory, and then have these > > > > > > > snapshots backed up to tape or off site for disaster recovery. I''m > > > > > > > thinking this would be done weekly or monthly, not nightly. > > > > > > > > > > > > > > Two questions: > > > > > > > > > > > > > > 1) Is this even possible? > > > > > > > > > > > > > > > > > > > 1) xm save <guest> <guest>.save > > > > > > 2) save a copy of the guest disks > > > > > > 3) save a copy of the <guest>.save file > > > > > > 4) xm restore <guest>.save > > > > > > > > > > > > > > .......... > > > > When the guest is saved/stopped, you can take a backup of the disks, > > > > and store the backup with the state/save-file. > > > > > > how does this behave with blktap2? i read that blktap2 passes all disk > > > I/O-requests from VMs to the userspace deamon through a character > > > device. > > > > > > As i read the release-notes correctly we should get a consistent FS, > > > without "xm save" just through using blktap2/vhd? > > > > > > > Depends what you mean with ''consistent''. > > consistent means, that all buffer-caches and IO-queues were > written/flushed to disks >That requires coordination with the guest OS. It cannot be done only from the hypervisor/dom0. Doing the snapshot only from dom0 without any coordination with the guest is called ''crash consistent'' snapshot. It''s roughly the same as if you pulled the power plug from a physical computer. You''ll get fsck when you restore that kind of snapshot and start the guest. And some applications might be in a bad state.> > > > If you want to do a disk snapshot online then you always need to coordinate > > the snapshot with the guest OS/kernel/apps - the guest needs to have > > the apps in a consistent state and all the buffers flushed when you take the disk snapshot. > > Yeah, but i understood, that xen 4.0 implemented "some magic" around > this topic. >Xen 4.0 has better tools for doing the dom0/hypervisor side of it, but you still need to do the guest OS side yourself, or by using some scripts together with the Xen tools.> > > > Windows provides VSS framework for this, but there''s nothing general in Linux for this. > > And also you need to coordinate that stuff with the snapshot, have the timing correct. > > > > So, even if you used blktap2/vhd, you''d have to trigger and coordinate the ''prepare apps and flush caches'' > > in the guest to happen at the correct time for the disk snapshot to be consistent. > > The XEN-Datasheet (http://www.xen.org/files/Xen_4_0_Datasheet.pdf) says: > > .... > Blktap2 > A new virtual hard disk (VHD) > implementation delivers high > performance VM snapshots and > cloning features as well as the > ability to do live virtual disk > snapshots without stopping a VM > process. > > So i thought it just works, eg through some kernel-hacking in pvops, or whatever. >Yeah well.. like said, that''s only the hypervisor/dom0 bits of it. Remember Xen 4.0 is just the core hypervisor, like an engine for a car. XCP implements the ''other'' needed bits (vm-snapshot-with-quiesce) through Citrix Windows PV drivers. XCP also has/uses blktap2.> > XenServer/XCP has method for this, through the Citrix windows PV drivers. > > So yeah.. blktap2 is just a part of the solution. You need more to actually do it properly. > > Hmm, ok - just found it in the docs, too > > .. > > 114 Snapshots: > 115 > 116 Pausing a guest will also plug the corresponding IO queue for blktap2 > 117 devices and stop blktap2 drivers. This can be used to implement a > 118 safe live snapshot of qcow and vhd disks. An example script "xmsnap" > 119 is shown in the tools/blktap2/drivers directory. This script will > 120 perform a live snapshot of a qcow disk. VHD files can use the > 121 "vhd-util snapshot" tool discussed above. If this snapshot command is > 122 applied to a raw file mounted with tap:tapdisk:AIO, include the -m > 123 flag and the driver will be reloaded as VHD. If applied to an already > 124 mounted VHD file, omit the -m flag. > 125 >See: xe vm-snapshot-with-quiesce That coordinates the backup with Windows guests using VSS, so that the applications are in a known/good state, and the filesystem/kernel has flushed all the buffers/caches.> So my next question is: > > blktap2/vhd seems great to do snapshots and clones and will have future-support for > thin-provisionig (like pre-allocation), but are there any advantages over lvm at the moment? >I haven''t done any benchmarks myself, but the blktap2 snapshots might be faster. Feel free to try and report back. And of course blktap2 has support for the VHD format. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Mon, Apr 12, 2010 at 4:01 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:> You''ll get fsck when you restore that kind of snapshot and start the guest. > And some applications might be in a bad state.even worse: if you mount the filesystem to perform the backup while the VM is suspended to disk (by xm save), it would automatically do a journal replay/revert/whatever. suppose it goes smoothly and you now have a consistent FS (usually it wil); now you resume the VM and.... it would complain, most likely crash, and maybe corrupt the filesystem why? simple, you changed the filesystem (by mounting it) while it was sleeping. from the point of view of the VM, in a blink, and without any warning, several critical disk structures were changed and the''re no longer consistent with the memory state. even if you mount the filesystem as R/O, most would perform a journal replay on mounting. there''s usually an obscure mount option to block it, or you might set the block device as R/O, but then you don''t have a consistent FS to backup. Better save the replay to a different medium, like a COW file, or an LVM R/W snapshot. and we''re back at the beginning.... and still it''s not a full backup, since the guest didn''t have any chance to flush it''s buffers or userspace caches (like all DBs do) really, just do backups from inside the guest. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Am Montag, den 12.04.2010, 09:25 -0500 schrieb Javier Guerra Giraldez:> On Mon, Apr 12, 2010 at 4:01 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote: > > You''ll get fsck when you restore that kind of snapshot and start the guest. > > And some applications might be in a bad state. > > even worse: if you mount the filesystem to perform the backup while > the VM is suspended to disk (by xm save), it would automatically do a > journal replay/revert/whatever. suppose it goes smoothly and you now > have a consistent FS (usually it wil); now you resume the VM and.... > it would complain, most likely crash, and maybe corrupt the filesystem > > why? simple, you changed the filesystem (by mounting it) while it was > sleeping. from the point of view of the VM, in a blink, and without > any warning, several critical disk structures were changed and the''re > no longer consistent with the memory state. > > even if you mount the filesystem as R/O, most would perform a journal > replay on mounting. there''s usually an obscure mount option to block > it, or you might set the block device as R/O, but then you don''t have > a consistent FS to backup. Better save the replay to a different > medium, like a COW file, or an LVM R/W snapshot. > > and we''re back at the beginning.... > > and still it''s not a full backup, since the guest didn''t have any > chance to flush it''s buffers or userspace caches (like all DBs do)Full ACK> really, just do backups from inside the guest.NACK - since Citrix has xe vm-snapshot :-) Is there something like "xm snapshot" planned for OS-XEN? i cant find anything @http://wiki.xensource.com/xenwiki/XenRoadMap Best Regards, Thomas _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi again, Am Montag, den 12.04.2010, 12:01 +0300 schrieb Pasi Kärkkäinen:> On Mon, Apr 12, 2010 at 10:40:01AM +0200, Thomas Halinka wrote:.....> > > Depends what you mean with ''consistent''. > > > > consistent means, that all buffer-caches and IO-queues were > > written/flushed to disks > > > > That requires coordination with the guest OS. It cannot be done > only from the hypervisor/dom0. > > Doing the snapshot only from dom0 without any coordination with the guest > is called ''crash consistent'' snapshot. It''s roughly the same as if you > pulled the power plug from a physical computer. > > You''ll get fsck when you restore that kind of snapshot and start the guest. > And some applications might be in a bad state. > > > > > > > If you want to do a disk snapshot online then you always need to coordinate > > > the snapshot with the guest OS/kernel/apps - the guest needs to have > > > the apps in a consistent state and all the buffers flushed when you take the disk snapshot. > > > > Yeah, but i understood, that xen 4.0 implemented "some magic" around > > this topic. > > > > Xen 4.0 has better tools for doing the dom0/hypervisor side of it, > but you still need to do the guest OS side yourself, > or by using some scripts together with the Xen tools.are there any examples for such triggering from within xen tools?> > > > > > > Windows provides VSS framework for this, but there''s nothing general in Linux for this. > > > And also you need to coordinate that stuff with the snapshot, have the timing correct. > > > > > > So, even if you used blktap2/vhd, you''d have to trigger and coordinate the ''prepare apps and flush caches'' > > > in the guest to happen at the correct time for the disk snapshot to be consistent. > > > > The XEN-Datasheet (http://www.xen.org/files/Xen_4_0_Datasheet.pdf) says: > > > > .... > > Blktap2 > > A new virtual hard disk (VHD) > > implementation delivers high > > performance VM snapshots and > > cloning features as well as the > > ability to do live virtual disk > > snapshots without stopping a VM > > process. > > > > So i thought it just works, eg through some kernel-hacking in pvops, or whatever. > > > > Yeah well.. like said, that''s only the hypervisor/dom0 bits of it. > Remember Xen 4.0 is just the core hypervisor, like an engine for a car. > > XCP implements the ''other'' needed bits (vm-snapshot-with-quiesce) > through Citrix Windows PV drivers.is something equal planned for OS-XEN, too? Something on the road-map or some outstanding projects?> > XCP also has/uses blktap2. > > > > XenServer/XCP has method for this, through the Citrix windows PV drivers. > > > So yeah.. blktap2 is just a part of the solution. You need more to actually do it properly. > > > > Hmm, ok - just found it in the docs, too > > > > .. > > > > 114 Snapshots: > > 115 > > 116 Pausing a guest will also plug the corresponding IO queue for blktap2 > > 117 devices and stop blktap2 drivers. This can be used to implement a > > 118 safe live snapshot of qcow and vhd disks. An example script "xmsnap" > > 119 is shown in the tools/blktap2/drivers directory. This script will > > 120 perform a live snapshot of a qcow disk. VHD files can use the > > 121 "vhd-util snapshot" tool discussed above. If this snapshot command is > > 122 applied to a raw file mounted with tap:tapdisk:AIO, include the -m > > 123 flag and the driver will be reloaded as VHD. If applied to an already > > 124 mounted VHD file, omit the -m flag. > > 125 > > > > See: > xe vm-snapshot-with-quiesceyeah, i know about that> > That coordinates the backup with Windows guests using VSS, so that the applications > are in a known/good state, and the filesystem/kernel has flushed all the buffers/caches. > > > So my next question is: > > > > blktap2/vhd seems great to do snapshots and clones and will have future-support for > > thin-provisionig (like pre-allocation), but are there any advantages over lvm at the moment? > > > > I haven''t done any benchmarks myself, but the blktap2 snapshots might be faster. > Feel free to try and report back. > > And of course blktap2 has support for the VHD format. > > -- Pasi > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Mon, Apr 12, 2010 at 09:25:52AM -0500, Javier Guerra Giraldez wrote:> On Mon, Apr 12, 2010 at 4:01 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote: > > You''ll get fsck when you restore that kind of snapshot and start the guest. > > And some applications might be in a bad state. > > even worse: if you mount the filesystem to perform the backup while > the VM is suspended to disk (by xm save), it would automatically do a > journal replay/revert/whatever. suppose it goes smoothly and you now > have a consistent FS (usually it wil); now you resume the VM and.... > it would complain, most likely crash, and maybe corrupt the filesystem >Uh, that''s why you take a SNAPSHOT of the disk before doing the backup, and then do the backup from the snapshot! You *don''t* touch the original guest disk while it''s paused/saved!!! -- Pasi> why? simple, you changed the filesystem (by mounting it) while it was > sleeping. from the point of view of the VM, in a blink, and without > any warning, several critical disk structures were changed and the''re > no longer consistent with the memory state. > > even if you mount the filesystem as R/O, most would perform a journal > replay on mounting. there''s usually an obscure mount option to block > it, or you might set the block device as R/O, but then you don''t have > a consistent FS to backup. Better save the replay to a different > medium, like a COW file, or an LVM R/W snapshot. > > and we''re back at the beginning.... > > and still it''s not a full backup, since the guest didn''t have any > chance to flush it''s buffers or userspace caches (like all DBs do) > > > > really, just do backups from inside the guest. > > > > -- > Javier_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Mon, Apr 12, 2010 at 04:40:27PM +0200, Thomas Halinka wrote:> Hi again, > > Am Montag, den 12.04.2010, 12:01 +0300 schrieb Pasi Kärkkäinen: > > On Mon, Apr 12, 2010 at 10:40:01AM +0200, Thomas Halinka wrote: > ..... > > > > Depends what you mean with ''consistent''. > > > > > > consistent means, that all buffer-caches and IO-queues were > > > written/flushed to disks > > > > > > > That requires coordination with the guest OS. It cannot be done > > only from the hypervisor/dom0. > > > > Doing the snapshot only from dom0 without any coordination with the guest > > is called ''crash consistent'' snapshot. It''s roughly the same as if you > > pulled the power plug from a physical computer. > > > > You''ll get fsck when you restore that kind of snapshot and start the guest. > > And some applications might be in a bad state. > > > > > > > > > > If you want to do a disk snapshot online then you always need to coordinate > > > > the snapshot with the guest OS/kernel/apps - the guest needs to have > > > > the apps in a consistent state and all the buffers flushed when you take the disk snapshot. > > > > > > Yeah, but i understood, that xen 4.0 implemented "some magic" around > > > this topic. > > > > > > > Xen 4.0 has better tools for doing the dom0/hypervisor side of it, > > but you still need to do the guest OS side yourself, > > or by using some scripts together with the Xen tools. > > are there any examples for such triggering from within xen tools? >Well.. usually you''d write your own script, which does the magic, and uses xen tools as a part of the script.> > > > > > > > > > Windows provides VSS framework for this, but there''s nothing general in Linux for this. > > > > And also you need to coordinate that stuff with the snapshot, have the timing correct. > > > > > > > > So, even if you used blktap2/vhd, you''d have to trigger and coordinate the ''prepare apps and flush caches'' > > > > in the guest to happen at the correct time for the disk snapshot to be consistent. > > > > > > The XEN-Datasheet (http://www.xen.org/files/Xen_4_0_Datasheet.pdf) says: > > > > > > .... > > > Blktap2 > > > A new virtual hard disk (VHD) > > > implementation delivers high > > > performance VM snapshots and > > > cloning features as well as the > > > ability to do live virtual disk > > > snapshots without stopping a VM > > > process. > > > > > > So i thought it just works, eg through some kernel-hacking in pvops, or whatever. > > > > > > > Yeah well.. like said, that''s only the hypervisor/dom0 bits of it. > > Remember Xen 4.0 is just the core hypervisor, like an engine for a car. > > > > XCP implements the ''other'' needed bits (vm-snapshot-with-quiesce) > > through Citrix Windows PV drivers. > > is something equal planned for OS-XEN, too? Something on the road-map or > some outstanding projects? >Not that I know of. I''ve been thinking of starting a "VSS for Linux" project for a long time, just too busy with everything to actually do it right now :( That''s something we would really much need.. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Mon, Apr 12, 2010 at 9:54 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:> Uh, that''s why you take a SNAPSHOT of the disk before doing the backup, > and then do the backup from the snapshot!then you don''t have to stop the VM, do you? unless you also want to save the VM state with the storage, which would be the only way to preserve userspace buffers... but restoring a full disk/ram/cpu backup even just a few hours after backup would still generate a lot or errors when ''suddenly'' all clients are gone. i really don''t see the point of Dom0 backups. you always need DomU cooperation, and that''s already solved by network backup systems -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Mon, Apr 12, 2010 at 10:10:11AM -0500, Javier Guerra Giraldez wrote:> On Mon, Apr 12, 2010 at 9:54 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote: > > Uh, that''s why you take a SNAPSHOT of the disk before doing the backup, > > and then do the backup from the snapshot! > > then you don''t have to stop the VM, do you? unless you also want to > save the VM state with the storage, which would be the only way to > preserve userspace buffers... but restoring a full disk/ram/cpu backup > even just a few hours after backup would still generate a lot or > errors when ''suddenly'' all clients are gone. > > i really don''t see the point of Dom0 backups. you always need DomU > cooperation, and that''s already solved by network backup systems >This thread started about a need to do disk+memory state backups. The method I described (in the first email) allows you to do that. If you also want to take consistent file backups from the disk snapshot/copy, then you need to do more coordination with the guest. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Mon, 2010-04-12 at 09:20 +0300, Pasi Kärkkäinen wrote:> On Sat, Apr 10, 2010 at 08:58:59AM -0400, James Pifer wrote: > > On Sat, 2010-04-10 at 13:55 +0300, Pasi Kärkkäinen wrote: > > > On Fri, Apr 09, 2010 at 08:57:16PM -0400, James Pifer wrote: > > > > Hi all. I''m using sles11 for my xen servers. I would really like to be > > > > able to do hot snapshots of disk AND memory, and then have these > > > > snapshots backed up to tape or off site for disaster recovery. I''m > > > > thinking this would be done weekly or monthly, not nightly. > > > > > > > > Two questions: > > > > > > > > 1) Is this even possible? > > > > > > > > > > 1) xm save <guest> <guest>.save > > > 2) save a copy of the guest disks > > > 3) save a copy of the <guest>.save file > > > 4) xm restore <guest>.save > > > > > > > Pasi, > > > > Thanks for the reply. Questions based on your response: > > > > 1) Is this dependent on the file system at all? > > > > No, it''s not. > > > 2) When you do an xm save, does it automatically pause the domU and then > > basically make a copy of the disk? > > > > No, it doesn''t. Just try it. "xm save" will pause the guest, and then save the > cpu and memory state to a file, and then stop the domain. > > Later you can resume (restore) the guest from the save-file. > > It doesn''t do anything with the storage/disks. > > > 3) I assume I would expect the process to take the same amount of time > > as a normal copy of the disk? > > > > Yes. > > When the guest is saved/stopped, you can take a backup of the disks, > and store the backup with the state/save-file. > > -- Pasi >Ok, now I understand. Just for background, I do look at each system as a standalone system and we install backup software in many of them. If we ever lost a file system or had a corrupted domU, it would be easier to restore the domUs from a backup than rebuild and restore. That being said, I''m still struggling with what file system we should run. We''re using ocfs2 now, and we''ve had some corruption problems. The above will work well for ocfs2, but it does take a while to copy a sparse disk. If I were to run CLVM or LVM, I could save the state of the CPU/Memory, but how would I backup the disk? Would it be any faster than using sparse files? What other file systems could I use? Again, my requirement is SLES11 with attached Xiotech SAN. Thanks again. James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users