Hi all, We are planning to use XEN VPS in production environment. Before it we need to design a realiable backup policy for obvious reasons. I view two approaches: a) Use a "snapshot" system for backup ALL VM file (image). b) Use a traditional backup system (rsync, tar, cpio) for backup SELECTED data into VM. I prefer A option but I''ve read in this list it is not possible unless you stop de VM during a few time (the time required for backup operation itself). ¿What backup solution do you use in your _production_ environment? The most important point is any downtime in VM is not viable. -- Thanks, Jordi Espasa Clofent _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jordi Espasa Clofent wrote:> Hi all, > > We are planning to use XEN VPS in production environment. Before it we > need to design a realiable backup policy for obvious reasons. I view > two approaches: > > a) Use a "snapshot" system for backup ALL VM file (image). > b) Use a traditional backup system (rsync, tar, cpio) for backup > SELECTED data into VM. > > I prefer A option but I''ve read in this list it is not possible unless > you stop de VM during a few time (the time required for backup > operation itself).If your vm image are files, and use LVM snapshot, you don''t need to STOP the VM for backup. This thread might help you : http://lists.xensource.com/archives/html/xen-users/2007-06/msg00688.html Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> If your vm image are files, and use LVM snapshot, you don''t need to STOP > the VM for backup. > This thread might help you : > http://lists.xensource.com/archives/html/xen-users/2007-06/msg00688.htmlGood info, but I''ve not installed dom0 in LVM based system... Maybe I should reconsider this point. -- Thanks, Jordi Espasa Clofent _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
but LVM snapshot still has risk, must be survived from a power failture. correct me if i am wrong... what I do now is shutdown, copy, start.... Fajar A. Nugraha wrote:>Jordi Espasa Clofent wrote: > > >>Hi all, >> >>We are planning to use XEN VPS in production environment. Before it we >>need to design a realiable backup policy for obvious reasons. I view >>two approaches: >> >>a) Use a "snapshot" system for backup ALL VM file (image). >>b) Use a traditional backup system (rsync, tar, cpio) for backup >>SELECTED data into VM. >> >>I prefer A option but I''ve read in this list it is not possible unless >>you stop de VM during a few time (the time required for backup >>operation itself). >> >> > >If your vm image are files, and use LVM snapshot, you don''t need to STOP >the VM for backup. >This thread might help you : >http://lists.xensource.com/archives/html/xen-users/2007-06/msg00688.html > >Regards, > >Fajar > >_______________________________________________ >Xen-users mailing list >Xen-users@lists.xensource.com >http://lists.xensource.com/xen-users > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> but LVM snapshot still has risk, must be survived from a power failture. > correct me if i am wrong... > what I do now is shutdown, copy, start....It seems yes, Xin (almost according the info I''ve read in present mail list). In my case, any downtime is non-viable, so shutdown/copy/start process it''s not a solution for me. -- Thanks, Jordi Espasa Clofent _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Xin Chen wrote:> but LVM snapshot still has risk, must be survived from a power failture. > correct me if i am wrong... > what I do now is shutdown, copy, start.... >Correct. However, for production system, you''d generally design a system that MUST be able to survive a power failure/hard reboot anyway. This means activating archive log (for Oracle), using Innodb (MySQL), using journaling filesystem, etc. So this shouldn''t be an issue. Reading Jordi''s post I assume his priority was how to keep the VM during backups, so LVM snapshot should be a feasible solution. Regards, Fajar> Fajar A. Nugraha wrote: > >> Jordi Espasa Clofent wrote: >> >> >>> Hi all, >>> >>> We are planning to use XEN VPS in production environment. Before it we >>> need to design a realiable backup policy for obvious reasons. I view >>> two approaches: >>> >>> a) Use a "snapshot" system for backup ALL VM file (image). >>> b) Use a traditional backup system (rsync, tar, cpio) for backup >>> SELECTED data into VM. >>> >>> I prefer A option but I''ve read in this list it is not possible unless >>> you stop de VM during a few time (the time required for backup >>> operation itself). >>> >> >> If your vm image are files, and use LVM snapshot, you don''t need to STOP >> the VM for backup. >> This thread might help you : >> http://lists.xensource.com/archives/html/xen-users/2007-06/msg00688.html >> >> Regards, >> >> Fajar_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Hi all, > > We are planning to use XEN VPS in production environment. Before it we > need to design a realiable backup policy for obvious reasons. I view two > approaches: > > a) Use a "snapshot" system for backup ALL VM file (image). > b) Use a traditional backup system (rsync, tar, cpio) for backup > SELECTED data into VM. > > I prefer A option but I''ve read in this list it is not possible unless > you stop de VM during a few time (the time required for backup operation > itself). > > ¿What backup solution do you use in your _production_ environment? The > most important point is any downtime in VM is not viable. >I''m using Bacula (www.bacula.org) currently. I have installed the client (fd) inside each domain (Windows HVM and Linux PV), and just back up to some storage provided by the hosting company. This storage is in turn backed up by them. I''ll be trying a test restore shortly (only just set it up), but that shouldn''t be a big deal. For a restore of a Windows domain (call it A), you should be able to create a new Windows domain (call it B), mount A''s disk(s) in it, and just restore to them. This has a big advantage over the ''snapshot'' version that it takes up far less space, and is a known and solved situation. The disadvantage of course is that the restore is more complicated (have to prepare the base system to restore into). So I guess you have balance that against your specific requirements - how often do you anticipate requiring a restore? James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
OK, but - if You create the LVM snapshot volume and make the backup of it, You assume the OS running in VM has written everything correctly to the logical volume and the logical volume is in consistent state. But this assumption seems to be dangerous for me, especially by Windows DomU - how can You be sure, the OS hdd driver has written everything to the logical volume before You create the snapshot volume? I would like also to use the LVM snapshots for backup, but this topic is a problem for me, which I could not solve yet... With regards Archie -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Fajar A. Nugraha Sent: Monday, July 02, 2007 11:52 AM To: xen-users Subject: Re: [Xen-users] Backup solution Xin Chen wrote:> but LVM snapshot still has risk, must be survived from a power failture. > correct me if i am wrong... > what I do now is shutdown, copy, start.... >Correct. However, for production system, you''d generally design a system that MUST be able to survive a power failure/hard reboot anyway. This means activating archive log (for Oracle), using Innodb (MySQL), using journaling filesystem, etc. So this shouldn''t be an issue. Reading Jordi''s post I assume his priority was how to keep the VM during backups, so LVM snapshot should be a feasible solution. Regards, Fajar> Fajar A. Nugraha wrote: > >> Jordi Espasa Clofent wrote: >> >> >>> Hi all, >>> >>> We are planning to use XEN VPS in production environment. Before it we >>> need to design a realiable backup policy for obvious reasons. I view >>> two approaches: >>> >>> a) Use a "snapshot" system for backup ALL VM file (image). >>> b) Use a traditional backup system (rsync, tar, cpio) for backup >>> SELECTED data into VM. >>> >>> I prefer A option but I''ve read in this list it is not possible unless >>> you stop de VM during a few time (the time required for backup >>> operation itself). >>> >> >> If your vm image are files, and use LVM snapshot, you don''t need to STOP >> the VM for backup. >> This thread might help you : >> http://lists.xensource.com/archives/html/xen-users/2007-06/msg00688.html >> >> Regards, >> >> Fajar_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users __________ Informace od NOD32 2373 (20070703) __________ Tato zprava byla proverena antivirovym systemem NOD32. http://www.nod32.cz _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Artur Linhart wrote:> OK, but - if You create the LVM snapshot volume and make the backup of it, > You assume the OS running in VM has written everything correctly to the > logical volume and the logical volume is in consistent state. >[snip]> But this assumption seems to be dangerous for me, especially by Windows DomU > - how can You be sure, the OS hdd driver has written everything to the > logical volume before You create the snapshot volume? > >In general, you don''t. However, like I said earlier, for production system you''d generally design a system that MUST be able to survive a power failure/hard reboot anyway. Part of doing this is ensuring that applications ALWAYS flush disk cache (with fsync, etc.) on every write (to transaction logs, datafiles, etc). Oracle AFAIK does this by default, MySQL can be setup to do so. As for "the logical volume is in consistent state", when snapshots are mounted there''s an automated journal replay (if you use journaling filesystem), meaning it should always be in consistent state. I believe it basically comes down to this : if your domU can survive a power failure/hard reboot, it should be able to use LVM snapshots for backup. As others have pointed, this is not the only available backup solution. You should choose one that fits your need best. Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Artur Linhart wrote:> OK, but - if You create the LVM snapshot volume and make the backup of it, > You assume the OS running in VM has written everything correctly to the > logical volume and the logical volume is in consistent state. > But this assumption seems to be dangerous for me, especially by Windows DomU > - how can You be sure, the OS hdd driver has written everything to the > logical volume before You create the snapshot volume? > > I would like also to use the LVM snapshots for backup, but this topic is a > problem for me, which I could not solve yet... > > With regards > > Archie >Windows snapshot backups are a serious, serious problem. Complex software plays enormously complex games to provide this. Frankly, the most reliable is to actually shut down the machine, *then* take an LVM snapshot, then reboot it to keep service active wile using the LVM snapshot for backup. Not pause: not suspend: *halt* the thing. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello, AFAIK Windows writes away cached data after 5 seconds. Otherwise I would recommend running a script calling "sync" inside the guest, then sync in dom0 (is this needed?), then make the LVM snapshot. The only problem is that there might be writes between the finished guest-sync and the not yet performed LVM snapshot. - How do backup software vendors quiesce the guest / VM? What is happening there besides the sync? There is also the sync-tool from Sysinternals (http://www.microsoft.com/technet/sysinternals/FileAndDisk/Sync.mspx). Although this is stated to be written for removable devices, you might check if this also works for regular devices - and please report back here ;) Regards, Volker.> OK, but - if You create the LVM snapshot volume and make the backup of it, > You assume the OS running in VM has written everything correctly to the > logical volume and the logical volume is in consistent state. > But this assumption seems to be dangerous for me, especially by Windows > DomU > - how can You be sure, the OS hdd driver has written everything to the > logical volume before You create the snapshot volume? > > I would like also to use the LVM snapshots for backup, but this topic is a > problem for me, which I could not solve yet... > > With regards > > Archie > > -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Fajar A. > Nugraha > Sent: Monday, July 02, 2007 11:52 AM > To: xen-users > Subject: Re: [Xen-users] Backup solution > > Xin Chen wrote: > > but LVM snapshot still has risk, must be survived from a power > failture. > > correct me if i am wrong... > > what I do now is shutdown, copy, start.... > > > > Correct. > However, for production system, you''d generally design a system that > MUST be able to survive a power failure/hard reboot anyway. This means > activating archive log (for Oracle), using Innodb (MySQL), using > journaling filesystem, etc. So this shouldn''t be an issue. > > Reading Jordi''s post I assume his priority was how to keep the VM during > backups, so LVM snapshot should be a feasible solution. > > Regards, > > Fajar > > > Fajar A. Nugraha wrote: > > > >> Jordi Espasa Clofent wrote: > >> > >> > >>> Hi all, > >>> > >>> We are planning to use XEN VPS in production environment. Before it we > >>> need to design a realiable backup policy for obvious reasons. I view > >>> two approaches: > >>> > >>> a) Use a "snapshot" system for backup ALL VM file (image). > >>> b) Use a traditional backup system (rsync, tar, cpio) for backup > >>> SELECTED data into VM. > >>> > >>> I prefer A option but I''ve read in this list it is not possible unless > >>> you stop de VM during a few time (the time required for backup > >>> operation itself). > >>> > >> > >> If your vm image are files, and use LVM snapshot, you don''t need to > STOP > >> the VM for backup. > >> This thread might help you : > >> > http://lists.xensource.com/archives/html/xen-users/2007-06/msg00688.html > >> > >> Regards, > >> > >> Fajar > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > > __________ Informace od NOD32 2373 (20070703) __________ > > Tato zprava byla proverena antivirovym systemem NOD32. > http://www.nod32.cz > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users-- Psssst! Schon vom neuen GMX MultiMessenger gehört? Der kanns mit allen: http://www.gmx.net/de/go/multimessenger _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello, I§ve thought about it and i think the good compromise seems to be for me 1. start "xm save" to archieve the current and consistent state of the memory 2. during the "xm save" command is running , the domain is paused, I can create the snapshot volume during this time (in script could be waited till the saved memory snapshot file will be created on the disk, if I understand it correctly, at this moment the domain must be already paused, so now the snapshot can be created 3. after the save is executed, the xm restore can be done again (or the command "xm save" can be started with the option -c to leave the domain running after the memory snapshot will be created, but this can be dangerous in the case the "xm save" would take too less time to create the snapshot volume during this time) The virtual machine state will be backed up as the memory snapshot plus the snapshot partition - if I have both of them, I can every time restore by "xm restore" the old state of the virtual machine - and from my point of view, it should be consistent. Or I am wrong? The run of the virtual server is in this case interrupted only for the time the memory snapshot is created - so, this can be something like seconds, or half a minute. I think, at least I, because I deal with no space-shuttle rutime control and management system :-), I can good live with 20 seconds unavailability of the server... With regards Archie -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Nico Kadel-Garcia Sent: Tuesday, July 10, 2007 7:24 PM To: Artur Linhart Cc: ''xen-users''; ''Fajar A. Nugraha'' Subject: Re: [Xen-users] Backup solution Artur Linhart wrote:> OK, but - if You create the LVM snapshot volume and make the backup of it, > You assume the OS running in VM has written everything correctly to the > logical volume and the logical volume is in consistent state. > But this assumption seems to be dangerous for me, especially by WindowsDomU> - how can You be sure, the OS hdd driver has written everything to the > logical volume before You create the snapshot volume? > > I would like also to use the LVM snapshots for backup, but this topic is a > problem for me, which I could not solve yet... > > With regards > > Archie >Windows snapshot backups are a serious, serious problem. Complex software plays enormously complex games to provide this. Frankly, the most reliable is to actually shut down the machine, *then* take an LVM snapshot, then reboot it to keep service active wile using the LVM snapshot for backup. Not pause: not suspend: *halt* the thing. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users __________ Informace od NOD32 2390 (20070710) __________ Tato zprava byla proverena antivirovym systemem NOD32. http://www.nod32.cz _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Artur Linhart - Linux communication wrote:> Hello, > > I§ve thought about it and i think the good compromise seems to be > for me > > 1. start "xm save" to archieve the current and consistent state of the > memory > 2. during the "xm save" command is running , the domain is paused, I can > create the snapshot volume during this time (in script could be waited till > the saved memory snapshot file will be created on the disk, if I understand > it correctly, at this moment the domain must be already paused, so now the > snapshot can be created > 3. after the save is executed, the xm restore can be done again (or the > command "xm save" can be started with the option -c to leave the domain > running after the memory snapshot will be created, but this can be dangerous > in the case the "xm save" would take too less time to create the snapshot > volume during this time) >This is.... optimistic. The saved state of RAM in such a process is potentially "dirty" with cached data not yet written to disc. And atomic operations, such as database operations, are potentially in mid-process because the write to disc is not complete and the database is still locked at the time of the save. And systems like Linux tend to keep as much possible of the paged material in RAM, rather than necessarily writing to disk. And some filesystems, such as ReiserFS, are *particularly* horrid about not completing file-system operations and then attempting to recover from them. With all that added together, I''d consider this reasonable if crunched for downtime, but not reasonable for high reliability of sensitive databases such as Oracle, db4, MySQL, or that mess of twisty little undocumented formats all different called an Outlook userfile. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users