I am moving away from NFS due to performance issues and going to iSCSI I have a 4 node environment that I am going to cluster. However, when I share out the iSCSI volume with all 4 servers and mount the volume as /var/vm on the servers they are unable to view the files or folders the other server creates. How can I get around this? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Jan 12, 2010 at 10:25 PM, Joseph Coleman <joe.coleman@infinitecampus.com> wrote:> I am moving away from NFS due to performance issues and going to iSCSI I > have a 4 node environment that I am going to cluster. However, when I share > out the iSCSI volume with all 4 servers and mount the volume as /var/vm on > the servers they are unable to view the files or folders the other server > creates. How can I get around this?What file system are you using? (I hope it is a cluster capable one like GFS or OCFS.) _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Joseph Coleman wrote:>I am moving away from NFS due to performance issues and going to >iSCSI I have a 4 node environment that I am going to cluster. >However, when I share out the iSCSI volume with all 4 servers and >mount the volume as /var/vm on the servers they are unable to view >the files or folders the other server creates. How can I get around >this?Step 1 to fixing this is to realise that iSCSI is **NOT** a shared file system - ie it will **NOT** replace NFS directly. Mounting a volume on more than one system at once without a cluster filesystem is absolutely guaranteed to destroy the filesystem as each system writes what **it** thinks is the correct data on the virtual disk. So you have two choices - mount it on one system and share it (eg via NFS) to the others, or use a cluster filesystem on all the systems that use it. What may also be a related issue you''d want to consider ... I, amongst many, have found performance to be dire when mounting the iSCSI volume in a DomU - it seems the Dom0 networking just don''t work well with it. I don''t think I''m alone in having had to attach the iSCSI volume on Dom0 and share it to the DomU as a virtual block device. -- Simon Hobson WANTED: "Software CD ROM Kit" for Canon CLBP 360-PS printer (Canon part no RH6-3612, or possibly RH6-3810, or RH6-3610 might do). I''ve a dead HD and need this CD so I can replace the disk and re-install the printer OS on it. If anyone knows where I might get hold of one I''d be grateful - requests to Canon drew a blank, it''s been out of support for years. Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> I, amongst many, have found performance to be dire when mounting the > iSCSI volume in a DomU - it seems the Dom0 networking just don''t work > well with it. I don''t think I''m alone in having had to attach the > iSCSI volume on Dom0 and share it to the DomU as a virtual block > device.This is exactly what I do, and the performance difference is significant. I also feel it''s far more manageable than having them in the domUs Best Regards, Nathan Eisenberg _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nathan Eisenberg wrote:> > I, amongst many, have found performance to be dire when mounting the >> iSCSI volume in a DomU - it seems the Dom0 networking just don''t work >> well with it. I don''t think I''m alone in having had to attach the >> iSCSI volume on Dom0 and share it to the DomU as a virtual block > > device.>This is exactly what I do, and the performance difference is >significant. I also feel it''s far more manageable than having them >in the domUsFor me it''s the other way round - it would have been far easier to manage in the DomU. it probably makes a difference how much you use iSCSI, in my case I only need one volume in one guest - and I figured it would be easier to mount it directly and make the guest independent of the host. -- Simon Hobson WANTED: "Software CD ROM Kit" for Canon CLBP 360-PS printer (Canon part no RH6-3612, or possibly RH6-3810, or RH6-3610 might do). I''ve a dead HD and need this CD so I can replace the disk and re-install the printer OS on it. If anyone knows where I might get hold of one I''d be grateful - requests to Canon drew a blank, it''s been out of support for years. Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Jan 12, 2010 at 09:13:14PM +0000, Simon Hobson wrote:> Joseph Coleman wrote: > > >I am moving away from NFS due to performance issues and going to > >iSCSI I have a 4 node environment that I am going to cluster. > >However, when I share out the iSCSI volume with all 4 servers and > >mount the volume as /var/vm on the servers they are unable to view > >the files or folders the other server creates. How can I get around > >this? > > Step 1 to fixing this is to realise that iSCSI is **NOT** a shared > file system - ie it will **NOT** replace NFS directly. Mounting a > volume on more than one system at once without a cluster filesystem > is absolutely guaranteed to destroy the filesystem as each system > writes what **it** thinks is the correct data on the virtual disk. > > So you have two choices - mount it on one system and share it (eg via > NFS) to the others, or use a cluster filesystem on all the systems > that use it. >or use CLVM.. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
You''re almost certainly not using a cluster-aware filesystem if you''re seeing this behavior - if you mount a filesystem on more than one machine at a time you *must* use a cluster-aware filesystem - otherwise you *will* corrupt your filesystem and lose the contents. -Nick>>> On 2010/01/12 at 13:25, Joseph Coleman <joe.coleman@infinitecampus.com> wrote: > I am moving away from NFS due to performance issues and going to iSCSI I have > a 4 node environment that I am going to cluster. However, when I share out > the iSCSI volume with all 4 servers and mount the volume as /var/vm on the > servers they are unable to view the files or folders the other server > creates. How can I get around this?-------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
What would you recommend I use for a filesystem? -----Original Message----- From: Nick Couchman [mailto:Nick.Couchman@seakr.com] Sent: Tuesday, January 12, 2010 8:54 PM To: Joseph Coleman; ''xen-users@lists.xensource.com'' Subject: Re: [Xen-users] iSCSI Volumes and Xen You''re almost certainly not using a cluster-aware filesystem if you''re seeing this behavior - if you mount a filesystem on more than one machine at a time you *must* use a cluster-aware filesystem - otherwise you *will* corrupt your filesystem and lose the contents. -Nick>>> On 2010/01/12 at 13:25, Joseph Coleman <joe.coleman@infinitecampus.com> wrote: > I am moving away from NFS due to performance issues and going to iSCSI I have > a 4 node environment that I am going to cluster. However, when I share out > the iSCSI volume with all 4 servers and mount the volume as /var/vm on the > servers they are unable to view the files or folders the other server > creates. How can I get around this?-------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
GFS and OCFS2 are among the more popular cluster-aware filesystems. RedHat distros support GFS by default; SuSE distros tend toward OCFS. There are a few others, too - a couple more open source and quite a few commercial offerings. Nick Couchman Manager, Information Technology -----Original Message----- From: Joseph Coleman <joe.coleman@infinitecampus.com> To: Nick Couchman <Nick.Couchman@seakr.com> To: xen-users@lists.xensource.com <xen-users@lists.xensource.com> Sent: 1/13/2010 6:56:05 AM Subject: RE: [Xen-users] iSCSI Volumes and Xen What would you recommend I use for a filesystem? -----Original Message----- From: Nick Couchman [mailto:Nick.Couchman@seakr.com] Sent: Tuesday, January 12, 2010 8:54 PM To: Joseph Coleman; ''xen-users@lists.xensource.com'' Subject: Re: [Xen-users] iSCSI Volumes and Xen You''re almost certainly not using a cluster-aware filesystem if you''re seeing this behavior - if you mount a filesystem on more than one machine at a time you *must* use a cluster-aware filesystem - otherwise you *will* corrupt your filesystem and lose the contents. -Nick>>> On 2010/01/12 at 13:25, Joseph Coleman <joe.coleman@infinitecampus.com> wrote: > I am moving away from NFS due to performance issues and going to iSCSI I have > a 4 node environment that I am going to cluster. However, when I share out > the iSCSI volume with all 4 servers and mount the volume as /var/vm on the > servers they are unable to view the files or folders the other server > creates. How can I get around this?-------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. -------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, Jan 13, 2010 at 07:56:05AM -0600, Joseph Coleman wrote:> What would you recommend I use for a filesystem? >And you don''t necessarily need a filesystem at all. You could just use Cluster LVM (CLVM) to store the guest disks. -- Pasi> -----Original Message----- > From: Nick Couchman [mailto:Nick.Couchman@seakr.com] > Sent: Tuesday, January 12, 2010 8:54 PM > To: Joseph Coleman; ''xen-users@lists.xensource.com'' > Subject: Re: [Xen-users] iSCSI Volumes and Xen > > You''re almost certainly not using a cluster-aware filesystem if you''re seeing this behavior - if you mount a filesystem on more than one machine at a time you *must* use a cluster-aware filesystem - otherwise you *will* corrupt your filesystem and lose the contents. > > -Nick > > >>> On 2010/01/12 at 13:25, Joseph Coleman <joe.coleman@infinitecampus.com> wrote: > > I am moving away from NFS due to performance issues and going to iSCSI I have > > a 4 node environment that I am going to cluster. However, when I share out > > the iSCSI volume with all 4 servers and mount the volume as /var/vm on the > > servers they are unable to view the files or folders the other server > > creates. How can I get around this? > > > > -------- > > This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen wrote:>And you don''t necessarily need a filesystem at all. >You could just use Cluster LVM (CLVM) to store the guest disks.I thought that would let you share an LVM volume across machines, but not take care of the filesystem itself. So you still need a cluster aware filesystem to run on top of the CLVM volume. -- Simon Hobson WANTED: "Software CD ROM Kit" for Canon CLBP 360-PS printer (Canon part no RH6-3612, or possibly RH6-3810, or RH6-3610 might do). I''ve a dead HD and need this CD so I can replace the disk and re-install the printer OS on it. If anyone knows where I might get hold of one I''d be grateful - requests to Canon drew a blank, it''s been out of support for years. Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
-----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Simon Hobson Sent: Wednesday, January 13, 2010 11:11 AM To: xen-users@lists.xensource.com Subject: Re: [Xen-users] iSCSI Volumes and Xen> Pasi Kärkkäinen wrote:>>And you don''t necessarily need a filesystem at all. >>You could just use Cluster LVM (CLVM) to store the guest disks.>I thought that would let you share an LVM volume >across machines, but not take care of the >filesystem itself. So you still need a cluster >aware filesystem to run on top of the CLVM volume.CLVM allows you to share the entire volume group, so individual hosts can carve out logical volumes at will for their own storage. Individual logical volumes may or may not be shared within a clustered volume group, depending on your needs. If shared, clustered filesystems are still required. If not, ext3 (or whatever) is fine. -Jeff _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jeff Sturm wrote:>CLVM allows you to share the entire volume group, so individual >hosts can carve out logical volumes at will for their own storage. > >Individual logical volumes may or may not be shared within a >clustered volume group, depending on your needs. If shared, >clustered filesystems are still required. If not, ext3 (or >whatever) is fine.And the OP specifically wanted a shared filesystem - not separate filesystems carved out of a shared storage space. So that brings it back to "he needs a clustered filesystem" - or go back to a network filesystem like NFS which he was using previously. -- Simon Hobson WANTED: "Software CD ROM Kit" for Canon CLBP 360-PS printer (Canon part no RH6-3612, or possibly RH6-3810, or RH6-3610 might do). I''ve a dead HD and need this CD so I can replace the disk and re-install the printer OS on it. If anyone knows where I might get hold of one I''d be grateful - requests to Canon drew a blank, it''s been out of support for years. Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> > On Wed, Jan 13, 2010 at 07:56:05AM -0600, Joseph Coleman wrote: > > What would you recommend I use for a filesystem? > > > > And you don''t necessarily need a filesystem at all. > You could just use Cluster LVM (CLVM) to store the guest disks. >Any word on snapshot support for cLVM? Currently it isn''t supported AFAIK. James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
>>> On 2010/01/13 at 14:19, "James Harper" <james.harper@bendigoit.com.au> wrote: >> >> On Wed, Jan 13, 2010 at 07:56:05AM -0600, Joseph Coleman wrote: >> > What would you recommend I use for a filesystem? >> > >> >> And you don''t necessarily need a filesystem at all. >> You could just use Cluster LVM (CLVM) to store the guest disks. >> > > Any word on snapshot support for cLVM? Currently it isn''t supported AFAIK. > > JamesI believe this functionality is still missing - I''ve done a recent installation of cLVM for Xen purposes and was dismayed to find that snapshots still do not work on shared volumes. -Nick -------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, Jan 13, 2010 at 02:21:27PM -0700, Nick Couchman wrote:> >>> On 2010/01/13 at 14:19, "James Harper" <james.harper@bendigoit.com.au> wrote: > >> > >> On Wed, Jan 13, 2010 at 07:56:05AM -0600, Joseph Coleman wrote: > >> > What would you recommend I use for a filesystem? > >> > > >> > >> And you don''t necessarily need a filesystem at all. > >> You could just use Cluster LVM (CLVM) to store the guest disks. > >> > > > > Any word on snapshot support for cLVM? Currently it isn''t supported AFAIK. > > > > James > > I believe this functionality is still missing - I''ve done a recent installation of cLVM for Xen purposes and was dismayed to find that snapshots still do not work on shared volumes. >Yeah, I think that''s how it is. There are some patches and workarounds for that though.. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I have a CentOS 5.4 dom0 server and would like to move from using locally attached storage (LVM volumes on hardware RAID) to centralised iSCSI storage. I would also like to add further dom0 servers for live migration and better fault tolerance (I really want to be able to maintain my servers without interrupting service). I had planned to use opensolaris or nexenta and create a zvol for each of my domUs and export that as a block device over iSCSI. The network is gigabit and would use aggregated links to give 8gbit of bandwidth between each storage node and the switches and 4gbit between each dom0 and the switches. Reading this thread suggests that exporting each volume as a block device results in poor performance, correct? I had wanted to use zvols for the nice snapshot features of ZFS (it is useful to take snapshots of a running zvol) but not at the cost of a serious performance hit. I need to use pvgrub in my domUs, too. Is this the case and is so is there another way which would perform acceptably and allow the above or a similar? Many thanks, Matt _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, Jan 14, 2010 at 9:07 PM, Matthew Law <matt@webcontracts.co.uk> wrote:> Reading this thread suggests that exporting each volume as a block device > results in poor performance, correct?Where does it say that? It should be just fine if you import the iscsi share on dom0.> I had wanted to use zvols for the > nice snapshot features of ZFS (it is useful to take snapshots of a running > zvol) but not at the cost of a serious performance hit. I need to use > pvgrub in my domUs, too.Actually, if you use opensolaris, the performance hit will come for the forced sync-io on opensolaris side to ensure data consistency. Using SSD as ZIL on opensolaris side should help a lot. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Reading this thread suggests that exporting each volume as a block device > results in poor performance, correct? I had wanted to use zvols for the > nice snapshot features of ZFS (it is useful to take snapshots of a running > zvol) but not at the cost of a serious performance hit. I need to use > pvgrub in my domUs, too.Keep in mind that snapshots are not backups. There''s no way that snapshot would be consistent unless you were also doing something on the domU to stop and flush all i/o and clear the scsi caches. John -- John Madden Sr UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thanks, Fajar. On Thu, January 14, 2010 2:20 pm, Fajar A. Nugraha wrote:> Where does it say that? It should be just fine if you import the iscsi > share on dom0.Sorry, that was my interpretation of the thread. Reading back through it I now realise the OP was talking about mounting the target directly in the domU. I am mounting the target in the dom0. I suppose I might need a script to ensure it is mounted before starting the domU...> Actually, if you use opensolaris, the performance hit will come for > the forced sync-io on opensolaris side to ensure data consistency. > Using SSD as ZIL on opensolaris side should help a lot.Yes, I was aware of this. It will be opensolaris or maybe nexenta if the budget will stretch that far... Thanks again, Matt. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, January 14, 2010 2:22 pm, John Madden wrote:> Keep in mind that snapshots are not backups. There''s no way that > snapshot would be consistent unless you were also doing something on the > domU to stop and flush all i/o and clear the scsi caches.Understood. Does suspending the domU do the job? I wanted to: - ''xm save'' the domU - snapshot the domU''s zvol on the storage server - ''xm restore'' the domU this will give me a consistent snapshot, right? I know this means the domU will be unavailable for the duration of the snapshot. In my experience zfs snapshots are very quick indeed - much quicker than lvm. Is the only backup option for a running domU to run a conventional backup client within it, such as bacula? Thanks, Matt. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, Jan 14, 2010 at 03:28:29PM -0000, Matthew Law wrote:> On Thu, January 14, 2010 2:22 pm, John Madden wrote: > > Keep in mind that snapshots are not backups. There''s no way that > > snapshot would be consistent unless you were also doing something on the > > domU to stop and flush all i/o and clear the scsi caches. > > Understood. Does suspending the domU do the job? I wanted to: > > - ''xm save'' the domU > - snapshot the domU''s zvol on the storage server > - ''xm restore'' the domU > > this will give me a consistent snapshot, right? >No, I don''t think this will give you consistent disk snapshot. Before "xm save" you need to to make sure all the applications (databases etc) are shut down, they have flushed their buffers/caches, the filesystem and kernel have flushed their caches, and THEN you can save the the domU, or take the snapshot directly. -- Pasi> I know this means the domU will be unavailable for the duration of the snapshot. > In my experience zfs snapshots are very quick indeed - much quicker than lvm. > > Is the only backup option for a running domU to run a conventional backup > client within it, such as bacula? > > Thanks, > > Matt. > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Is the only backup option for a running domU to run a conventional backup > client within it, such as bacula?If it were my data, I''d do conventional backups. Others here report success with snapshots, however, so I''ll let them chime in. I''ve never looked into the xm save/restore stuff. John -- John Madden Sr UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, Jan 14, 2010 at 10:28 PM, Matthew Law <matt@webcontracts.co.uk> wrote:> On Thu, January 14, 2010 2:22 pm, John Madden wrote: >> Keep in mind that snapshots are not backups. There''s no way that >> snapshot would be consistent unless you were also doing something on the >> domU to stop and flush all i/o and clear the scsi caches. > > Understood. Does suspending the domU do the job? I wanted to: > > - ''xm save'' the domU > - snapshot the domU''s zvol on the storage server > - ''xm restore'' the domU > > this will give me a consistent snapshot, right? I know this means the > domU will be unavailable for the duration of the snapshot. In my > experience zfs snapshots are very quick indeed - much quicker than lvm.> > Is the only backup option for a running domU to run a conventional backup > client within it, such as bacula?I''d step back a bit and look at what kind of application are you trying to back up. If they''re applications that can survive unclean shutdown or server power outage (like most modern ACID database with journaling), then most likely a simple zfs/lvm snapshot is enough. "xm suspend" or "xm save" is similar to having suspend/hibernate on the server, not needed if you already have zfs/lvm snapshot. In some cases, taking zfs/lvm snapshot can actually be BETTER compared to having backup software on domU. For example: you run MySQL on domU with innodb storage engine, and your backup software can only backup filesystem. In this scenario, if you backup the innodb files directly from domU, and domU doesn''t use LVM, then most likely the backup will be useless as the files will not be from the same time (i.e. some could be changed when the backup process is running). However, if you do this from dom0 side with zfs/lvm snapshot, the files will be consistent enough that MySQL should be able to replay the journal and function normally afterwards. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> If it were my data, I''d do conventional backups.That''s what we do, as well. I have tried filesystem snapshots, and they work, but it''s much harder to document for other SA''s. Everyone understands the concept of traditional backup mechanisms, though, so that''s what I stick with. The exception to this is that I will on occasion take a disk image during a maintenance cycle - that way, I have a good, existing environment I can restore a backup into - or that I can use to create the domU in a test environment and test DR procedures, patches, configurations, etc. My process for that is shutdown, create snapshot, boot domU, use dd to copy the snapshot to a file, delete snapshot. Best Regards, Nathan Eisenberg _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, January 14, 2010 4:55 pm, Fajar A. Nugraha wrote:> I''d step back a bit and look at what kind of application are you > trying to back up. > If they''re applications that can survive unclean shutdown or server > power outage (like most modern ACID database with journaling), then > most likely a simple zfs/lvm snapshot is enough. "xm suspend" or "xm > save" is similar to having suspend/hibernate on the server, not needed > if you already have zfs/lvm snapshot. > > In some cases, taking zfs/lvm snapshot can actually be BETTER compared > to having backup software on domU. > For example: you run MySQL on domU with innodb storage engine, and > your backup software can only backup filesystem. In this scenario, if > you backup the innodb files directly from domU, and domU doesn''t use > LVM, then most likely the backup will be useless as the files will not > be from the same time (i.e. some could be changed when the backup > process is running). However, if you do this from dom0 side with > zfs/lvm snapshot, the files will be consistent enough that MySQL > should be able to replay the journal and function normally afterwards.I don''t have any idea what my users will be running on them. I was hoping that taking a daily snapshot of the ''suspended state'' of the domU and archiving it off the dom0 would be an acceptable ''last resort''. Even if the data is inconsistent, it would be nice to be able to get back to a running, stable domU OS if I have to recover from something catastrophic. If they need quality incremental backups they need to do those themselves with the backup scheme for their apps and data. I would help with storing those off the domU, but I can''t realistically backup the entire thing with great certainty, it would seem. Hope this makes sense and doesn''t sound reckless because that isn''t the intention. Thanks, Matt _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com [mailto:xen-users- > bounces@lists.xensource.com] On Behalf Of John Madden > Sent: Thursday, January 14, 2010 11:07 AM > To: matt@webcontracts.co.uk > Cc: xen-users@lists.xensource.com > Subject: Re: [Xen-users] iSCSI Volumes and Xen > > > Is the only backup option for a running domU to run a conventionalbackup> > client within it, such as bacula? > > If it were my data, I''d do conventional backups. Others here report > success with snapshots, however, so I''ll let them chime in. I''venever> looked into the xm save/restore stuff. >My take: Snapshots aren''t backups, and backups aren''t snapshots. Both are useful but neither completely takes the place of the other. Any storage strategy should be tailored to the circumstances. I''ve been taking frequent (dirty) snapshots on all our development systems for a while--they are cheap and easy, and completely nonintrusive. I''ve never attempted to use one of these snapshots to get a full system restore, but I have used them on occasion when a developer was a little overzealous with "rm -rf" and then asked: "Any chance I can get these file(s) back?" Mounting ext3 snapshots, even when dirty, seems to work fine. Full system backups are best done when a host is down. Any kind of safe online backup is application-dependent--for example, I think MySQL can be configured for synchronous writes of tablespace and binlog data, allowing for point-in-time recovery from a LVM snapshot. But I haven''t tried it, and please don''t take my word for it. Certain filesystems support a "freeze" option, like GFS, that theoretically would help to obtain a consistent snapshot, but that probably only checkpoints buffers and journals, and certainly doesn''t guarantee the state of any files open for writing by applications. In summary, I''ll keep my snapshots, but don''t have any illusion that they are a substitute for regular backups. -Jeff _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
>>> On 2010/01/14 at 09:06, John Madden <jmadden@ivytech.edu> wrote: >> Is the only backup option for a running domU to run a conventional backup >> client within it, such as bacula? > > If it were my data, I''d do conventional backups. Others here report > success with snapshots, however, so I''ll let them chime in. I''ve never > looked into the xm save/restore stuff. > > John >Backups and Snapshots have different purposes and aren''t mutually exclusive. I use both - backups are stored on separate media so that if I lose my entire disk array I have a copy of the data, snapshots allow for quick recovery in the case of an operator error, application bug, guarding against rogue updates, etc. ZFS''s ability to send/receive may mitigate the separate storage issue usually associated with snapshots, as it will allow you to snapshot and then send that data to another array. Also, even with conventional backups, you still need a way to halt transactions on your applications for long enough to be able to get a clean copy of the data. The problem is the same for both snapshots and backups - in both cases, you either need to make sure that the application is capable of recovering from an unclean shutdown or that you''re flushing everything to disk before you suck the data off. -Nick -------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Matthew Law
2010-Jan-14 19:45 UTC
[Xen-users] Backup running domUs (was iSCSI Volumes and Xen)
On Thu, January 14, 2010 6:33 pm, Nick Couchman wrote:> Backups and Snapshots have different purposes and aren''t mutually > exclusive. I use both - backups are stored on separate media so that if I > lose my entire disk array I have a copy of the data, snapshots allow for > quick recovery in the case of an operator error, application bug, guarding > against rogue updates, etc. ZFS''s ability to send/receive may mitigate > the separate storage issue usually associated with snapshots, as it will > allow you to snapshot and then send that data to another array. Also, > even with conventional backups, you still need a way to halt transactions > on your applications for long enough to be able to get a clean copy of the > data. The problem is the same for both snapshots and backups - in both > cases, you either need to make sure that the application is capable of > recovering from an unclean shutdown or that you''re flushing everything to > disk before you suck the data off.How do the so-called ''bare metal recovery'' solutions do it on real running boxes? - I''ve always assumed they had some hook into the kernel to force a flush of the IO or maybe some scripts for installed daemons. I suppose the only way one can guarantee the snapshot is consistent (yes, I know its still only a snapshot) is to actually shutdown the domU properly and snapshot the lvm before restarting it? Interesting issues. This must be something that people run into a lot. Is there something being worked on to do this through the hypervisor somehow? Do the likes of kvm and vmware have the same problem? Thanks, Matt _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nick Couchman
2010-Jan-14 23:13 UTC
[Xen-users] Re: Backup running domUs (was iSCSI Volumes and Xen)
> How do the so-called ''bare metal recovery'' solutions do it on real running > boxes? - I''ve always assumed they had some hook into the kernel to force a > flush of the IO or maybe some scripts for installed daemons.I imagine it depends on the solution - I''m sure some of them have hooks into the kernel to make sure data is flushed to disk before creating a snapshot/image, but I also imagine that some of them assume that journaling and such will take care of most of the data issues.> > I suppose the only way one can guarantee the snapshot is consistent (yes, > I know its still only a snapshot) is to actually shutdown the domU > properly and snapshot the lvm before restarting it?Maybe, I dunno - Microsoft Windows has VSS, which allows applications to register and provides hooks for backup applications and such to interact with the system while insuring data integrity. I''m sure there are other things out there that allow for similar functionality, and it would certainly be cool if Xen could take advantage of the VSS hooks - perhaps in the GPLPV drivers. I''m not trying to put any more work on James, just throwing the idea out there that, at least for Windows machines, VSS integration might actually be a good place to start. -Nick -------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users