Rudi Ahlers
2010-Oct-14 11:24 UTC
[Xen-users] best practices in using shared storage for XEN Virtual Machines and auto-failover?
Hi all, Can anyone pleas tell me what would be best practice to use shared storage with virtual machines, especially when it involved high availability / automated failover between 2 XEN servers? i.e. if I setup 2x identical XEN servers, each with say 16GB RAM, 4x 1GB NIC''s, etc. Then I need the xen domU''s to auto failover between the 2 servers if either goes down (hardware failure / overload / kernel updates / etc). What is the best way to connect a NAS / SAN to these 2 servers for this kind of setup to work flawlessly? The NAS can export iSCSI, NFS, SMB, etc. I''m sure I could even use ATAOE if needed -- Kind Regards Rudi Ahlers SoftDux Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
<admin@xenhive.com>
2010-Oct-14 12:36 UTC
RE: [Xen-users] best practices in using shared storage for XEN VirtualMachines and auto-failover?
It is easier to use NFS than iSCSI for shared storage target. If you use iSCSI, you will need to use a cluster aware file system to avoid data corruption. If your storage target can do both, just use NFS. -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Rudi Ahlers Sent: Thursday, October 14, 2010 6:25 AM To: xen-users Subject: [Xen-users] best practices in using shared storage for XEN VirtualMachines and auto-failover? Hi all, Can anyone pleas tell me what would be best practice to use shared storage with virtual machines, especially when it involved high availability / automated failover between 2 XEN servers? i.e. if I setup 2x identical XEN servers, each with say 16GB RAM, 4x 1GB NIC''s, etc. Then I need the xen domU''s to auto failover between the 2 servers if either goes down (hardware failure / overload / kernel updates / etc). What is the best way to connect a NAS / SAN to these 2 servers for this kind of setup to work flawlessly? The NAS can export iSCSI, NFS, SMB, etc. I''m sure I could even use ATAOE if needed -- Kind Regards Rudi Ahlers SoftDux Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jeff Sturm
2010-Oct-14 13:42 UTC
RE: [Xen-users] best practices in using shared storage for XEN VirtualMachines and auto-failover?
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com [mailto:xen-users- > bounces@lists.xensource.com] On Behalf Of Rudi Ahlers > Sent: Thursday, October 14, 2010 7:25 AM > To: xen-users > Subject: [Xen-users] best practices in using shared storage for XENVirtualMachines> and auto-failover? > > Hi all, > > Can anyone pleas tell me what would be best practice to use sharedstorage with> virtual machines, especially when it involved high availability /automated failover> between 2 XEN servers?With 2 servers, I hear good things about DRBD, if you don''t want to go the SAN route. If you have a SAN make sure it is sufficiently redundant--i.e. two (or more) power supplies, redundant Ethernet, spare controllers, etc. And of course RAID 10 or similar RAID level to guard against single-drive failure.> i.e. if I setup 2x identical XEN servers, each with say 16GB RAM, 4x1GB NIC''s, etc.> Then I need the xen domU''s to auto failover between the 2 servers ifeither goes down> (hardware failure / overload / kernel updates / etc).Pay close attention to power and networking. With 4 NICs available per host, I''d go for a bonded pair for general network traffic, and a multipath pair for I/O. Use at least two switches. If you get it right you should be able to lose one switch or one power circuit and maintain connectivity to your critical hosts. In my experience with high availability, the #1 mistake I see is overthinking the esoteric failure modes and missing the simple stuff. The #2 mistake is inadequate monitoring to detect single device failures. I''ve seen a lot of mistakes that are simple to correct: - Plugging a bonded Ethernet pair into the same switch. - Connecting dual power supplies to the same PDU. - Oversubscribing a power circuit. When a power supply fails, power draw on the remaining supply will increase--make sure this increase doesn''t overload and trip a breaker. - Ignoring a drive failure until the 2nd drive fails. You can use any of a variety of clustering tools, like heartbeat, to automate the domU failover. Make sure you can''t get into split-brain mode, where a domU can start on two nodes at once--that would quickly corrupt a shared filesystem. With any shared storage configuration, node fencing is generally an essential requirement.> What is the best way to connect a NAS / SAN to these 2 servers forthis kind of setup> to work flawlessly? The NAS can export iSCSI, NFS, SMB, etc. I''m sureI could even> use ATAOE if neededFor my money I''d go with iSCSI (or AoE), partition my block storage and export whole block devices as disk images for the domU guests. If your SAN can''t easily partition your storage, consider a clustered logical volume manager like CLVM on RHCS. -Jeff _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jeff Sturm
2010-Oct-14 13:45 UTC
RE: [Xen-users] best practices in using shared storage for XENVirtualMachines and auto-failover?
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com [mailto:xen-users- > bounces@lists.xensource.com] On Behalf Of admin@xenhive.com > Sent: Thursday, October 14, 2010 8:37 AM > To: ''xen-users'' > Subject: RE: [Xen-users] best practices in using shared storage forXENVirtualMachines> and auto-failover? > > It is easier to use NFS than iSCSI for shared storage target. If youuse iSCSI, you will> need to use a cluster aware file system to avoid data corruption. Ifyour storage> target can do both, just use NFS.You don''t need a clustered filesystem. You don''t need a shared filesystem at all. Just partition your shared block storage (with a clustered logical volume manager if needed) and use a logical volume for each domU image. -Jeff _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Freddie Cash
2010-Oct-14 15:37 UTC
Re: [Xen-users] best practices in using shared storage for XENVirtualMachines and auto-failover?
On Thu, Oct 14, 2010 at 6:45 AM, Jeff Sturm <jeff.sturm@eprize.com> wrote:>> -----Original Message----- >> From: xen-users-bounces@lists.xensource.com [mailto:xen-users- >> bounces@lists.xensource.com] On Behalf Of admin@xenhive.com >> Sent: Thursday, October 14, 2010 8:37 AM >> To: ''xen-users'' >> Subject: RE: [Xen-users] best practices in using shared storage for > XENVirtualMachines >> and auto-failover? >> >> It is easier to use NFS than iSCSI for shared storage target. If you > use iSCSI, you will >> need to use a cluster aware file system to avoid data corruption. If > your storage >> target can do both, just use NFS. > > You don''t need a clustered filesystem. You don''t need a shared > filesystem at all. Just partition your shared block storage (with a > clustered logical volume manager if needed) and use a logical volume for > each domU image.Trying to get this straight in my head. :) Would you run the iSCSI initiator in the host and attach the block device to the guest? Or would you run the iSCSI initiator in the guest? How does the migration from one host machine to another work with iSCSI? -- Freddie Cash fjwcash@gmail.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jeff Sturm
2010-Oct-14 15:59 UTC
RE: [Xen-users] best practices in using shared storage forXENVirtualMachines and auto-failover?
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com [mailto:xen-users- > bounces@lists.xensource.com] On Behalf Of Freddie Cash > Sent: Thursday, October 14, 2010 11:38 AM > To: xen-users > Subject: Re: [Xen-users] best practices in using shared storage forXENVirtualMachines > and auto-failover? > > > You don't need a clustered filesystem. You don't need a shared > > filesystem at all. Just partition your shared block storage (with a > > clustered logical volume manager if needed) and use a logical volume > > for each domU image. > > Would you run the iSCSI initiator in the host and attach the block device to the guest? > Or would you run the iSCSI initiator in the guest?I like to do the former. That makes the guest easier--you don't have to figure out things like modifying initrd to do boot-from-SAN.> How does the migration from one host machine to another work with iSCSI?No different than migration with any other shared storage. Just make sure each dom0 has a working initiator, and is logged into the appropriate targets. If your SAN has ACLs, you'll need to allow access from every possible dom0. -Jeff _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Freddie Cash
2010-Oct-14 16:09 UTC
Re: [Xen-users] best practices in using shared storage forXENVirtualMachines and auto-failover?
On Thu, Oct 14, 2010 at 8:59 AM, Jeff Sturm <jeff.sturm@eprize.com> wrote:>> -----Original Message----- >> From: xen-users-bounces@lists.xensource.com [mailto:xen-users- >> bounces@lists.xensource.com] On Behalf Of Freddie Cash >> Sent: Thursday, October 14, 2010 11:38 AM >> To: xen-users >> Subject: Re: [Xen-users] best practices in using shared storage forXENVirtualMachines >> and auto-failover? >> >> > You don''t need a clustered filesystem. You don''t need a shared >> > filesystem at all. Just partition your shared block storage (with a >> > clustered logical volume manager if needed) and use a logical volume >> > for each domU image. >> >> Would you run the iSCSI initiator in the host and attach the block device to the guest? >> Or would you run the iSCSI initiator in the guest? > > I like to do the former. That makes the guest easier--you don''t have to figure out things like modifying initrd to do boot-from-SAN. > >> How does the migration from one host machine to another work with iSCSI? > > No different than migration with any other shared storage. Just make sure each dom0 has a working initiator, and is logged into the appropriate targets. If your SAN has ACLs, you''ll need to allow access from every possible dom0.So how does the migration work? This is the part that is still tripping me up. :) You have HostA configured with the iSCSI initiator, with VM1 using the block device for storage, running all happily. Then you tell VM1 to migrate to HostB. Does HostB already have the iSCSI initiator running? Does it get started when the migration starts? What happens to the initiator on HostA? I understand how it works with an NFS share and a disk image file. It''s the whole "exported block device on the network" bit that''s messing me up. -- Freddie Cash fjwcash@gmail.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
John Madden
2010-Oct-14 17:04 UTC
Re: [Xen-users] best practices in using shared storage forXENVirtualMachines and auto-failover?
> Then you tell VM1 to migrate to HostB. > > Does HostB already have the iSCSI initiator running? Does it get > started when the migration starts? What happens to the initiator on > HostA?Yes, both dom0''s should be connected to the storage and be able to see it at all times. You have to fence or otherwise "uhm, guarantee" that only one dom0 will be writing to the block devices at any point in time. cLVM won''t do this for you but other components of the cluster services will.> I understand how it works with an NFS share and a disk image file. > It''s the whole "exported block device on the network" bit that''s > messing me up.Try to not distinguish too much between the two in this case. You can''t go writing to a block device from two different hosts at the same time without a cluster filesystem, but even live migration doesn''t allow such writes anyway. John -- John Madden Sr UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
khris4
2010-Oct-14 17:51 UTC
[Xen-users] Re: best practices in using shared storage for XEN Virtual Machines and auto-failover?
iscsi and drbd would be a great way to go. That is the setup i use at my location -- View this message in context: http://xen.1045712.n5.nabble.com/best-practices-in-using-shared-storage-for-XEN-Virtual-Machines-and-auto-failover-tp3211911p3212531.html Sent from the Xen - User mailing list archive at Nabble.com. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Rudi Ahlers
2010-Oct-15 07:39 UTC
Re: [Xen-users] best practices in using shared storage for XENVirtualMachines and auto-failover?
On Thu, Oct 14, 2010 at 5:37 PM, Freddie Cash <fjwcash@gmail.com> wrote:> Trying to get this straight in my head. :) > > Would you run the iSCSI initiator in the host and attach the block > device to the guest? > Or would you run the iSCSI initiator in the guest? > > How does the migration from one host machine to another work with iSCSI? > > -- > Freddie Cash > fjwcash@gmail.com > > _______________________________________________This is actually what I want to know as well. I know the different technologies (AOE / iSCSI / NFS / etc) all have different pro''s & cons, but how do I actually use them optimally? AOE seems to have less network overhead since it runs on the network layer and not on TCP or IP, so this is probably a good option to go for. I''m thinking of setting up a few NAS servers with CentOS + CLVM, or maybe even a few NAS servers with Gluster and then export the file systems from them to the XEN servers. But, do I export it to the dom0, or the domU''s directly? That I don''t know. -- Kind Regards Rudi Ahlers SoftDux Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Rudi Ahlers
2010-Oct-15 09:11 UTC
Re: [Xen-users] best practices in using shared storage for XEN VirtualMachines and auto-failover?
On Thu, Oct 14, 2010 at 3:42 PM, Jeff Sturm <jeff.sturm@eprize.com> wrote:>> -----Original Message----- >> From: xen-users-bounces@lists.xensource.com [mailto:xen-users- >> bounces@lists.xensource.com] On Behalf Of Rudi Ahlers >> Sent: Thursday, October 14, 2010 7:25 AM >> To: xen-users >> Subject: [Xen-users] best practices in using shared storage for XEN > VirtualMachines >> and auto-failover? >> >> Hi all, >> >> Can anyone pleas tell me what would be best practice to use shared > storage with >> virtual machines, especially when it involved high availability / > automated failover >> between 2 XEN servers? > > With 2 servers, I hear good things about DRBD, if you don''t want to go > the SAN route. If you have a SAN make sure it is sufficiently > redundant--i.e. two (or more) power supplies, redundant Ethernet, spare > controllers, etc. And of course RAID 10 or similar RAID level to guard > against single-drive failure.I am planning on setting up a SAN with a few Gluster / CLVM servers - just need to decide which one first, but I''m going to attemp high availability + load balancing + ease-of-upgrade-with-no-downtime. Each server will run RAID10 (maybe RAID6?)> Pay close attention to power and networking. With 4 NICs available per > host, I''d go for a bonded pair for general network traffic, and a > multipath pair for I/O. Use at least two switches. If you get it right > you should be able to lose one switch or one power circuit and maintain > connectivity to your critical hosts.So would you bond eth0 & eth1, and then eth2 & eth3 together? But then connect the bonded eth0+1 one one switch, and eth2+3 on another switch for failover? Or would you have eth0 & eth2 on one switch, and eth1 & eth3 on the other? Is this actually possible? I presume the 2 switches should also be connected together (preferably via fiber?) and then setup Spanning Tree? Or should I seperate the 2 networks,and connect them indivually to the internet?> > In my experience with high availability, the #1 mistake I see is > overthinking the esoteric failure modes and missing the simple stuff. > The #2 mistake is inadequate monitoring to detect single device > failures. I''ve seen a lot of mistakes that are simple to correct: > > - Plugging a bonded Ethernet pair into the same switch. > - Connecting dual power supplies to the same PDU. > - Oversubscribing a power circuit. When a power supply fails, power > draw on the remaining supply will increase--make sure this increase > doesn''t overload and trip a breaker. > - Ignoring a drive failure until the 2nd drive fails. > > You can use any of a variety of clustering tools, like heartbeat, to > automate the domU failover. Make sure you can''t get into split-brain > mode, where a domU can start on two nodes at once--that would quickly > corrupt a shared filesystem. With any shared storage configuration, > node fencing is generally an essential requirement. > >> What is the best way to connect a NAS / SAN to these 2 servers for > this kind of setup >> to work flawlessly? The NAS can export iSCSI, NFS, SMB, etc. I''m sure > I could even >> use ATAOE if needed > > For my money I''d go with iSCSI (or AoE), partition my block storage and > export whole block devices as disk images for the domU guests. If your > SAN can''t easily partition your storage, consider a clustered logical > volume manager like CLVM on RHCS. > > -Jeff >I am considering CLVM, or Gluster - just need to play with them and decide which one I prefer :)> > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >-- Kind Regards Rudi Ahlers SoftDux Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
<admin@xenhive.com>
2010-Oct-15 14:37 UTC
RE: [Xen-users] best practices in using shared storage forXENVirtualMachines and auto-failover?
You would export the NFS so the dom0 can connect to it to store data. The domU would not need to access the NFS share (or iSCSI target or what ever storage you used). As far as the domU are concerned, the targets that the dom0 has mounted would simply appear to be local drives.> _______________________________________________This is actually what I want to know as well. I know the different technologies (AOE / iSCSI / NFS / etc) all have different pro''s & cons, but how do I actually use them optimally? AOE seems to have less network overhead since it runs on the network layer and not on TCP or IP, so this is probably a good option to go for. I''m thinking of setting up a few NAS servers with CentOS + CLVM, or maybe even a few NAS servers with Gluster and then export the file systems from them to the XEN servers. But, do I export it to the dom0, or the domU''s directly? That I don''t know. -- Kind Regards Rudi Ahlers SoftDux Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jeff Sturm
2010-Oct-15 16:25 UTC
RE: [Xen-users] best practices in using shared storage for XEN VirtualMachines and auto-failover?
> -----Original Message----- > From: Rudi Ahlers [mailto:Rudi@SoftDux.com] > Sent: Friday, October 15, 2010 5:11 AM > To: Jeff Sturm > Cc: xen-users > Subject: Re: [Xen-users] best practices in using shared storage for XEN > VirtualMachines and auto-failover? > > On Thu, Oct 14, 2010 at 3:42 PM, Jeff Sturm <jeff.sturm@eprize.com> wrote: > > With 2 servers, I hear good things about DRBD, if you don''t want to go > > the SAN route. If you have a SAN make sure it is sufficiently > > redundant--i.e. two (or more) power supplies, redundant Ethernet, > > spare controllers, etc. And of course RAID 10 or similar RAID level > > to guard against single-drive failure. > > I am planning on setting up a SAN with a few Gluster / CLVM servers - just need to > decide which one first, but I''m going to attemp high availability + load balancing + > ease-of-upgrade-with-no-downtime. Each server will run RAID10 (maybe RAID6?)Sounds like a fun project. Never tried Gluster, myself.> So would you bond eth0 & eth1, and then eth2 & eth3 together? But then connect the > bonded eth0+1 one one switch, and eth2+3 on another switch for failover? Or would > you have eth0 & eth2 on one switch, and eth1 & > eth3 on the other? Is this actually possible?We generally do the latter. It''s certainly possible, with active-backup bonding mode for IP traffic on eth0/1, and multipath for SAN traffic on eth2/3. The switches are not required to cooperate in this mode (unlike active-active or LACP), but should remain connected to one another in case two hosts have their active links on different switches. You''ll see that happen if you unplug a cable on the active link on any host--the backup link will become active for that host only.> I presume the 2 switches should also be > connected together (preferably via fiber?) and then setup Spanning Tree? Or should I > seperate the 2 networks,and connect them indivually to the internet?You wouldn''t use spanning tree with bonding--use one or the other. I''ve had some luck with Linux bridging and spanning tree, but the network delays in learning mode may be intolerable. (Linux bridging has no support for RSTP, yet.) Try it yourself. I''d encourage you to run your own tests by physically unplugging cables and powering down switches before you go into production. -Jeff _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users