Hi All, I''m thinking about the best way to set this up, and would greatly appreciate your learned opinions on it. I''m thinking of the following: - 2 Storage servers, running LVM, heartbeat, drbd (primary/standy) and iscsi. This will protect against 1 storage server failing. - 2 Xen hosts, running heartbeat to ensure the domU''s are available. If not, migrate all hosts on to other xen host. This will protect against 1 xen host failure. Any opinions on this arrangement of setup or links to resources discussing it would be much appreciated. Also any alternative ways to provide the same HA would be useful for comparison. - Any Pitfalls? - Gaps in the availability? (split-brain possibilities?) - How would you add in additional xen hosts? would they always need to be paired in this arrangement (1 fails over to the other..) - Is a clustered filesystem required? Thanks in advance for any advice or opinions on this. Regards, Mark _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thomas Halinka
2010-Nov-04 14:46 UTC
Re: [Xen-users] high availability iscsi and xen setup
Hi Marc, Am Donnerstag, den 04.11.2010, 12:58 +0000 schrieb Mark Adams:> Hi All, > > I''m thinking about the best way to set this up, and would greatly > appreciate your learned opinions on it. I''m thinking of the following: > > - 2 Storage servers, running LVM, heartbeat, drbd (primary/standy) and iscsi. > This will protect against 1 storage server failing.ACK> - 2 Xen hosts, running heartbeat to ensure the domU''s are available. If > not, migrate all hosts on to other xen host. This will protect against > 1 xen host failure.ACK> > Any opinions on this arrangement of setup or links to resources > discussing it would be much appreciated.If ur interested i could provide a link to my wiki, which describes such a setup> Also any alternative ways to > provide the same HA would be useful for comparison. > > - Any Pitfalls?nope - works like a charm> - Gaps in the availability? (split-brain possibilities?)Im running 2 iSCSI-Linux-Targets with a bunch of XEN-Boxes...> > - How would you add in additional xen hosts? would they always need to be > paired in this arrangement (1 fails over to the other..)No need for pairing. Just use hb2 with crm and udev for static device-names...> > - Is a clustered filesystem required?NOT required, but im testing another kind of such a setup. My receipt is to run XEN-Boxes with glusterfs as filebases-Diskbackend... http://www.gluster.com/community/documentation/index.php/GlusterFS_and_Xen first tests were impressive and performance was higher than iscsi, since im running ~60VMs over 10GBit-Nics, where the iscsi-targets were the bottleneck :-(> > Thanks in advance for any advice or opinions on this. > > Regards, > Markhth, thomas> _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Thomas, Thanks for your response. On Thu, Nov 04, 2010 at 03:46:44PM +0100, Thomas Halinka wrote:> Hi Marc, > > Am Donnerstag, den 04.11.2010, 12:58 +0000 schrieb Mark Adams: > > Hi All, > > > > I''m thinking about the best way to set this up, and would greatly > > appreciate your learned opinions on it. I''m thinking of the following: > > > > - 2 Storage servers, running LVM, heartbeat, drbd (primary/standy) and iscsi. > > This will protect against 1 storage server failing. > > ACK > > > - 2 Xen hosts, running heartbeat to ensure the domU''s are available. If > > not, migrate all hosts on to other xen host. This will protect against > > 1 xen host failure. > > ACK > > > > > Any opinions on this arrangement of setup or links to resources > > discussing it would be much appreciated. > > If ur interested i could provide a link to my wiki, which describes such > a setupThat would be excellent, thanks. Do you also do any multipathing so you have network redundancy? or do you deal with this in some other way?> > > Also any alternative ways to > > provide the same HA would be useful for comparison. > > > > - Any Pitfalls? > > nope - works like a charm > > > - Gaps in the availability? (split-brain possibilities?) > > Im running 2 iSCSI-Linux-Targets with a bunch of XEN-Boxes... > > > > > - How would you add in additional xen hosts? would they always need to be > > paired in this arrangement (1 fails over to the other..) > > No need for pairing. Just use hb2 with crm and udev for static > device-names... > > > > - Is a clustered filesystem required? > > NOT required, but im testing another kind of such a setup. My receipt is > to run XEN-Boxes with glusterfs as filebases-Diskbackend... > > http://www.gluster.com/community/documentation/index.php/GlusterFS_and_Xen > > first tests were impressive and performance was higher than iscsi, since > im running ~60VMs over 10GBit-Nics, where the iscsi-targets were the > bottleneck :-( > > > > > Thanks in advance for any advice or opinions on this. > > > > Regards, > > Mark > > hth, > > thomas > >Cheers _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thomas Halinka
2010-Nov-04 16:11 UTC
Re: [Xen-users] high availability iscsi and xen setup
Hi Mark, Am Donnerstag, den 04.11.2010, 15:45 +0000 schrieb Mark Adams:> Hi Thomas, Thanks for your response. > > On Thu, Nov 04, 2010 at 03:46:44PM +0100, Thomas Halinka wrote: > > Hi Marc, > > > > Am Donnerstag, den 04.11.2010, 12:58 +0000 schrieb Mark Adams: > > > Hi All, > > > > > > I''m thinking about the best way to set this up, and would greatly > > > appreciate your learned opinions on it. I''m thinking of the following: > > > > > > - 2 Storage servers, running LVM, heartbeat, drbd (primary/standy) and iscsi. > > > This will protect against 1 storage server failing. > > > > ACK > > > > > - 2 Xen hosts, running heartbeat to ensure the domU''s are available. If > > > not, migrate all hosts on to other xen host. This will protect against > > > 1 xen host failure. > > > > ACK > > > > > > > > Any opinions on this arrangement of setup or links to resources > > > discussing it would be much appreciated. > > > > If ur interested i could provide a link to my wiki, which describes such > > a setup > > That would be excellent, thanks. Do you also do any multipathing so you > have network redundancy? or do you deal with this in some other way?in my tests multipath bonding and multipath had similar read-Performance, but multipath had much faster writes than a 802.3ad-Trunk so i just went with mulipathing... cu, thomas> > > > > > Also any alternative ways to > > > provide the same HA would be useful for comparison. > > > > > > - Any Pitfalls? > > > > nope - works like a charm > > > > > - Gaps in the availability? (split-brain possibilities?) > > > > Im running 2 iSCSI-Linux-Targets with a bunch of XEN-Boxes... > > > > > > > > - How would you add in additional xen hosts? would they always need to be > > > paired in this arrangement (1 fails over to the other..) > > > > No need for pairing. Just use hb2 with crm and udev for static > > device-names... > > > > > > - Is a clustered filesystem required? > > > > NOT required, but im testing another kind of such a setup. My receipt is > > to run XEN-Boxes with glusterfs as filebases-Diskbackend... > > > > http://www.gluster.com/community/documentation/index.php/GlusterFS_and_Xen > > > > first tests were impressive and performance was higher than iscsi, since > > im running ~60VMs over 10GBit-Nics, where the iscsi-targets were the > > bottleneck :-( > > > > > > > > Thanks in advance for any advice or opinions on this. > > > > > > Regards, > > > Mark > > > > hth, > > > > thomas > > > > > Cheers > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, Nov 04, 2010 at 05:11:34PM +0100, Thomas Halinka wrote:> Hi Mark, > > Am Donnerstag, den 04.11.2010, 15:45 +0000 schrieb Mark Adams: > > Hi Thomas, Thanks for your response. > > > > On Thu, Nov 04, 2010 at 03:46:44PM +0100, Thomas Halinka wrote: > > > Hi Marc, > > > > > > Am Donnerstag, den 04.11.2010, 12:58 +0000 schrieb Mark Adams: > > > > Hi All, > > > > > > > > I''m thinking about the best way to set this up, and would greatly > > > > appreciate your learned opinions on it. I''m thinking of the following: > > > > > > > > - 2 Storage servers, running LVM, heartbeat, drbd (primary/standy) and iscsi. > > > > This will protect against 1 storage server failing. > > > > > > ACK > > > > > > > - 2 Xen hosts, running heartbeat to ensure the domU''s are available. If > > > > not, migrate all hosts on to other xen host. This will protect against > > > > 1 xen host failure. > > > > > > ACK > > > > > > > > > > > Any opinions on this arrangement of setup or links to resources > > > > discussing it would be much appreciated. > > > > > > If ur interested i could provide a link to my wiki, which describes such > > > a setup > > > > That would be excellent, thanks. Do you also do any multipathing so you > > have network redundancy? or do you deal with this in some other way? > > in my tests multipath bonding and multipath had similar > read-Performance, but multipath had much faster writes than a > 802.3ad-Trunk so i just went with mulipathing...Hi Thomas - Thanks again. Can you provide me with the link to your wiki? Regards, Mark> > > cu, > > thomas > > > > > > > > > > Also any alternative ways to > > > > provide the same HA would be useful for comparison. > > > > > > > > - Any Pitfalls? > > > > > > nope - works like a charm > > > > > > > - Gaps in the availability? (split-brain possibilities?) > > > > > > Im running 2 iSCSI-Linux-Targets with a bunch of XEN-Boxes... > > > > > > > > > > > - How would you add in additional xen hosts? would they always need to be > > > > paired in this arrangement (1 fails over to the other..) > > > > > > No need for pairing. Just use hb2 with crm and udev for static > > > device-names... > > > > > > > > - Is a clustered filesystem required? > > > > > > NOT required, but im testing another kind of such a setup. My receipt is > > > to run XEN-Boxes with glusterfs as filebases-Diskbackend... > > > > > > http://www.gluster.com/community/documentation/index.php/GlusterFS_and_Xen > > > > > > first tests were impressive and performance was higher than iscsi, since > > > im running ~60VMs over 10GBit-Nics, where the iscsi-targets were the > > > bottleneck :-( > > > > > > > > > > > Thanks in advance for any advice or opinions on this. > > > > > > > > Regards, > > > > Mark > > > > > > hth, > > > > > > thomas > > > > > > > > Cheers > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, Nov 4, 2010 at 4:46 PM, Thomas Halinka <lists@thohal.de> wrote:> If ur interested i could provide a link to my wiki, which describes such > a setup >Would you mind sharing this info with me as well please? I need to setup something like this as well.> > Im running 2 iSCSI-Linux-Targets with a bunch of XEN-Boxes... >What kind of hardware, and networking (i.e. 1GB / 10GB / infinband?) are you using? Are you using normal server hardware for the iSCSI server, or do you use commercial NAS appliances ?> > NOT required, but im testing another kind of such a setup. My receipt is > to run XEN-Boxes with glusterfs as filebases-Diskbackend... > > http://www.gluster.com/community/documentation/index.php/GlusterFS_and_Xen > > first tests were impressive and performance was higher than iscsi, since > im running ~60VMs over 10GBit-Nics, where the iscsi-targets were the > bottleneck :-( >I''m looking at using GlusterFS as well, but understand that it doesn''t perform well with plenty of small files (which is typical in web hosting), but rather with fewer larger files. What kind of data do you have? And how redundant is your setup? As matter of interest, what kind of uptime have you been able to achieve with your setup, as a whole? I mean, if you need to upgrade the kernel of the servers, then surely you can do them one-at-a-time and reboot each server in sequence, but the whole project should still be 100% available? Let''s say the network portion is 100% redundant, and you have multiple redundant UPS + generators. What kind of uptime could you get with your setup? -- Kind Regards Rudi Ahlers SoftDux Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Hi All, > > I''m thinking about the best way to set this up, and would greatly > appreciate your learned opinions on it. I''m thinking of the following: > > - 2 Storage servers, running LVM, heartbeat, drbd (primary/standy) andiscsi.> This will protect against 1 storage server failing. > - 2 Xen hosts, running heartbeat to ensure the domU''s are available.If> not, migrate all hosts on to other xen host. This will protectagainst> 1 xen host failure. > > Any opinions on this arrangement of setup or links to resources > discussing it would be much appreciated. Also any alternative ways to > provide the same HA would be useful for comparison.That''s the way I''d do it, subject to some performance testing.> > - Any Pitfalls? > > - Gaps in the availability? (split-brain possibilities?) >I''d have 3 networks: 1. drbd network. 2 gigabit (or 1 10G) ports bonded on each storage server connected directly to the other - no switch. 2. storage network. 2 gigabit (or 1 10G) ports bonded on each xen server and storage server, connected into a switch with some decent HA features (redundant PSU''s probably), or stacked switches. 3. regular networks for normal DomU traffic So the DRBD servers would need 4 gigabit ports, bonded in pairs. I assume from what you specified that you are running 2 completely separate instances of HA - one to manage DRBD and iSCSI failover and the other to manage Xen. Split brain problems are the worst problems you''ll face. If storage server #1 goes down at 9pm (power failure and this machine is on a circuit with a dud UPS) and everything fails over to #2, then everything goes down at 10pm (someone forgot to fill up the generators or the UPS runs out of battery or something), power comes back at 11pm but #2 fails to boot and everything starts using storage server #1 which is out of date. You''d have to be having a really bad day for that to happen though. James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> > - Is a clustered filesystem required? > > NOT required, but im testing another kind of such a setup. My receipt is > to run XEN-Boxes with glusterfs as filebases-Diskbackend... > > http://www.gluster.com/community/documentation/index.php/GlusterFS_and_Xen > > first tests were impressive and performance was higher than iscsi, since > im running ~60VMs over 10GBit-Nics, where the iscsi-targets were the > bottleneck :-( >That's the way the Microsoft literature recommends for Hyper-V on a SAN. They use VHD files on a shared filesystem. I guess because you can lock the entire VHD file you mitigate a lot of the performance problems of a shared filesystem as each host is using its own chunk of the filesystem holding its VHD file. I assume that 'dynamic' VHD's that grow on demand take a bit of a performance hit when they need to grow though. I'm impressed that the performance is actually higher that raw iSCSI. Do you have an explanation for that? How is each host accessing the shared storage? James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi James, On Fri, Nov 05, 2010 at 10:09:12AM +1100, James Harper wrote:> > Hi All, > > > > I''m thinking about the best way to set this up, and would greatly > > appreciate your learned opinions on it. I''m thinking of the following: > > > > - 2 Storage servers, running LVM, heartbeat, drbd (primary/standy) and > iscsi. > > This will protect against 1 storage server failing. > > - 2 Xen hosts, running heartbeat to ensure the domU''s are available. > If > > not, migrate all hosts on to other xen host. This will protect > against > > 1 xen host failure. > > > > Any opinions on this arrangement of setup or links to resources > > discussing it would be much appreciated. Also any alternative ways to > > provide the same HA would be useful for comparison. > > That''s the way I''d do it, subject to some performance testing. > > > > > - Any Pitfalls? > > > > - Gaps in the availability? (split-brain possibilities?) > > > > I''d have 3 networks: > > 1. drbd network. 2 gigabit (or 1 10G) ports bonded on each storage > server connected directly to the other - no switch. > 2. storage network. 2 gigabit (or 1 10G) ports bonded on each xen server > and storage server, connected into a switch with some decent HA features > (redundant PSU''s probably), or stacked switches. > 3. regular networks for normal DomU trafficI was originally planning on having 8 gigabit ports on each DRBD (storage) server, going in to 2 seperate switches using multipathing. each xen host would also have 2 gigabit links in to each switch also using multipathing. This way I would be covered for switch failure also, any thoughts on that? This would mean running DRBD and "SAN" traffic on the same network, is this not advisable? alternatively direct connected 10G CX4 port for DRBD traffic and 4 ports (2 in to each switch with multipathing) would also be good I guess? If anyone has any links or further info about xen ha with multipathing that would be very helpful, or other suggestions for having auto failover for a switch failure scenario..> > So the DRBD servers would need 4 gigabit ports, bonded in pairs. > > I assume from what you specified that you are running 2 completely > separate instances of HA - one to manage DRBD and iSCSI failover and the > other to manage Xen.Yes that''s right.> > Split brain problems are the worst problems you''ll face. If storage > server #1 goes down at 9pm (power failure and this machine is on a > circuit with a dud UPS) and everything fails over to #2, then everything > goes down at 10pm (someone forgot to fill up the generators or the UPS > runs out of battery or something), power comes back at 11pm but #2 fails > to boot and everything starts using storage server #1 which is out of > date. You''d have to be having a really bad day for that to happen > though.I think that''s an acceptable risk, there''s not much you can really do to mitigate it? I guess stonith to make sure #1 doesnt come back, your left with nothing but atleast not the "bad" data?> > JamesCheers! Mark _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> > I was originally planning on having 8 gigabit ports on each DRBD > (storage) server, going in to 2 seperate switches using multipathing. > each xen host would also have 2 gigabit links in to each switch also > using multipathing.I guess it depends on your disk bandwidth. If you have >200Mbytes/second disk throughput then you''ll need to add more network ports to prevent it becoming a bottleneck. I''d definitely be plumbing the DRBD network connections through directly though. At least if you have a switch failure you won''t get split brain.> This way I would be covered for switch failure also, any thoughts on > that? This would mean running DRBD and "SAN" traffic on the same > network, is this not advisable? > > alternatively direct connected 10G CX4 port for DRBD traffic and 4ports> (2 in to each switch with multipathing) would also be good I guess?Every write goes to the primary DRBD server which then sends it to the secondary DRBD server. I figure that keeping them on separate networks is a good idea.> > > > Split brain problems are the worst problems you''ll face. If storage > > server #1 goes down at 9pm (power failure and this machine is on a > > circuit with a dud UPS) and everything fails over to #2, theneverything> > goes down at 10pm (someone forgot to fill up the generators or theUPS> > runs out of battery or something), power comes back at 11pm but #2fails> > to boot and everything starts using storage server #1 which is outof> > date. You''d have to be having a really bad day for that to happen > > though. > > I think that''s an acceptable risk, there''s not much you can really doto> mitigate it? I guess stonith to make sure #1 doesnt come back, yourleft> with nothing but atleast not the "bad" data?When #1 comes back up DRBD will activate it''s split brain policy but you can set that to be anything you want. Data loss is inevitable though as both #1 and #2 contain data that doesn''t exist on the other. But you''re right, it''s pretty much impossible to do anything about it - #1 would need to know that #2 was primary for a while while #1 was down. This might be known by the xen servers so maybe you could do something there but what? You''d basically need to keep the whole system down until someone came in and fixed #2, so I guess you need to chose between downtime or data loss. One thing you could do maybe is stack DRBD on each xen server for the active LV. This makes migrations expensive though (need to sync up the DRBD on the new xen node) and would be really fiddly to implement for what is a very rare possibility James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Nov 05, 2010 at 10:54:20AM +1100, James Harper wrote:> > > > I was originally planning on having 8 gigabit ports on each DRBD > > (storage) server, going in to 2 seperate switches using multipathing. > > each xen host would also have 2 gigabit links in to each switch also > > using multipathing. > > I guess it depends on your disk bandwidth. If you have >200Mbytes/second > disk throughput then you''ll need to add more network ports to prevent it > becoming a bottleneck. I''d definitely be plumbing the DRBD network > connections through directly though. At least if you have a switch > failure you won''t get split brain. > > > This way I would be covered for switch failure also, any thoughts on > > that? This would mean running DRBD and "SAN" traffic on the same > > network, is this not advisable? > > > > alternatively direct connected 10G CX4 port for DRBD traffic and 4 > ports > > (2 in to each switch with multipathing) would also be good I guess? > > Every write goes to the primary DRBD server which then sends it to the > secondary DRBD server. I figure that keeping them on separate networks > is a good idea.Thanks for your replies James. With regards to network redundancy, what do to ensure a switch failure doesn''t affect your uptime? Do you have any opinion on including multipathing in the setup? Cheers, Mark _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> > > > > Any opinions on this arrangement of setup or links to resources > > discussing it would be much appreciated. > > If ur interested i could provide a link to my wiki, which describes such > a setup >Hi Thomas - It would be great if you could post the link to your wiki about this. Cheers, Mark _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thursday 04 November 2010 17:11:34 Thomas Halinka wrote:> Hi Mark, > > Am Donnerstag, den 04.11.2010, 15:45 +0000 schrieb Mark Adams: > > Hi Thomas, Thanks for your response. > > > > On Thu, Nov 04, 2010 at 03:46:44PM +0100, Thomas Halinka wrote: > > > Hi Marc, > > > > > > Am Donnerstag, den 04.11.2010, 12:58 +0000 schrieb Mark Adams: > > > > Hi All, > > > > > > > > I''m thinking about the best way to set this up, and would greatly > > > > appreciate your learned opinions on it. I''m thinking of the > > > > following: > > > > > > > > - 2 Storage servers, running LVM, heartbeat, drbd (primary/standy) > > > > and iscsi. > > > > > > > > This will protect against 1 storage server failing. > > > > > > ACK > > > > > > > - 2 Xen hosts, running heartbeat to ensure the domU''s are available. > > > > If > > > > > > > > not, migrate all hosts on to other xen host. This will protect > > > > against 1 xen host failure. > > > > > > ACK > > > > > > > Any opinions on this arrangement of setup or links to resources > > > > discussing it would be much appreciated. > > > > > > If ur interested i could provide a link to my wiki, which describes > > > such a setup > > > > That would be excellent, thanks. Do you also do any multipathing so you > > have network redundancy? or do you deal with this in some other way? > > in my tests multipath bonding and multipath had similar > read-Performance, but multipath had much faster writes than a > 802.3ad-Trunk so i just went with mulipathing...Am using a similar setup as well, also with multipathing. Multipathing does not seem to add a spectacular speed gain, though my guess is that could be helped by adding bonding to both paths. Challenges are to get the performance high enough for all VMs you want to run. If having spent enough on RAID hardware and disks, I find the network part for the storage the biggest botlleneck. iSCSI sucks in performance. AoE is in that regard probably a better choice but I shied away from it sine being less industry-standard. B. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, Nov 04, 2010 at 12:58:30PM +0000, Mark Adams wrote:> - Gaps in the availability? (split-brain possibilities?)If one machine fails, you have to reliably kill it. This is usually done with some sort of remote controlled power supply. If you completely loose all communication channels between the two machines, this is not longer possible. If you ignore that, you''ll get split-brain in this case. The only way do avoid split-brain is a quorum decision with an uneven number of votes.> - Is a clustered filesystem required?No.> Thanks in advance for any advice or opinions on this.You don''t want to go that way, at least not with automatic failover. Bastian -- Too much of anything, even love, isn''t necessarily a good thing. -- Kirk, "The Trouble with Tribbles", stardate 4525.6 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users