Nicolas Sebrecht
2012-Jun-27 08:06 UTC
[Gluster-users] about HA infrastructure for hypervisors
Hi,
We are going to try glusterfs for our new HA servers.
To get full HA, I'm thinking of building it this way:
+----------------+ +----------------+
| | | |
| KVM hypervisor |-----+ +-------| KVM hypervisor |
| | | | | |
+----------------+ | | +----------------+
| |
+------+
|switch|
+------+
| |
+---------------+ | | +---------------+
| | | | | |
| Glusterfs 3.3 |------+ +--------| Glusterfs 3.3 |
| server A | | server B |
| | | |
+---------------+ +---------------+
The idea is to have HA if either one KVM hypervisor or one Glusterfs
server stop working (failure, maintenance, etc).
Some points:
- We don't care much about duplicating the network (we're going to have
spare materials only).
- Glusterfs servers will use gluster replication to get HA.
- Each Glusterfs server will have SSD disks in a RAID (1 or 10, I guess).
- Most of the time, both KVM hypervisor will have VM running.
1. Is this a correct/typicall infrastructure?
2. We still didn't decide what physical network to choose between FC, FCoE
and Infiniband. What would you suggest for both performance and easy
configuration?
Is it possible to use FC or FCoE for a HA Glusterfs cluster? If so, how
to configure Glusterfs nodes?
3. Would it be better to split the Glusterfs namespace into two gluster
volumes (one for each hypervisor), each running on a Glusterfs server
(for the normal case where all servers are running)?
Thanks,
--
Nicolas Sebrecht
Gerald Brandt
2012-Jun-27 11:02 UTC
[Gluster-users] about HA infrastructure for hypervisors
----- Original Message -----> From: "Nicolas Sebrecht" <nsebrecht at piing.fr> > To: "gluster-users" <gluster-users at gluster.org> > Sent: Wednesday, June 27, 2012 3:06:30 AM > Subject: [Gluster-users] about HA infrastructure for hypervisors > > Hi, > > We are going to try glusterfs for our new HA servers. > > To get full HA, I'm thinking of building it this way: > > +----------------+ +----------------+ > | | | | > | KVM hypervisor |-----+ +-------| KVM hypervisor | > | | | | | | > +----------------+ | | +----------------+ > | | > +------+ > |switch| > +------+ > | | > +---------------+ | | +---------------+ > | | | | | | > | Glusterfs 3.3 |------+ +--------| Glusterfs 3.3 | > | server A | | server B | > | | | | > +---------------+ +---------------+ > > > The idea is to have HA if either one KVM hypervisor or one Glusterfs > server stop working (failure, maintenance, etc). > > Some points: > - We don't care much about duplicating the network (we're going to > have > spare materials only). > - Glusterfs servers will use gluster replication to get HA. > - Each Glusterfs server will have SSD disks in a RAID (1 or 10, I > guess). > - Most of the time, both KVM hypervisor will have VM running. > > 1. Is this a correct/typicall infrastructure? > > 2. We still didn't decide what physical network to choose between FC, > FCoE > and Infiniband. What would you suggest for both performance and easy > configuration? > > Is it possible to use FC or FCoE for a HA Glusterfs cluster? If so, > how > to configure Glusterfs nodes? > > 3. Would it be better to split the Glusterfs namespace into two > gluster > volumes (one for each hypervisor), each running on a Glusterfs server > (for the normal case where all servers are running)? > > > Thanks, > > -- > Nicolas SebrechtHi, If your switch breaks, you are done. Put each Gluster server on it's own switch.> +----------------+ +----------------+ > | | | | > | KVM hypervisor |---+ +-------| KVM hypervisor | > | | | | | | > +----------------+ | | +----------------+ > | | > +------+ +------+ > |switch| |switch| > +------+ +------+ > | | | | > +---------------+ | | | | +---------------+ > | | | | | +-----| | > | Glusterfs 3.3 |--+ +------)--------| Glusterfs 3.3 | > | server A | | | server B | > | |-----------+ | | > +---------------+ +---------------+
Brian Candler
2012-Jun-27 20:01 UTC
[Gluster-users] about HA infrastructure for hypervisors
On Wed, Jun 27, 2012 at 10:06:30AM +0200, Nicolas Sebrecht wrote:> We are going to try glusterfs for our new HA servers. > > To get full HA, I'm thinking of building it this way: > > +----------------+ +----------------+ > | | | | > | KVM hypervisor |-----+ +-------| KVM hypervisor | > | | | | | | > +----------------+ | | +----------------+ > | | > +------+ > |switch| > +------+ > | | > +---------------+ | | +---------------+ > | | | | | | > | Glusterfs 3.3 |------+ +--------| Glusterfs 3.3 | > | server A | | server B | > | | | | > +---------------+ +---------------+I've made a test setup like this, but unfortunately I haven't yet been able to get half-decent performance out of glusterfs 3.3 as a KVM backend. It may work better if you use local disk for the VM images, and within the VM mount the glusterfs volume for application data. Alternatively, look at something like ganeti (which by default runs on top of drbd+LVM, although you can also use it to manage a cluster which uses a shared file store backend like gluster) Maybe 3.3.1 will be better. But today, your investment in SSDs is quite likely to be wasted :-(> The idea is to have HA if either one KVM hypervisor or one Glusterfs > server stop working (failure, maintenance, etc).You'd also need some mechanism for starting each VM on node B if node A fails. You can probably script that, although there are lots of hazards for the unwary. Maybe better to have the failover done manually.> 2. We still didn't decide what physical network to choose between FC, FCoE > and Infiniband.Have you ruled out 10G ethernet? If so, why? (note: using SFP+ ports, either with fibre SFP+s or SFP+ coax cables, gives much better latency that 10G over RJ45/CAT6)> 3. Would it be better to split the Glusterfs namespace into two gluster > volumes (one for each hypervisor), each running on a Glusterfs server > (for the normal case where all servers are running)?I don't see how that would help - I expect you would mount both volumes on both KVM nodes anyway, to allow you to do live migration.