Gluster users - Feb 2012 - question re. current state of art/practice

[pardon if a duplicate - doesn't seem to have gone through the first 
time, earlier today]

Hi Folks,

We've been running a 2-node, high-availability cluster - basically xen
w/ pacemaker and DRBD for replicating disks.  We recently purchased 2
additional,  servers, and I'm thinking about combining all 4 machines
into a 4-node cluster - which takes us out of DRBD space and requires
some other kind of filesystem replication.

Gluster, Ceph, Sheepdog, and XtreemFS seem to keep coming up as things
that might work, but... Sheepdog is too tied to KVM, XtreemFS and Ceph
are still experimental.  Gluster seems to stand out as mature and well
supported (particularly with the RedHat acquisition), and seems to have
matured a lot since we last looked.

For a number of reasons (mostly rack space limits and cost) we have 4 1U
machines - with 4 drives in each - no real opportunity to separate
storage from processing.  The machines each have 4 GigE ports - we're
using 2 for external connections, and reserving 2 for server-to-server
communication.

All of which leads to several questions:

i.  Is it now reasonable to consider running Gluster and Xen on the same
boxes, without hitting too much of a performance penalty?

ii. Anybody out there running a small cluster along these lines?

iii. Any particular suggestions, caveats, guidance, pointers to
documentation, .... (particularly for a Debian environment, though it
wouldn't be that hard to migrate our Dom0s.

iv. Any thoughts re. phase-in and migration - i.e. how to get from 2
production nodes running Xen,DRBD,Pacemaker to 4 nodes running
Gluster/Xen/<pacemaker or some other failover capability>?

Or are we barking up the wrong tree entirely?

Thanks very much,

Miles Fidelman


-- 
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra

On Wed, Feb 15, 2012 at 08:22:18PM -0500, Miles Fidelman wrote:
> i.  Is it now reasonable to consider running Gluster and Xen on the same
> boxes, without hitting too much of a performance penalty?
What's in the hardware? What kind of loads are you expecting it to handle?
For a bunch of lightly-to-moderately-used services, I have two modest
servers, each with 2 8-core AMD CPUs and big-but-slow SAS drives in RAID5,
that are running KVM VMs that happen to be each on dedicated DRBD partitions
mirrored across the servers, while the servers are also providing a couple
of Gluster filesystems primarily mounted from elsewhere by NFS. Which is
certifiably insane. Except it works.

I'm sure I would be in serious trouble if any of the services were high
load, or if the demands on the Gluster file systems were greater -
especially in combination with the DRBD traffic. But it's all been very well
behaved. So there are at least some cases where VMs and Gluster (although
note the VMs are not _on_ Gluster in this case) can share hardware without
coming close to melting it down. A lot of cores helps. A situation where
nothing's seriously pounding any of the file systems helps.

And there's a reason I'm not running the VMs on Gluster (yet). I
experimented with that on an older version, and as noted by others it wasn't
suited for it. The newest version is reputed to be far improved, but is only
in beta. In other words having file systems on Gluster that are mounted by
VMs, which are themselves on the same systems but not themselves stored on
Gluster, given not-too-power-hungry stuff, you may be fine. To put the VMs
themselves on Gluster ... probably better to wait until 3.3 is out of beta.

Whit

On Wed, Feb 15, 2012 at 08:22:18PM -0500, Miles Fidelman
wrote:
> We've been running a 2-node, high-availability cluster - basically xen
> w/ pacemaker and DRBD for replicating disks.  We recently purchased 2
> additional,  servers, and I'm thinking about combining all 4 machines
> into a 4-node cluster - which takes us out of DRBD space and requires
> some other kind of filesystem replication.
> 
> Gluster, Ceph, Sheepdog, and XtreemFS seem to keep coming up as things
> that might work, but... Sheepdog is too tied to KVM
... although if you're considering changing DRBD->Gluster, then changing
Xen->KVM is perhaps worth considering too?

Having VM images as single files in a filesystem arguably allows you to back
them up more easily, and if there are hotspots you can identify them or
shift load around manually.  But I do like the idea of the fully distributed
block device provided by sheepdog.
> i.  Is it now reasonable to consider running Gluster and Xen on the same
> boxes, without hitting too much of a performance penalty?
I have been testing Gluster on 24-disk nodes :
  - 2 HBAs per node (one 16-port and one 8-port)
  - single CPU chip (one node is dual-core i3, one is quad-core Xeon)
  - 8GB RAM
  - 10G ethernet
and however I hit it, the CPU is mostly idle. I think the issue for you is
more likely to be one of latency rather than throughput or CPU utilisation,
and if you have multiple VMs accessing the disk concurrently then latency
becomes less important.

However, I should add that I'm not running VMs on top of this, just doing
filesystem tests (and mostly reads at this stage).

For what gluster 3.3 will bring to the table, see this:
http://community.gluster.org/q/can-i-use-glusterfs-as-an-alternative-network-storage-backing-for-vm-hosting/

Regards,

Brian.