Gluster users - Sep 2020 - Fwd: New GlusterFS deployment, doubts on 1 brick per host vs 1 brick per drive.

Hello all,

I'm setting up GlusterFS on 2 hw w/ same configuration, 8  hdds. This
deployment will grow later on.

I'm undecided between these different configurations and am seeing comments
or advice from more experienced users of GlusterFS.

Here is the summary of 3 options:
1. 1 brick per host, Gluster "distributed" volumes, internal
redundancy at
brick level
2. 1 brick per drive, Gluster "distributed replicated" volumes, no
internal
redundancy
3. 1 brick per host, Gluster "distributed replicated" volumes, no
internal
redundancy

I don't know how the performance of these different configurations would
compare. The workload is HPC and ML-training, data-heavy, metadata-light.


# 1 brick per host, simplified cluster management, higher blast-radius

having 1 brick per host (/data/bricks/hdd0) where each brick is a ZFS
raid10 of 8 hdd.
Pros:
* I know ZFS raid10 performs very well.
* simpler management of Gluster at the Host-brick level.
* using Gluster in "distributed" mode, no replication (is this a pro?)
* don't need to worry about GlusterFS performance with "distributed
replicated"
* future hosts can have raidz2 + hot spares or whatnot
Cons:
* large blast radius, if a zfs volume goes bad or node goes bad, I lose
data.
* not using "distributed replicated" (is this a con?)
* I can't use hosts without internal redundancy later on?


# 1 brick per hard disk, fine grained device management on Gluster, smaller
blast-radius.

Having 1 brick per drive (/data/bricks/hddN for 1 to X drives on box), each
brick would still use ZFS.
Pros:
* 1 drive blast radius, the ideal.
* GlusterFS w/ distributed replicated
* no complicated host-fault management or runbook, I can use hosts with low
availability
Cons:
* distributed replicated performance vs zfs raid10
* managing on gluster at the disk level can be more time consuming
* managing disk spares and replacements w/ gluster


# 1 brick per host, coarse grained brick management on Gluster, Gluster for
replication.

having 1 brick per host (/data/bricks/hdd0) where each brick is a ZFS raid0
of 8 hdd.

Pros:
* I know ZFS raid0 performs very well.
* simpler management of Gluster at the Host-brick level.
* using Gluster for redundancy (is this a pro?)
* system can use lower redundancy hosts
Cons:
* large blast radius, replicating a whole host takes longer.

Any comments or advice is welcome


Thanks in advance,

-- Miguel Filipe
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200909/225c99b3/attachment.html>

Il 09/09/20 15:30, Miguel Mascarenhas Filipe ha scritto:

I'm a noob, but IIUC this is the option giving the best performance:
> 2. 1 brick per drive, Gluster "distributed replicated" volumes,
no
> internal redundancy
Clients can write to both servers in parallel and read scattered (read
performance using multiple files ~ 16x vs 2x with a single disk per
host). Moreover it's easier to extend.
But why ZFS instead of XFS ? In my experience it's heavier.

PS: add a third host ASAP, at least for arbiter volumes (replica 3
arbiter 1). Split brain can be a real pain to fix!

-- 
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Universit? di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786

Il 2020-09-09 15:30 Miguel Mascarenhas Filipe ha
scritto:
> I'm setting up GlusterFS on 2 hw w/ same configuration, 8  hdds. This
> deployment will grow later on.
Hi, I really suggest avoiding a replica 2 cluster unless it is for 
testing only. Be sure to add an arbiter at least (using a replica 2 
arbiter 1 cluster).
> I'm undecided between these different configurations and am seeing
> comments or advice from more experienced users of GlusterFS.
> 
> Here is the summary of 3 options:
> 1. 1 brick per host, Gluster "distributed" volumes, internal
> redundancy at brick level
I strongly suggest against it: any server reboot will cause troubles for 
mounted clients. I will end with *lower* uptime than a single server.
> 2. 1 brick per drive, Gluster "distributed replicated" volumes,
no
> internal redundancy
This would increase Gluster performance via multiple bricks; however a 
single failed disk will put the entire note out-of-service. Moreover, 
Gluster heals are much slower processes than a simple RAID1/ZFS mirror 
recover.
> 3. 1 brick per host, Gluster "distributed replicated" volumes, no
> internal redundancy
Again, a suggest against it: a single failed disk will put the entire 
note out-of-service *and* will cause massive heal as all data need to be 
copied from the surviving node, which is a long and stressful event for 
the other node (and for the sysadmin).

In short, I would not use Gluster without *both* internal and 
brick-level redundancy. For a simple setup, I suggest option #1 but in 
replica setup (rather than distributed). You can increase the number of 
briks (mountpoint) via multiple zfs datasets, if needed.

Regards.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti at assyoma.it - info at assyoma.it
GPG public key ID: FF5F32A8