Gluster users - Oct 2016 - Production cluster planning

On 10/26/2016 02:04 PM, Joe Julian wrote:
> On 10/26/2016 02:02 PM, Gandalf Corvotempesta wrote:
>> 2016-10-26 22:59 GMT+02:00 Joe Julian <joe at julianfamily.org>:
>>> Personally, I prefer raid0 bricks just to get the throughput to 
>>> saturate my
>>> network, then I use replicate to meet my availability requirements
>>> (typically replica 3).
>> Isn't the ZFS cache on SSD enough to saturate the network?
>> I'll use replica 3, but I don't feel safe having the whole node
in
>> RAID0, a single disk failure means the whole node to heal from scratch
>> and commodity disks are prone to fail
>
> I just add enough disks to saturate (and I don't like zfs, personally) 
> per-brick. So with 30 disks on a server, I typically do 5-disk raid-0 
> and create 6 bricks per server.
>
And yes, they can fail, but 20TB is small enough to heal pretty quickly.

2016-10-26 23:07 GMT+02:00 Joe Julian <joe at
julianfamily.org>:
> And yes, they can fail, but 20TB is small enough to heal pretty quickly.
20TB small enough to build quickly? On which network? Gluster doesn't
have a dedicated cluster network, if the cluster is being hevily
accessed, the healing will slow down everything else (or everything
else will slow down the healing)

Anyway, you can heal quickly, but I still prefere to have data safe on
each node. If you start with 3 server at once, probably each disk is
coming from the same batch, thus a massive disks failure is easy to
get.
If you loose only 2 disks, one for each server, from the same replica
group, you are game over. With RAID6, you have to loose 5 disks from
the same replica group.

In my environment, I can create 4 RAID-0 on each server (3 disks on
each RAID0), or 2 RAID-6 with 6 disks each, or 1 RAID-6 with 12 disks
or 1 RAID-7 with 12 disks (RAID-7 with less than 12 disks is
non-sense)
I don't know which one is better.