Gluster users - Feb 2020 - Gluster setup for virtualization cluster

Greetings!

I am currently evaluating our options to replace our old mixture of IBM 
SAN storage boxes. This will be a strategic decision for the next years.
One of the solutions I am reviewing is a GlusterFS installation.

Planned usage:
- Central NFS server for around 25 systems providing around 400 docker 
containers
- Central storage for a small VMWare vCenter cluster and a RedHat 
virtualization cluster. In total maybe around 15 machines

The following requirements ensue from this:
- Fast storage
- High availability


After reading all kind of tutorials and documentation, I came to the 
conclusion that for the expected traffic a "Distributed Replicate 
Volume" is the proper setup.

Nothing has been purchased but I think about following small setup for 
the beginning (call it PoC):

4 x server, each with 8 x 1.8TB 10k SAS disks in a RAID60
Two 10 GBit interfaces per server: One for communication betweens the 4 
systems only (separate VLAN), the other one for regular traffic between 
clients and servers.


Does this all make sense?
Generally speaking: Is such a setup capable of providing fast enough 
storage for a virtualization cluster?
Do you have any hints?

Thanks

Markus

On February 17, 2020 4:59:17 AM GMT+02:00, Markus Kern <gluster at
military.de> wrote:
>Greetings!
>
>I am currently evaluating our options to replace our old mixture of IBM
>
>SAN storage boxes. This will be a strategic decision for the next
>years.
>One of the solutions I am reviewing is a GlusterFS installation.
>
>Planned usage:
>- Central NFS server for around 25 systems providing around 400 docker 
>containers
>- Central storage for a small VMWare vCenter cluster and a RedHat 
>virtualization cluster. In total maybe around 15 machines
>
>The following requirements ensue from this:
>- Fast storage
>- High availability
>
>
>After reading all kind of tutorials and documentation, I came to the 
>conclusion that for the expected traffic a "Distributed Replicate 
>Volume" is the proper setup.
>
>Nothing has been purchased but I think about following small setup for 
>the beginning (call it PoC):
>
>4 x server, each with 8 x 1.8TB 10k SAS disks in a RAID60
>Two 10 GBit interfaces per server: One for communication betweens the 4
>
>systems only (separate VLAN), the other one for regular traffic between
>
>clients and servers.
>
>
>Does this all make sense?
>Generally speaking: Is such a setup capable of providing fast enough 
>storage for a virtualization cluster?
>Do you have any hints?
>
>Thanks
>
>Markus
>
>
>
>________
>
>Community Meeting Calendar:
>
>APAC Schedule -
>Every 2nd and 4th Tuesday at 11:30 AM IST
>Bridge: https://bluejeans.com/441850968
>
>NA/EMEA Schedule -
>Every 1st and 3rd Tuesday at 01:00 PM EDT
>Bridge: https://bluejeans.com/441850968
>
>Gluster-users mailing list
>Gluster-users at gluster.org
>https://lists.gluster.org/mailman/listinfo/gluster-users
Hi Markus,

It all depends on the IOPS requirements and the capabilities of the disks. Are
these  spinning or SSD/NVME ?

For me a Raid60  is an overkill, as  you need 'replica  2 arbiter  1' or
pure 'replica 3' when you care about your data.

I'd recommend you to add a separate node (either physical or Virtual)  as an
Arbiter.

Most probably a raid5/50  will be enough, but you can adapt in the POC stage.

If  you plan to use  NFS - consider NFS Ganesha in a HA cluster,  as  the
built-in NFS is deprecated (and in some  distributions you need to rebuild from
source).Also consider it as a 'gateway' which means that the NFS Ganesha
should have more NICs.

If NFS Ganesha is used - you won't need the second group of NICs on the
Gluster nodes.  Ganesha directly speeks with all nodes in the storage pool.

So, you can try with 3  Gluster nodes  and a node for NFS Ganesha and later
scale-out.


Best Regards,
Strahil Nikolov

Il 2020-02-17 03:59 Markus Kern ha scritto:
> Greetings!
> 
> I am currently evaluating our options to replace our old mixture of
> IBM SAN storage boxes. This will be a strategic decision for the next
> years.
> One of the solutions I am reviewing is a GlusterFS installation.
> 
> Planned usage:
> - Central NFS server for around 25 systems providing around 400 docker
> containers
> - Central storage for a small VMWare vCenter cluster and a RedHat
> virtualization cluster. In total maybe around 15 machines
> 
> The following requirements ensue from this:
> - Fast storage
> - High availability
> 
> 
> After reading all kind of tutorials and documentation, I came to the
> conclusion that for the expected traffic a "Distributed Replicate
> Volume" is the proper setup.
> 
> Nothing has been purchased but I think about following small setup for
> the beginning (call it PoC):
> 
> 4 x server, each with 8 x 1.8TB 10k SAS disks in a RAID60
> Two 10 GBit interfaces per server: One for communication betweens the
> 4 systems only (separate VLAN), the other one for regular traffic
> between clients and servers.
> 
> 
> Does this all make sense?
> Generally speaking: Is such a setup capable of providing fast enough
> storage for a virtualization cluster?
> Do you have any hints?
> 
> Thanks
> 
> Markus
I evaluated such a setup, but I decided against it when using a small 
number of nodes/brick.

The key reason was bad sync performance even when using ramdisks *and* 
two local bricks (ie: minimal network overhead). You can read more here: 
https://lists.gluster.org/pipermail/gluster-users/2020-January/037601.html

The interesting thing is that when increasing the number of bricks, 
performance scaled well. So it seems gluster *can* be good at 
virtualization, but it need a large number of bricks (eg: an entire 
server rack or one-brick-for-physical-disk approach). This matches the 
experiences shared by other sysadmin.

Moreover, in order to have efficient resync/healing after a node reboot, 
you need to enable sharding (ie: the virtual disks will be divided in 
many small chunks). I was somewhat unconfortable doing that, as any 
problem with gluster incapable to mount the share would lead to quite 
trickly "file reconstruction puzzle".

So I ended with local storage (and a hot-standby server) rather than 
Gluster. If anyone has some different stories to share, I really am all 
ears.

Regards.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it [1]
email: g.danti at assyoma.it - info at assyoma.it
GPG public key ID: FF5F32A8