Gluster users - Nov 2019 - Sudden, dramatic performance drops with Glusterfs

There are options that can help a little bit with the ls/find.

Still, many devs will need to know your settings, so the volume's info is
very important.

Try the 'noatime,nodiratime' (if ZFS supports them).
Also, as this is a new cluster you can try to setup XFS and verify if the issue
is the same.
RedHat provide an XFS options' calculator but it requires aby kind of
subscription (even dev subscription is enough).


P.S.: As this is a new cluster, I would recommend you to switch to gluster v6.6
as v7 is too new (for my taste).

If the issue on XFS cannot be reproduced - the issue is either in the ZFS or in
the kernel tunables (sysctl).

I'm not sure what is the most suitable I/O scheduler for ZFS, so you should
check that too.



Edit: What kind of workload do you expect (size and number files, read:write
ratio, etc).

Best Regards,
Strahil Nikolov
On Nov 8, 2019 10:32, Michael Rightmire <Michael.Rightmire at KIT.edu>
wrote:
>
> Hi? Strahil, 
>
> Thanks for the reply. See below. 
>
> Also, as an aside, I tested by installing a single Cenots 7 machine with
the ZBOD, installed gluster and ZFSonLinux as recommended at..
>
?https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Administrator%20Guide/Gluster%20On%20ZFS/
>
> And created a gluster volume consisting of one brick made up of a local ZFS
raidz2, copied about 4 TB of data to it, and am having the same issue.
>
> The biggest part of the issue is with things like "ls" and
"find". IF I read a single file, or write a single file it works
great. But if I run rsync (which does alot of listing, writing, renaming, etc)
it is slow as garbage. I.e. a find command that will finish in 30 seconds when
run directly on the underlying ZFS directory, takes about an hour.
>
> ?
> Strahil wrote on 08-Nov-19 05:39:
>
> Hi Michael,
>
> What is your 'gluster volume info <VOL> ' showing.
>
> I've been playing with the install (since it's a fresh machine) so
I can't give you verbatim output. However, it was showing two bricks, one on
each server, started, and apparently healthy.
>>
>> How much is your zpool full ? Usually when it gets too full, the ZFS
performance drops seriosly.
>
> The zpool is only at about 30% usage. It's a new server setup.
> We have about 10TB of data on a 30TB volume (made up of two 30TB ZFS raidz2
bricks, each residing on different servers, via a 10GB dedicated Ethernet
connection.)
>>
>> Try to rsync a file directly to one of the bricks, then to the other
brick (don't forget to remove the files after that, as gluster will not know
about them).
>
> If I rsync manually, or scp a file directly to the zpool bricks (outside of
gluster) I get 30-100MBytes/s (depending on what I'm copying.)
> If I rsync THROUGH gluster (via the glusterfs mounts) I get 1 - 5MB/s
>>
>> What are your mounting options ? Usually 'noatime,nodiratime'
are a good start.
>
> I'll try these. Currently using ...
> (mounting TO serverA) serverA:/homes /glusterfs/homes??? glusterfs
defaults,_netdev 0 0
>>
>> Are you using ZFS provideed by Ubuntu packagees or directly from ZOL
project ?
>
> ZFS provided by Ubuntu 18 repo...
> ? libzfs2linux/bionic-updates,now 0.7.5-1ubuntu16.6 amd64
[installed,automatic]
> ? zfs-dkms/bionic-updates,bionic-updates,now 0.7.5-1ubuntu16.6 all
[installed]
> ? zfs-zed/bionic-updates,now 0.7.5-1ubuntu16.6 amd64 [installed,automatic]
> ? zfsutils-linux/bionic-updates,now 0.7.5-1ubuntu16.6 amd64 [installed]
>
> Gluster provided by. "add-apt-repository ppa:gluster/glusterfs-5"
...
> ? glusterfs 5.10
> ? Repository revision: git://git.gluster.org/glusterfs.git
>
> ? 
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> On Nov 6, 2019 12:50, Michael Rightmire <Michael.Rightmire at
KIT.edu> wrote:
>>>
>>> Hello list!
>>>
>>> I'm new to Glusterfs in general. We have chosen to use it as
our distributed file system on a new set of HA file servers.
>>>
>>> The setup is: 
>>> 2 SUPERMICRO SuperStorage Server 6049PE1CR36L with 24-4TB spinning
disks and NVMe for cache and slog.
>>> HBA not RAID card 
>>> Ubuntu 18.04 server (on both systems)
>>> ZFS filestorage
>>> Glusterfs 5.10
>>>
>>> Step one was to install Ubuntu, ZFS, and gluster. This all went
without issue.
>>> We have 3 ZFS raidz2 identical on both servers
>>> We have three glusterfs mirrored volumes - 1 attached to each raidz
<
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20191109/086378c9/attachment-0001.html>