[Gluster-users] Sudden, dramatic performance drops with Glusterfs

Strahil
Fri, 08 Nov 2019 04:39:19 -0000

Hi Michael,

What is your 'gluster volume info <VOL> ' showing.

How much is your zpool full ? Usually when it gets too full, the ZFS performance
drops seriosly.

Try to rsync a file directly to one of the bricks, then to the other brick
(don't forget to remove the files after that, as gluster will not know about
them).

What are your mounting options ? Usually 'noatime,nodiratime' are a good
start.

Are you using ZFS provideed by Ubuntu packagees or directly from ZOL project ?

Best Regards,
Strahil NikolovOn Nov 6, 2019 12:50, Michael Rightmire <Michael.Rightmire at
KIT.edu> wrote:
>
> Hello list!
>
> I'm new to Glusterfs in general. We have chosen to use it as our
distributed file system on a new set of HA file servers.
>
> The setup is:
> 2 SUPERMICRO SuperStorage Server 6049PE1CR36L with 24-4TB spinning disks
and NVMe for cache and slog.
> HBA not RAID card
> Ubuntu 18.04 server (on both systems)
> ZFS filestorage
> Glusterfs 5.10
>
> Step one was to install Ubuntu, ZFS, and gluster. This all went without
issue.
> We have 3 ZFS raidz2 identical on both servers
> We have three glusterfs mirrored volumes - 1 attached to each raidz on each
server. I.e.
>
> And mounted the gluster volumes as (for example) "/glusterfs/homes
-> /zpool/homes". I.e.
> gluster volume create homes replica 2 transport tcp
server1:/zpool-homes/homes server2:/zpool-homes/homes force
> (on server1) server1:/homes???? 44729413504 16032705152 28696708352? 36%
/glusterfs/homes
>
> The problem is, the performance has deteriorated terribly.?
> We needed to copy all of our data from the old server to the new glusterfs
volumes (appx. 60TB).
> We decided to do this with multiple rsync commands (like 400 simultanous
rsyncs)
> The copy went well for the first 4 days, with an average across all rsyncs
of? 150-200 MBytes per second.
> Then, suddenly, on the fourth day, it dropped to about 50 MBytes/s.
> Then, by the end of the day, down to ~5MBytes/s (five).
> I've stopped the rsyncs, and I can still copy an individual file across
to the glusterfs shared directory at 100MB/s.
> But actions such as "ls -la" or "find" take forever!
>
> Are there obvious flaws in my setup to correct?
> How can I better troubleshoot this?
>
> Thanks!
> --
>
> Mike
>
> ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20191108/48730835/attachment.html>