Gluster community:
Due to some recent issues with performance from one of our (internal) clients
that uses several GlusterFS setups, this is mainly an open question for generic
possible improvements.
(apart from the fact the users themselves do some 'less smart' things on
it....)
The setup used: a 3 node (replica=3) RHEL7 gluster (with RHGS, so currently that
is GlusterFS 6.0-37.1.el7rhgs). Each Gluster has 1 volume, exported through
NFS-Ganesha.
Each node also has a virtual IP, managed by pacemaker/corosync following
standard RedHat setup documentation
Size is not that big, 600GB space with around half of that actually used.
GlusterFS servers themselves each have 4 cores and 12GB memory. It might also be
important to note that these are VMware hosted nodes that make use of SAN
storage for the datastores.
Connected to that NFS (ganesha) exported share are just over 100 clients, all
RHEL6 and RHEL7, some spanning 10 network hops away. All of those clients are
(currently) using the same virtual-IP, so all end up on the same server.
(we did already advise them to spread that across the three servers).
Certain subfolders of the share hold (at times) large numbers of (small) files
that *should* peak at around 50.000 files, into a single, unhashed, directory
(this will of course make simple ls and find commands via NFS quite slow).
Note that I mentioned 'should', since at times it had anywhere between
250.000 and 1 million files in it (which of course is not advised). Using some
kind of hashing (subfolders spread per day/hour etc) was also already advised.
Problems that are often seen:
- Any kind of operation on VMware such as a vMotion, creating a VM snapshot etc.
on the node that has these 100+ clients connected causes such a temporary pause
that pacemaker decides to switch the resources (causing a failover of the
virtual IP address, thus clients connected suffer delay). One would expect this
to last just shy under a minute, then clients would happily continue. However
connected clients are stuck with a non-working mountpoint (commands as df, ls,
find etc simply hang.. they go into an uninterruptible sleep).
Mount are 'hard' mounts to insure guaranteed writes.
- Once the number of files are over the 100.000 mark (again into a single,
unhashed, folder) any operation on that share becomes very sluggish (even a df,
on a client, would take 20/30 seconds, a find command would take minutes to
complete).
If anyone can spot any ideas for improvement ?
Some config info (below is from a sandbox setup using the same values as the
affected gluster):
For Ganesha:
/etc/ganesha/ganesha.conf:
# BEGIN ANSIBLE MANAGED BLOCK
NFSv4 {
minor_versions = 0;
}
# END ANSIBLE MANAGED BLOCK
%include
"/var/run/gluster/shared_storage/nfs-ganesha/exports/export.BLAH.conf"
/var/run/gluster/shared_storage/nfs-ganesha/exports/export.BLAH.conf:
EXPORT{
Export_Id = 2;
Path = "/BLAH";
FSAL {
name = GLUSTER;
hostname="localhost";
volume="BLAH";
}
Access_type = RW;
Disable_ACL = true;
Squash="No_root_squash";
Pseudo="/BLAH";
Protocols = "3", "4" ;
Transports = "UDP","TCP";
SecType = "sys";
}
# gluster v info BLAH
Volume Name: BLAH
Type: Replicate
Volume ID: 6fee713c-4258-44d8-a849-f8d6b2991631
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: tlrvrhgluster03:/gluster/BLAH/export
Brick2: tlrvrhgluster02:/gluster/BLAH/export
Brick3: tlrvrhgluster01:/gluster/BLAH/export
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
ganesha.enable: on
features.cache-invalidation: on
performance.client-io-threads: off
nfs.disable: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
cluster.server-quorum-type: server
cluster.quorum-count: 2
performance.cache-refresh-timeout: 10
cluster.quorum-type: fixed
cluster.enable-shared-storage: enable
nfs-ganesha: enable
# lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
BLAH gfsvg Vwi-aotz-- 10.00g gfspool 36.45
gfspool gfsvg twi-aotz-- 48.00g 7.59 1.76
[gfspool_tdata] gfsvg Twi-ao---- 48.00g
[gfspool_tmeta] gfsvg ewi-ao---- 1.00g
[lvol0_pmspare] gfsvg ewi------- 48.00m
auditlv systemvg -wi-ao---- 252.00m
homelv systemvg -wi-ao---- 1.00g
rootlv systemvg -wi-ao---- 16.00g
swaplv systemvg -wi-ao---- 2.00g
tmplv systemvg -wi-ao---- 2.00g
varcorelv systemvg -wi-ao---- 1.00g
varloglv systemvg -wi-ao---- 6.00g
varlv systemvg -wi-ao---- 6.00g
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20201016/d128144e/attachment.html>