Riccardo Murri
2019-Oct-29 09:59 UTC
[Gluster-users] Performance drop when upgrading from 3.8 to 6.5
Hello, I recently upgraded[2] our servers from GlusterFS 3.8 (old GlusterFS repo for Ubuntu 16.04) to 6.0 (gotten from the GlusterFS PPA for Ubuntu 16.04 "xenial"). The sustained write performance nearly dropped to half it was before. We copy a large (a few 10'000s) number of image files (each 2 to 10 MB size) from the microscope where they were acquired to a SAMBA server which mounts[1] the GlusterFS volume; before the upgrade, we could write at about 40MB/s, after the upgrade, this dropped to 20MB/s. This is the version of server and client software installed: ``` $ dpkg -l '*gluster*' Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==============================-====================-====================-=================================================================ii glusterfs-client 6.5-ubuntu1~xenial1 amd64 clustered file-system (client package) ii glusterfs-common 6.5-ubuntu1~xenial1 amd64 GlusterFS common libraries and translator modules ii glusterfs-server 6.5-ubuntu1~xenial1 amd64 clustered file-system (server package) ``` Op version has been upped to 60000: ``` $ sudo gluster volume get all cluster.op-version Option Value ------ ----- cluster.op-version 60000 $ sudo gluster volume get all cluster.max-op-version Option Value ------ ----- cluster.max-op-version 60000 ``` Running `sudo gluster volume status all clients` reports that all clients are on op-version 60000, too. Any suggestions on what to look for or changes to try out? Thanks, Riccardo [1]: I have tried both the config where SAMBA 4.8 is using the vfs_glusterfs.so backend, and the one where `smbd` is just writing to a locally-mounted directory. Doesn't seem to make a difference. [2]: Actually, since the servers are VMs on an OpenStack cloud, I created new virtual machines, installed GlusterFS 6 fresh, mounted the old bricks in the same brick locations, and restarted the cluster. I had to fiddle a bit with the files in `/var/lib/glusterfs` because the hostnames and IPs changed but did not do anything else than `sed -e s/old_hostname/new_hostname/` or similarly renaming files. In particular, I did not touch the extended attributes in the brick directory.
Anoop C S
2019-Oct-29 10:07 UTC
[Gluster-users] Performance drop when upgrading from 3.8 to 6.5
On Tue, 2019-10-29 at 10:59 +0100, Riccardo Murri wrote:> Hello, > > I recently upgraded[2] our servers from GlusterFS 3.8 (old GlusterFS > repo for Ubuntu 16.04) to 6.0 (gotten from the GlusterFS PPA for > Ubuntu 16.04 "xenial"). > > The sustained write performance nearly dropped to half it was before. > We copy a large (a few 10'000s) number of image files (each 2 to 10 > MB > size) from the microscope where they were acquired to a SAMBA server > which mounts[1] the GlusterFS volume; before the upgrade, we could > write at about 40MB/s, after the upgrade, this dropped to 20MB/s. > > This is the version of server and client software installed: > ``` > $ dpkg -l '*gluster*' > Desired=Unknown/Install/Remove/Purge/Hold > > Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig- > > aWait/Trig-pend > > / Err?=(none)/Reinst-required (Status,Err: uppercase=bad) > > > / > > > Name Version Architecture > Description > +++-==============================-====================- > ====================- > =================================================================> ii glusterfs-client 6.5-ubuntu1~xenial1 amd64 > clustered file-system (client package) > ii glusterfs-common 6.5-ubuntu1~xenial1 amd64 > GlusterFS common libraries and translator modules > ii glusterfs-server 6.5-ubuntu1~xenial1 amd64 > clustered file-system (server package) > ``` > Op version has been upped to 60000: > ``` > $ sudo gluster volume get all cluster.op-version > Option Value > ------ ----- > cluster.op-version 60000 > > $ sudo gluster volume get all cluster.max-op-version > Option Value > ------ ----- > cluster.max-op-version 60000 > ``` > > Running `sudo gluster volume status all clients` reports that all > clients are on op-version 60000, too. > > Any suggestions on what to look for or changes to try out? > > Thanks, > Riccardo > > [1]: I have tried both the config where SAMBA 4.8 is using the > vfs_glusterfs.so backend, and the one where `smbd` is just writing to > a locally-mounted directory. Doesn't seem to make a difference.Samba v4.8 is an EOL ed version. Please consider updating Samba to at least v4.9(rather v4.10) or higher. Can you paste the output of `testparm -s` along with the output of `gluster volume info <VOLNAME>` ?> [2]: Actually, since the servers are VMs on an OpenStack cloud, I > created new virtual machines, installed GlusterFS 6 fresh, mounted > the > old bricks in the same brick locations,How did you mount old bricks in the new location?> and restarted the cluster. I > had to fiddle a bit with the files in `/var/lib/glusterfs` because > the > hostnames and IPs changed but did not do anything else than `sed -e > s/old_hostname/new_hostname/` or similarly renaming files. In > particular, I did not touch the extended attributes in the brick > directory. > ________ > > Community Meeting Calendar: > > APAC Schedule - > Every 2nd and 4th Tuesday at 11:30 AM IST > Bridge: https://bluejeans.com/118564314 > > NA/EMEA Schedule - > Every 1st and 3rd Tuesday at 01:00 PM EDT > Bridge: https://bluejeans.com/118564314 > > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users