Strahil
2019-Oct-29 13:11 UTC
[Gluster-users] Performance drop when upgrading from 3.8 to 6.5
Hi Riccardo, You can set your mounts with 'noatime,nodiratime' options for better performance. Best Regards, Strahil NikolovOn Oct 29, 2019 12:50, Riccardo Murri <riccardo.murri at gmail.com> wrote:> > Hello Anoop, > > many thanks for your fast reply!? My comments inline below: > > > > > [1]: I have tried both the config where SAMBA 4.8 is using the > > > vfs_glusterfs.so backend, and the one where `smbd` is just writing to > > > a locally-mounted directory.? Doesn't seem to make a difference. > > > > Samba v4.8 is an EOL ed version. Please consider updating Samba to at > > least v4.9(rather v4.10) or higher. > > This is going to be tricky: I could find no backport package of recent > SAMBA to Ubuntu 16.04; I am using this one which has SAMBA 4.8 > https://launchpad.net/~mumblepins > > More recent packages from either the Ubuntu or Debian repositories do > not build on Ubuntu 16.04 because of changes in the packaging > infrastructure. > > Anyway, I was running SAMBA 4.8 before the upgrade and still getting > 40MB/s, so I don't think SAMBA is the core of the issue... > > > Can you paste the output of `testparm -s` along with the output of > > `gluster volume info <VOLNAME>` ? > > Here's `testparm -s` on the server using `vfs_glusterfs` (the "active" > share is the one with the perf problems):: > > ``` > $ testparm -s > Load smb config files from /etc/samba/smb.conf > rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384) > WARNING: The "syslog only" option is deprecated > Processing section "[homes]" > Processing section "[active]" > Loaded services file OK. > WARNING: some services use vfs_fruit, others don't. Mounting them in > conjunction on OS X clients results in undefined behaviour. > > Server role: ROLE_STANDALONE > > # Global parameters > [global] > ??? dns proxy = No > ??? load printers = No > ??? map to guest = Bad User > ??? name resolve order = lmhosts > ??? netbios name = REDACTED1 > ??? obey pam restrictions = Yes > ??? pam password change = Yes > ??? passwd chat = *Enter\snew\s*\spassword:* %n\n > *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* . > ??? passwd program = /usr/bin/passwd %u > ??? printcap cache time = 0 > ??? printcap name = /dev/null > ??? security = USER > ??? server role = standalone server > ??? server string = SAMBA Server %v > ??? syslog only = Yes > ??? unix password sync = Yes > ??? workgroup = REDACTED > ??? idmap config * : backend = tdb > > > [homes] > ??? browseable = No > ??? comment = Work Directories > ??? create mask = 0700 > ??? directory mask = 0700 > ??? read only = No > ??? valid users = %S > ??? vfs objects = fruit streams_xattr > > > [active] > ??? create mask = 0775 > ??? directory mask = 0775 > ??? kernel share modes = No > ??? path = /active > ??? read only = No > ??? vfs objects = glusterfs > ??? glusterfs:volume = glusterfs > ??? glusterfs:volfile_server = glusterfs5 glusterfs4 glusterfs3 > glusterfs2 glusterfs1 > ??? glusterfs:logfile = /var/log/samba/glusterfs-vol-active.log > ??? glusterfs:loglevel = 1 > ``` > > > Here's `testparm -s` on the server writing directly to the GlusterFS > mount point:: > > ``` > $ testparm -s > Load smb config files from /etc/samba/smb.conf > rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384) > WARNING: The "syslog only" option is deprecated > Processing section "[homes]" > Processing section "[active]" > Loaded services file OK. > WARNING: some services use vfs_fruit, others don't. Mounting them in > conjunction on OS X clients results in undefined behaviour. > > Server role: ROLE_STANDALONE > > # Global parameters > [global] > ??? allow insecure wide links = Yes > ??? dns proxy = No > ??? load printers = No > ??? map to guest = Bad User > ??? name resolve order = lmhosts > ??? netbios name = REDACTED2 > ??? obey pam restrictions = Yes > ??? pam password change = Yes > ??? passwd chat = *Enter\snew\s*\spassword:* %n\n > *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* . > ??? passwd program = /usr/bin/passwd %u > ??? printcap cache time = 0 > ??? printcap name = /dev/null > ??? security = USER > ??? server role = standalone server > ??? server string = SAMBA Server %v > ??? syslog only = Yes > ??? unix password sync = Yes > ??? workgroup = REDACTED > ??? idmap config * : backend = tdb > > > [homes] > ??? browseable = No > ??? comment = Work Directories > ??? create mask = 0700 > ??? directory mask = 0700 > ??? read only = No > ??? valid users = %S > ??? vfs objects = fruit streams_xattr > > > [active] > ??? create mask = 0775 > ??? directory mask = 0775 > ??? path = /data/active > ??? read only = No > ??? wide links = Yes > ``` > > Here's the volume info: > ``` > $ sudo gluster volume info glusterfs > > Volume Name: glusterfs > Type: Distribute > Volume ID: a3358ff6-5cec-4a65-9ecf-a63bbe56dfd9 > Status: Started > Snapshot Count: 0 > Number of Bricks: 5 > Transport-type: tcp > Bricks: > Brick1: glusterfs5:/srv/glusterfs > Brick2: glusterfs4:/srv/glusterfs > Brick3: glusterfs3:/srv/glusterfs > Brick4: glusterfs1:/srv/glusterfs > Brick5: glusterfs2:/srv/glusterfs > Options Reconfigured: > diagnostics.client-log-level: WARNING > diagnostics.brick-log-level: INFO > features.uss: disable > features.barrier: disable > performance.client-io-threads: on > transport.address-family: inet > nfs.disable: on > snap-activate-on-create: enable > ``` > > > > > [2]: Actually, since the servers are VMs on an OpenStack cloud, I > > > created new virtual machines, installed GlusterFS 6 fresh, mounted > > > the old bricks in the same brick locations, > > > > How did you mount old bricks in the new location? > > They're `ext4` disks: > ``` > $ fgrep glusterfs /etc/fstab > LABEL=/srv/glusterfs /srv/glusterfs ext4 rw 0 0 > ``` > > Actually, the GlusterFS volume is 95% full -- could this have an impact? > > Thanks, > Riccardo > ________ > > Community Meeting Calendar: > > APAC Schedule - > Every 2nd and 4th Tuesday at 11:30 AM IST > Bridge: https://bluejeans.com/118564314 > > NA/EMEA Schedule - > Every 1st and 3rd Tuesday at 01:00 PM EDT > Bridge: https://bluejeans.com/118564314 > > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users
Riccardo Murri
2019-Nov-01 04:24 UTC
[Gluster-users] Performance drop when upgrading from 3.8 to 6.5
Hello all, I have done some further testing and found out that I get the bad performance with a freshly-installed cluster running 6.6. Also the performance drop is there with plain `rsync` into the GlusterFS mountpoint, so SAMBA plays no role in it. In other words, for my installations, performance of 6.5 and 6.6 is *half* of what 3.8 used to deliver. Was any default option changed (e.g., in the FUSE client) or in the xlator stack that I can start looking at as a potential culprit? Or any direction where to look at for debugging? Thanks for any help! Riccardo
Riccardo Murri
2019-Nov-04 11:15 UTC
[Gluster-users] Performance drop when upgrading from 3.8 to 6.5
Hello Strahil,> You can set your mounts with 'noatime,nodiratime' options for better performance.Thanks for the suggestion! I'll try that eventually, but I don't think `noatime` will make much difference on write-mostly workload. Thanks, R