Strahil
2019-Nov-01 16:43 UTC
[Gluster-users] Performance drop when upgrading from 3.8 to 6.5
I'm using replicated volumes. In your case, you got a distribited volume. Have you noticed if slowness is only when accessing the files from specific node ? Best Regards, Strahil NikolovOn Nov 1, 2019 17:28, Riccardo Murri <riccardo.murri at gmail.com> wrote:> > Hello Strahil > > > What options do you use i? your cluster? > > I'm not sure what exact info you would like to see? > > Here's how clients mount the GlusterFS volume: > ``` > $ fgrep gluster /proc/mounts > tp-glusterfs5:/glusterfs /net/glusterfs fuse.glusterfs > rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 > 0 0 > ``` > > Here's some server-side info: > > ``` > $ sudo gluster volume info glusterfs > > Volume Name: glusterfs > Type: Distribute > Volume ID: a3358ff6-5cec-4a65-9ecf-a63bbe56dfd9 > Status: Started > Snapshot Count: 0 > Number of Bricks: 5 > Transport-type: tcp > Bricks: > Brick1: pelkmanslab-tp-glusterfs5:/srv/glusterfs > Brick2: pelkmanslab-tp-glusterfs4:/srv/glusterfs > Brick3: pelkmanslab-tp-glusterfs3:/srv/glusterfs > Brick4: pelkmanslab-tp-glusterfs1:/srv/glusterfs > Brick5: pelkmanslab-tp-glusterfs2:/srv/glusterfs > Options Reconfigured: > diagnostics.client-log-level: WARNING > diagnostics.brick-log-level: INFO > features.uss: disable > features.barrier: disable > performance.client-io-threads: on > transport.address-family: inet > nfs.disable: on > snap-activate-on-create: enable > > $ sudo gluster volume get all all > Option????????????????????????????????? Value > ------????????????????????????????????? ----- > cluster.server-quorum-ratio???????????? 51 > cluster.enable-shared-storage?????????? disable > cluster.op-version????????????????????? 60000 > cluster.max-op-version????????????????? 60000 > cluster.brick-multiplex???????????????? disable > cluster.max-bricks-per-process????????? 250 > cluster.localtime-logging?????????????? disable > cluster.daemon-log-level??????????????? INFO > > $ cat /etc/glusterfs/glusterd.vol > volume management > ??? type mgmt/glusterd > ??? option working-directory /var/lib/glusterd > ??? option transport-type socket,rdma > ??? option transport.socket.keepalive-time 10 > ??? option transport.socket.keepalive-interval 2 > ??? option transport.socket.read-fail-log off > ??? option transport.socket.listen-port 24007 > ??? option transport.rdma.listen-port 24008 > ??? option ping-timeout 0 > ??? option event-threads 1 > #?? option lock-timer 180 > #?? option transport.address-family inet6 > #?? option base-port 49152 > ??? option max-port? 60999 > end-volume > ``` > > Both server and clients are running v6.5 and op-version is 60000 everywhere. > > Thanks, > Riccardo
Riccardo Murri
2019-Nov-01 19:21 UTC
[Gluster-users] Performance drop when upgrading from 3.8 to 6.5
Dear Strahil,> Have you noticed if slowness is only when accessing the files from specific node ?I am copying a largest of image files into the GlusterFS volume -- slowness is on the aggregated performance (e.g., it takes ~300 minutes to copy 376GB worth of files). Given the high number of files (O(100'000)), I guess they're +/- equally distributed across nodes. Report of `df -h` across server nodes shows no imbalance. Thanks, R