David Cowe
2016-Aug-30 12:53 UTC
[Gluster-users] Terrible performance when a gluster node is down/rebooting
Hello,
We have a 5 node gluster setup with the following configuration;
[root at gb0015nasslow01 Test]# gluster vol info all
Volume Name: data
Type: Tier
Volume ID: 702daa3d-b3fa-4e66-bcec-a13f7ec1d47d
Status: Started
Number of Bricks: 6
Transport-type: tcp
Hot Tier :
Hot Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick1: glusternasbackup01:/bricks/hotbrick/brick
Brick2: glusternasfast02:/bricks/brick4/brick
Brick3: glusternasfast01:/bricks/brick3/brick
Cold Tier:
Cold Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick4: glusternasslow01:/bricks/brick1/brick
Brick5: glusternasslow02:/bricks/brick2/brick
Brick6: glusternasbackup01:/bricks/coldbrick/brick
Options Reconfigured:
performance.write-behind-window-size: 100mb
performance.cache-size: 20GB
storage.batch-fsync-delay-usec: 0
server.allow-insecure: on
performance.stat-prefetch: off
performance.readdir-ahead: on
features.ctr-enabled: on
cluster.tier-mode: cache
When we shutdown/reboot/pull the power from 1 of the nodes, the other
servers are very laggy when even doing an ls within the glusterfs mounted
filesystem (/glusterdata).
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 50G 4.7G 46G 10% /
devtmpfs 48G 0 48G 0% /dev
tmpfs 48G 0 48G 0% /dev/shm
tmpfs 48G 66M 48G 1% /run
tmpfs 48G 0 48G 0% /sys/fs/cgroup
/dev/sdb1 110T 221M 110T 1% /bricks/brick1
/dev/sda2 497M 201M 297M 41% /boot
/dev/sda1 200M 9.5M 191M 5% /boot/efi
/dev/mapper/centos-home 225G 33M 224G 1% /home
localhost:data 109T 464G 108T 1% /glusterdata
tmpfs 9.5G 0 9.5G 0% /run/user/0
During the time the node is down or coming back up, we not only get the lag
described above on the server but file reads and writes to the gluster
volume pretty much stall.
Can anyone give me some much needed advice as surely this should not be the
case?
*packages installed;*
yum list gluster*
Installed Packages
glusterfs.x86_64 3.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-api.x86_64 3.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-cli.x86_64 3.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-client-xlators.x86_64 3.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-fuse.x86_64 3.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-libs.x86_64 3.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-server.x86_64 3.7.14-1.el7
@glusterfs-3.7-x86_64
yum list samba*
Installed Packages
samba.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64
samba-client.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64
samba-client-libs.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64
samba-common.noarch
4.2.10-7.el7_2
@centos7-updates-x86_64
samba-common-libs.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64
samba-common-tools.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64
samba-libs.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64
samba-vfs-glusterfs.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64
samba-winbind.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64
samba-winbind-clients.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64
samba-winbind-modules.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64
yum list ctdb*
Installed Packages
ctdb.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64
Background - We use CTDB for VIP's and heartbeat and we present our gluster
volume via samba to our Windows clients.
Regards,
David
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160830/bd3d63f0/attachment.html>