PEPONNET, Cyril N (Cyril)
2014-Nov-21 18:55 UTC
[Gluster-users] rmtab file is a bottleneck when lot of clients are accessing a volume through NFS
Folks, This feature: http://review.gluster.org/#/c/4430/ Create bottleneck when several clients are accessing a nfs volume. On our setup: Gluster 3.5.2 under centos7. Hardware: dual Xeon? CPU E5-2640 64GB RAM SSD for rootfs 10Gb NIC Context: Around 700 nfs clients for small file or vm images. The issue: NFS client got intermittent hang (every minutes / for 10s each time). Even an ?rpcinfo -t server nfs 3" will hang. Gluster nfs process literally eat the CPU of the server The cause: the rmtab file located in /var/lib/glusterd/nfs/ is flushed from memory to /var/lib/glusterd/nfs/rmtab.tmp. During this time, NFS server hang literraly. [cid:2BF1DF8D-4144-48FF-A63E-F5D2F61285CA at eng.timetra.com] [cid:D11F71BF-A4FD-4B96-B7A1-17AB2F7114A3 at eng.timetra.com] Workaroud: Move the file to memory for faster I/O using this option: set nfs.mount-rmtab: /dev/shm/glusterfs.rmtab Result: We still have some hang but for ~300ms now, the load average of the server is WAY better. Personal thought: This feature is not usable and should be disabled by default. I raised Bug 1166862<https://bugzilla.redhat.com/show_bug.cgi?id=1166862>. -- Cyril Peponnet -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141121/b404808c/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: loadaverage.png Type: image/png Size: 116283 bytes Desc: loadaverage.png URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141121/b404808c/attachment.png> -------------- next part -------------- A non-text attachment was scrubbed... Name: diskusage.png Type: image/png Size: 120872 bytes Desc: diskusage.png URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141121/b404808c/attachment-0001.png>