Gluster users - Jan 2016 - Poor and very slow performance on glusterfs mounted volume

Hello

Im using 2 standalone servers as share storage servers with glusterfs 3.7 when i
change directory and run command inside a mounted volume for each operation i
have to wait at least 5 seconds for result and prompt
Im using replicate option and delay maybe for write but not for reading 

im using gluster as my /var/www/ share along all of my instance and servers but
for a simple ls or cd command to this directory i have to wait 5 seconds any
suggestion for me ?

here is some information about the servers and glusterfs :

OS : Ubuntu 14.04.3 LTS
Version info : glusterfs 3.7.6 built on Nov  9 2015 15:17:05

Ping between 2 nodes result : 
4 packets transmitted, 4 received, 0% packet loss, time 2998ms
rtt min/avg/max/mdev = 0.042/0.056/0.071/0.014 ms

iperf : 
Interval           Transfer        Bandwidth
0.0-10.0 sec  10.5 GBytes  9.03 Gbits/sec

dd test : 
1573468160 bytes (1.6 GB) copied, 70.8977 s, 22.2 MB/s



 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160113/b96d8225/attachment.html>

Hello,

we are aware of this, as you can find in mail list archives.

You can get things better, but not completely remove this issue for now. It
mostly depend on number of files in folder when doing ls.
Small files provide much lower performance than big one during copy & co.

Replicate option slow down read, as your client contact both servers to get
metada.

Things I did to get things better in this setup:

fstab option (on client) :
defaults,direct-io-mode=disable,_netdev,backupvolfile-server=mysecondservername
fstab option (on gluster server volumes):
 defaults,noatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k,nobarrier

*sysctl (clients and servers):*
vm.swappiness=0
net.core.rmem_max=67108864
net.core.wmem_max=67108864
# increase Linux autotuning TCP buffer limit to 32MB
net.ipv4.tcp_rmem=4096 87380 33554432
net.ipv4.tcp_wmem=4096 65536 33554432
# increase the length of the processor input queue
net.core.netdev_max_backlog=30000
# recommended default congestion control is htcp
net.ipv4.tcp_congestion_control=htcp

*Gluster volume options:*
performance.cache-size: 1GB
performance.io-thread-count: 16
performance.readdir-ahead: enable
performance.read-ahead: disable
performance.client-io-threads: on
performance.write-behind-window-size: 1MB
cluster.lookup-optimize: on
client.event-threads: 4
server.event-threads: 4
cluster.readdir-optimize: on


Cordialement,
Mathieu CHATEAU
http://www.lotp.fr

2016-01-13 7:39 GMT+01:00 Arash Shams <ara4sh at hotmail.com>:
> Hello
>
> Im using 2 standalone servers as share storage servers with glusterfs 3.7
> when i change directory and run command inside a mounted volume for each
> operation i have to wait at least 5 seconds for result and prompt
> Im using replicate option and delay maybe for write but not for reading
>
> im using gluster as my /var/www/ share along all of my instance and
> servers but for a simple ls or cd command to this directory i have to wait
> 5 seconds any suggestion for me ?
>
> here is some information about the servers and glusterfs :
>
> OS : Ubuntu 14.04.3 LTS
> Version info : glusterfs 3.7.6 built on Nov  9 2015 15:17:05
>
> Ping between 2 nodes result :
> 4 packets transmitted, 4 received, 0% packet loss, time 2998ms
> rtt min/avg/max/mdev = 0.042/0.056/0.071/0.014 ms
>
> iperf :
> Interval           Transfer        Bandwidth
> 0.0-10.0 sec  10.5 GBytes  9.03 Gbits/sec
>
> dd test :
> 1573468160 bytes (1.6 GB) copied, 70.8977 s, 22.2 MB/s
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160113/b53c6007/attachment.html>

On 01/13/2016 12:09 PM, Arash Shams wrote:
> Hello
>
> Im using 2 standalone servers as share storage servers with glusterfs 
> 3.7 when i change directory and run command inside a mounted volume 
> for each operation i have to wait at least 5 seconds for result and 
> prompt
> Im using replicate option and delay maybe for write but not for reading
>
> im using gluster as my /var/www/ share along all of my instance and 
> servers but for a simple ls or cd command to this directory i have to 
> wait 5 seconds any suggestion for me ?
Anuradha made some improvements recently to address this. 
https://bugzilla.redhat.com/show_bug.cgi?id=1250803 contains more 
information. The fix is available in next 3.7.x release.

Pranith
>
> here is some information about the servers and glusterfs :
>
> OS : Ubuntu 14.04.3 LTS
> Version info : glusterfs 3.7.6 built on Nov  9 2015 15:17:05
>
> Ping between 2 nodes result :
> 4 packets transmitted, 4 received, 0% packet loss, time 2998ms
> rtt min/avg/max/mdev = 0.042/0.056/0.071/0.014 ms
>
> iperf :
> Interval           Transfer        Bandwidth
> 0.0-10.0 sec  10.5 GBytes  9.03 Gbits/sec
>
> dd test :
> 1573468160 bytes (1.6 GB) copied, 70.8977 s, 22.2 MB/s
>
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160113/d381a6d9/attachment.html>