Gluster users - Dec 2015 - Gluster - Performance issue while copying bulk files/folders

Hi,


I have production gluster file service used as a shared storage where the
content management system uses it as document root. I have run in to a
performance issue with the gluster/fuse client.

Looking for your thoughts and experience in resolving Gluster performance
issues:

Gluster Infrastructure

Gluster version :GlusterFS 3.7.6

2 gluster nodes of the same config below

Redhat EL7.0-64
Memory : 4GB
Processor : 2 x 2.0 Ghz
Network : 100 Mbps
File Storage Volume : NETAPP Storage LUN with 2.0 IOPS/GB

Gluster Volume information:

[root at GlusterFileServe1 ~]# gluster volume info

Volume Name: prodcmsroot
Type: Replicate
Volume ID: f1284bf0-1939-46f9-a672-a7716e362947
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: Server1:/glusterfs/brick1/prodcmsroot
Brick2: Server2:/glusterfs/brick1/prodcmsroot
Options Reconfigured:
performance.io-thread-count: 64
performance.cache-size: 1073741824
performance.readdir-ahead: on
performance.write-behind-window-size: 524288

[root at GlusterFileServe1  ~]#

The replication between Gluster node are quick and consistent.

The apache webservers are accessing the Gluster volume using native gluster
fuse client and located in the same VLAN as the Gluster Server.

GlusterFileServe1:/prodcmsroot  /mnt/glusterfs glusterfs
direct-io-mode=disable,defaults,_netdev 0 0

The server utilization (memory,cpu,network and disk 1/0) is relatively low

I am experiencing very slow performance while copying multiple file/folders
(approx 75 MB) and it takes atleast approx 35 min. Even copy a folder (with
multiple files/subfolders) within the Gluster volume take the same time.

However, if I do dd to check the copy speed, I get the below result.

[root at ClientServer ~]#  time sh -c "dd if=/dev/zero
of=/mnt/testmount/test.tmp bs=4k count=20000 && sync"
20000+0 records in
20000+0 records out
81920000 bytes (82 MB) copied, 17.1357 s, 4.8 MB/s

real    0m17.337s
user    0m0.031s
sys     0m0.317s


Anyone experience the same kind of performance issue, please let me know
your thoughts.

Cheers
Srikanth
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20151209/0682c3b8/attachment.html>

Hi members,

Really appreciate if you can share your thoughts or any feedback for
resolving the slow copy issue

Regards
Srikanth
On 10-Dec-2015 2:12 AM, "Srikanth Mampilakal" <srikanth.mampilakal
at gmail.com>
wrote:
> Hi,
>
>
> I have production gluster file service used as a shared storage where the
> content management system uses it as document root. I have run in to a
> performance issue with the gluster/fuse client.
>
> Looking for your thoughts and experience in resolving Gluster performance
> issues:
>
> Gluster Infrastructure
>
> Gluster version :GlusterFS 3.7.6
>
> 2 gluster nodes of the same config below
>
> Redhat EL7.0-64
> Memory : 4GB
> Processor : 2 x 2.0 Ghz
> Network : 100 Mbps
> File Storage Volume : NETAPP Storage LUN with 2.0 IOPS/GB
>
> Gluster Volume information:
>
> [root at GlusterFileServe1 ~]# gluster volume info
>
> Volume Name: prodcmsroot
> Type: Replicate
> Volume ID: f1284bf0-1939-46f9-a672-a7716e362947
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: Server1:/glusterfs/brick1/prodcmsroot
> Brick2: Server2:/glusterfs/brick1/prodcmsroot
> Options Reconfigured:
> performance.io-thread-count: 64
> performance.cache-size: 1073741824
> performance.readdir-ahead: on
> performance.write-behind-window-size: 524288
>
> [root at GlusterFileServe1  ~]#
>
> The replication between Gluster node are quick and consistent.
>
> The apache webservers are accessing the Gluster volume using native
> gluster fuse client and located in the same VLAN as the Gluster Server.
>
> GlusterFileServe1:/prodcmsroot  /mnt/glusterfs glusterfs
> direct-io-mode=disable,defaults,_netdev 0 0
>
> The server utilization (memory,cpu,network and disk 1/0) is relatively low
>
> I am experiencing very slow performance while copying multiple
> file/folders (approx 75 MB) and it takes atleast approx 35 min. Even copy a
> folder (with multiple files/subfolders) within the Gluster volume take the
> same time.
>
> However, if I do dd to check the copy speed, I get the below result.
>
> [root at ClientServer ~]#  time sh -c "dd if=/dev/zero
> of=/mnt/testmount/test.tmp bs=4k count=20000 && sync"
> 20000+0 records in
> 20000+0 records out
> 81920000 bytes (82 MB) copied, 17.1357 s, 4.8 MB/s
>
> real    0m17.337s
> user    0m0.031s
> sys     0m0.317s
>
>
> Anyone experience the same kind of performance issue, please let me know
> your thoughts.
>
> Cheers
> Srikanth
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20151211/64a96114/attachment.html>

On 12/09/2015 07:03 AM, Srikanth Mampilakal wrote:
> However, if I do dd to check the copy speed, I get the below result.
>
> [root at ClientServer ~]#  time sh -c "dd if=/dev/zero
> of=/mnt/testmount/test.tmp bs=4k count=20000 && sync"
> 20000+0 records in
> 20000+0 records out
> 81920000 bytes (82 MB) copied, 17.1357 s, 4.8 MB/s
>
> real    0m17.337s
> user    0m0.031s
> sys     0m0.317s

The block size used is not ideal for the gluster native client. Can you 
please attempt with a higher block size - 64k or 128k & check the 
throughput?

Regards,
Vijay