Srikanth Mampilakal
2015-Dec-10 14:29 UTC
[Gluster-users] Gluster - Performance issue while copying bulk files/folders
Hi members, Really appreciate if you can share your thoughts or any feedback for resolving the slow copy issue Regards Srikanth On 10-Dec-2015 2:12 AM, "Srikanth Mampilakal" <srikanth.mampilakal at gmail.com> wrote:> Hi, > > > I have production gluster file service used as a shared storage where the > content management system uses it as document root. I have run in to a > performance issue with the gluster/fuse client. > > Looking for your thoughts and experience in resolving Gluster performance > issues: > > Gluster Infrastructure > > Gluster version :GlusterFS 3.7.6 > > 2 gluster nodes of the same config below > > Redhat EL7.0-64 > Memory : 4GB > Processor : 2 x 2.0 Ghz > Network : 100 Mbps > File Storage Volume : NETAPP Storage LUN with 2.0 IOPS/GB > > Gluster Volume information: > > [root at GlusterFileServe1 ~]# gluster volume info > > Volume Name: prodcmsroot > Type: Replicate > Volume ID: f1284bf0-1939-46f9-a672-a7716e362947 > Status: Started > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: Server1:/glusterfs/brick1/prodcmsroot > Brick2: Server2:/glusterfs/brick1/prodcmsroot > Options Reconfigured: > performance.io-thread-count: 64 > performance.cache-size: 1073741824 > performance.readdir-ahead: on > performance.write-behind-window-size: 524288 > > [root at GlusterFileServe1 ~]# > > The replication between Gluster node are quick and consistent. > > The apache webservers are accessing the Gluster volume using native > gluster fuse client and located in the same VLAN as the Gluster Server. > > GlusterFileServe1:/prodcmsroot /mnt/glusterfs glusterfs > direct-io-mode=disable,defaults,_netdev 0 0 > > The server utilization (memory,cpu,network and disk 1/0) is relatively low > > I am experiencing very slow performance while copying multiple > file/folders (approx 75 MB) and it takes atleast approx 35 min. Even copy a > folder (with multiple files/subfolders) within the Gluster volume take the > same time. > > However, if I do dd to check the copy speed, I get the below result. > > [root at ClientServer ~]# time sh -c "dd if=/dev/zero > of=/mnt/testmount/test.tmp bs=4k count=20000 && sync" > 20000+0 records in > 20000+0 records out > 81920000 bytes (82 MB) copied, 17.1357 s, 4.8 MB/s > > real 0m17.337s > user 0m0.031s > sys 0m0.317s > > > Anyone experience the same kind of performance issue, please let me know > your thoughts. > > Cheers > Srikanth > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151211/64a96114/attachment.html>
Игорь Бирюлин
2015-Dec-10 15:42 UTC
[Gluster-users] Gluster - Performance issue while copying bulk files/folders
Hello, I doesn't use version 3.7 in production. I think need try new options (I don't know how it improve work with big files but it should improve work with small files): gluster v set prodcmsroot client.event-threads 4 gluster v set prodcmsroot server.event-threads 4 gluster v set prodcmsroot cluster.lookup-optimize on ??re info about this: https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/Small_File_Performance_Enhancements.html And may be increase number of threads in IO threads translator: gluster v set prodcmsroot performance.io-thread-count 48 I recommend try this on test environment. 2015-12-10 17:29 GMT+03:00 Srikanth Mampilakal <shrikanth1980 at gmail.com>:> Hi members, > > Really appreciate if you can share your thoughts or any feedback for > resolving the slow copy issue > > Regards > Srikanth > On 10-Dec-2015 2:12 AM, "Srikanth Mampilakal" < > srikanth.mampilakal at gmail.com> wrote: > >> Hi, >> >> >> I have production gluster file service used as a shared storage where the >> content management system uses it as document root. I have run in to a >> performance issue with the gluster/fuse client. >> >> Looking for your thoughts and experience in resolving Gluster performance >> issues: >> >> Gluster Infrastructure >> >> Gluster version :GlusterFS 3.7.6 >> >> 2 gluster nodes of the same config below >> >> Redhat EL7.0-64 >> Memory : 4GB >> Processor : 2 x 2.0 Ghz >> Network : 100 Mbps >> File Storage Volume : NETAPP Storage LUN with 2.0 IOPS/GB >> >> Gluster Volume information: >> >> [root at GlusterFileServe1 ~]# gluster volume info >> >> Volume Name: prodcmsroot >> Type: Replicate >> Volume ID: f1284bf0-1939-46f9-a672-a7716e362947 >> Status: Started >> Number of Bricks: 1 x 2 = 2 >> Transport-type: tcp >> Bricks: >> Brick1: Server1:/glusterfs/brick1/prodcmsroot >> Brick2: Server2:/glusterfs/brick1/prodcmsroot >> Options Reconfigured: >> performance.io-thread-count: 64 >> performance.cache-size: 1073741824 >> performance.readdir-ahead: on >> performance.write-behind-window-size: 524288 >> >> [root at GlusterFileServe1 ~]# >> >> The replication between Gluster node are quick and consistent. >> >> The apache webservers are accessing the Gluster volume using native >> gluster fuse client and located in the same VLAN as the Gluster Server. >> >> GlusterFileServe1:/prodcmsroot /mnt/glusterfs glusterfs >> direct-io-mode=disable,defaults,_netdev 0 0 >> >> The server utilization (memory,cpu,network and disk 1/0) is relatively low >> >> I am experiencing very slow performance while copying multiple >> file/folders (approx 75 MB) and it takes atleast approx 35 min. Even copy a >> folder (with multiple files/subfolders) within the Gluster volume take the >> same time. >> >> However, if I do dd to check the copy speed, I get the below result. >> >> [root at ClientServer ~]# time sh -c "dd if=/dev/zero >> of=/mnt/testmount/test.tmp bs=4k count=20000 && sync" >> 20000+0 records in >> 20000+0 records out >> 81920000 bytes (82 MB) copied, 17.1357 s, 4.8 MB/s >> >> real 0m17.337s >> user 0m0.031s >> sys 0m0.317s >> >> >> Anyone experience the same kind of performance issue, please let me know >> your thoughts. >> >> Cheers >> Srikanth >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151210/2ef329d7/attachment.html>
Anuradha Talur
2015-Dec-11 05:15 UTC
[Gluster-users] Gluster - Performance issue while copying bulk files/folders
Response inline. ----- Original Message -----> From: "Srikanth Mampilakal" <shrikanth1980 at gmail.com> > To: gluster-users at gluster.org > Sent: Thursday, December 10, 2015 7:59:04 PM > Subject: Re: [Gluster-users] Gluster - Performance issue while copying bulk files/folders > > > > Hi members, > > Really appreciate if you can share your thoughts or any feedback for > resolving the slow copy issue > > Regards > Srikanth > On 10-Dec-2015 2:12 AM, "Srikanth Mampilakal" < srikanth.mampilakal at gmail.com > > wrote: > > > > Hi, > > > I have production gluster file service used as a shared storage where the > content management system uses it as document root. I have run in to a > performance issue with the gluster/fuse client. > > Looking for your thoughts and experience in resolving Gluster performance > issues: > > Gluster Infrastructure > > Gluster version :GlusterFS 3.7.6 > > 2 gluster nodes of the same config below > > Redhat EL7.0-64 > Memory : 4GB > Processor : 2 x 2.0 Ghz > Network : 100 Mbps > File Storage Volume : NETAPP Storage LUN with 2.0 IOPS/GB > > Gluster Volume information: > > [root at GlusterFileServe1 ~]# gluster volume info > > Volume Name: prodcmsroot > Type: Replicate > Volume ID: f1284bf0-1939-46f9-a672-a7716e362947 > Status: Started > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: Server1:/glusterfs/brick1/prodcmsroot > Brick2: Server2:/glusterfs/brick1/prodcmsroot > Options Reconfigured: > performance.io-thread-count: 64 > performance.cache-size: 1073741824 > performance.readdir-ahead: on > performance.write-behind-window-size: 524288 > > [root at GlusterFileServe1 ~]# > > The replication between Gluster node are quick and consistent. > > The apache webservers are accessing the Gluster volume using native gluster > fuse client and located in the same VLAN as the Gluster Server. > > GlusterFileServe1:/prodcmsroot /mnt/glusterfs glusterfs > direct-io-mode=disable,defaults,_netdev 0 0 > > The server utilization (memory,cpu,network and disk 1/0) is relatively low > > I am experiencing very slow performance while copying multiple file/folders > (approx 75 MB) and it takes atleast approx 35 min. Even copy a folder (with > multiple files/subfolders) within the Gluster volume take the same time. > > However, if I do dd to check the copy speed, I get the below result. > > [root at ClientServer ~]# time sh -c "dd if=/dev/zero of=/mnt/testmount/test.tmp > bs=4k count=20000 && sync" > 20000+0 records in > 20000+0 records out > 81920000 bytes (82 MB) copied, 17.1357 s, 4.8 MB/s > > real 0m17.337s > user 0m0.031s > sys 0m0.317s > > > Anyone experience the same kind of performance issue, please let me know your > thoughts. >Hi Srikanth, Could you please provide the following information so that the reason behind slow copy can be deduced? 1) Profile info of the volume when you copy dirs/files into glusterfs. 2) Profile info of the volume when you copy dirs/files within glusterfs. The following steps should help you with profile info: 1) gluster volume profile <VOLNAME> start 2) Perform copy operations 3) gluster volume profile <VOLNAME> info (you will get stats of the FOPs at this point) 4) gluster volume profile <VOLNAME> stop Please follow steps 1 through 4 twice. Once for copy into glusterfs and once for copy within.> Cheers > Srikanth > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users-- Thanks, Anuradha.