thr3ads.net - Gluster users - [Gluster-users] Gluster - Performance issue while copying bulk files/folders [Dec 2015]

If this information is useful, please help other people find it:
Share via:

Anuradha Talur

2015-Dec-11 05:15 UTC

[Gluster-users] Gluster - Performance issue while copying bulk files/folders

Response inline.

----- Original Message -----> From: "Srikanth Mampilakal" <shrikanth1980 at gmail.com>
> To: gluster-users at gluster.org
> Sent: Thursday, December 10, 2015 7:59:04 PM
> Subject: Re: [Gluster-users] Gluster - Performance issue while copying bulk
files/folders
> 
> 
> 
> Hi members,
> 
> Really appreciate if you can share your thoughts or any feedback for
> resolving the slow copy issue
> 
> Regards
> Srikanth
> On 10-Dec-2015 2:12 AM, "Srikanth Mampilakal" <
srikanth.mampilakal at gmail.com
> > wrote:
> 
> 
> 
> Hi,
> 
> 
> I have production gluster file service used as a shared storage where the
> content management system uses it as document root. I have run in to a
> performance issue with the gluster/fuse client.
> 
> Looking for your thoughts and experience in resolving Gluster performance
> issues:
> 
> Gluster Infrastructure
> 
> Gluster version :GlusterFS 3.7.6
> 
> 2 gluster nodes of the same config below
> 
> Redhat EL7.0-64
> Memory : 4GB
> Processor : 2 x 2.0 Ghz
> Network : 100 Mbps
> File Storage Volume : NETAPP Storage LUN with 2.0 IOPS/GB
> 
> Gluster Volume information:
> 
> [root at GlusterFileServe1 ~]# gluster volume info
> 
> Volume Name: prodcmsroot
> Type: Replicate
> Volume ID: f1284bf0-1939-46f9-a672-a7716e362947
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: Server1:/glusterfs/brick1/prodcmsroot
> Brick2: Server2:/glusterfs/brick1/prodcmsroot
> Options Reconfigured:
> performance.io-thread-count: 64
> performance.cache-size: 1073741824
> performance.readdir-ahead: on
> performance.write-behind-window-size: 524288
> 
> [root at GlusterFileServe1 ~]#
> 
> The replication between Gluster node are quick and consistent.
> 
> The apache webservers are accessing the Gluster volume using native gluster
> fuse client and located in the same VLAN as the Gluster Server.
> 
> GlusterFileServe1:/prodcmsroot /mnt/glusterfs glusterfs
> direct-io-mode=disable,defaults,_netdev 0 0
> 
> The server utilization (memory,cpu,network and disk 1/0) is relatively low
> 
> I am experiencing very slow performance while copying multiple file/folders
> (approx 75 MB) and it takes atleast approx 35 min. Even copy a folder (with
> multiple files/subfolders) within the Gluster volume take the same time.
> 
> However, if I do dd to check the copy speed, I get the below result.
> 
> [root at ClientServer ~]# time sh -c "dd if=/dev/zero
of=/mnt/testmount/test.tmp
> bs=4k count=20000 && sync"
> 20000+0 records in
> 20000+0 records out
> 81920000 bytes (82 MB) copied, 17.1357 s, 4.8 MB/s
> 
> real 0m17.337s
> user 0m0.031s
> sys 0m0.317s
> 
> 
> Anyone experience the same kind of performance issue, please let me know
your
> thoughts.
> Hi Srikanth,
 
Could you please provide the following information so that the reason behind
slow copy can be deduced?

1) Profile info of the volume when you copy dirs/files into glusterfs.
2) Profile info of the volume when you copy dirs/files within glusterfs.

The following steps should help you with profile info:
1) gluster volume profile <VOLNAME> start
2) Perform copy operations
3) gluster volume profile <VOLNAME> info (you will get stats of the FOPs
at this point)
4) gluster volume profile <VOLNAME> stop

Please follow steps 1 through 4 twice. Once for copy into glusterfs and once for
copy
within.
> Cheers
> Srikanth
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
-- 
Thanks,
Anuradha.

Srikanth Mampilakal

2015-Dec-15 11:47 UTC

head link

[Gluster-users] Gluster - Performance issue while copying bulk files/folders

Hi Anuradha,

Please find the Gluster Volume Profile details

time cp -RPp drupal\ code/ /mnt/testmount/copytogluster




*Profile info of the volume when you copy dirs/files into glusterfs.*


*Time taken to copy (70 MB files/Folder)*

[root at GFSCLIENT01 temp]# time cp -RPp /mnt/testmount/
/mnt/testmount/copytogluster

real    29m40.985s
user    0m0.172s
sys     0m1.688s



[root at GFSNODE01 ~]# gluster volume profile gv1 info
Brick: GFSNODE01:/mnt/perfDisk/gv1

--------------------------------------
Cumulative Stats:
   Block Size:                 16b+                  32b+
 64b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   19                    11
 75

   Block Size:                128b+                 256b+
512b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   77                   221
297

   Block Size:               1024b+                2048b+
 4096b+
 No. of Reads:                    0                     0
  0
No. of Writes:                  344                   305
336

   Block Size:               8192b+               16384b+
32768b+
 No. of Reads:                    0                     0
  0
No. of Writes:                  160                   200
 87

   Block Size:              65536b+              131072b+
 No. of Reads:                    0                     0
No. of Writes:                   59                    38
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us           2198
RELEASE
      0.00       0.00 us       0.00 us       0.00 us             18
 RELEASEDIR
      0.00      39.75 us      22.00 us      59.00 us              4
READDIR
      0.01      63.12 us       3.00 us     143.00 us              8
OPENDIR
      0.01     108.83 us      27.00 us     194.00 us              6
 GETXATTR
      0.11      58.07 us      28.00 us     124.00 us            170
 STAT
      0.54     113.57 us      46.00 us     258.00 us            440
 SETXATTR
      0.79      97.28 us      23.00 us     224.00 us            745
 STATFS
      1.37      57.40 us      12.00 us     428.00 us           2198
FLUSH
      3.70      77.12 us      15.00 us     322.00 us           4420
 FINODELK
      3.94      68.70 us      14.00 us     259.00 us           5278
ENTRYLK
      4.98     205.68 us      70.00 us    2874.00 us           2229
WRITE
      5.15    1077.38 us     202.00 us  112584.00 us            440
MKDIR
      5.27     110.26 us      33.00 us    5589.00 us           4397
REMOVEXATTR
      7.88     118.30 us      28.00 us   11471.00 us           6130
SETATTR
      9.23     190.97 us      33.00 us  107884.00 us           4450
 FXATTROP
     16.06     672.52 us     112.00 us  177035.00 us           2199
 CREATE
     20.24      80.67 us      11.00 us     454.00 us          23102
INODELK
     20.74     160.46 us      24.00 us   33476.00 us          11901
 LOOKUP

    Duration: 3007 seconds
   Data Read: 0 bytes
Data Written: 24173066 bytes

-----------------------------------------------------------------------------------------------

*Profile info of the volume when you copy dirs/files within glusterfs.*

Time taken to copy (70 MB files/folders)

[root at GFSCLIENT01 testmount]# time cp -RPp copytogluster/data/
/mnt/testmount/copywithinglustervol/

real    37m50.407s
user    0m0.248s
sys     0m1.979s

[root at GFSNODE01 ~]# gluster volume profile gv1 info
Brick: GFSNODE01:/mnt/perfDisk/gv1


Interval 8 Stats:
   Block Size:                 64b+                 128b+
256b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   11                     5
 11

   Block Size:                512b+                1024b+
 2048b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   14                    18
 13

   Block Size:               4096b+                8192b+
16384b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   11                     6
  3

   Block Size:              32768b+               65536b+
 No. of Reads:                    0                     1
No. of Writes:                    1                     1
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us             94
RELEASE
      0.00       0.00 us       0.00 us       0.00 us              4
 RELEASEDIR
      0.03      37.00 us      37.00 us      37.00 us              1
FSTAT
      0.04      22.50 us      12.00 us      33.00 us              2
READDIR
      0.05      20.67 us      17.00 us      28.00 us              3
 STAT
      0.06      77.00 us      77.00 us      77.00 us              1
 READ
      0.07      41.50 us      14.00 us      69.00 us              2
 GETXATTR
      0.10      30.25 us       2.00 us      53.00 us              4
OPENDIR
      0.13      51.67 us      42.00 us      60.00 us              3
 SETXATTR
      0.35     139.67 us     127.00 us     152.00 us              3
MKDIR
      0.59      25.46 us      16.00 us      40.00 us             28
 STATFS
      1.48      18.90 us      11.00 us      38.00 us             94
FLUSH
      3.87      24.00 us      13.00 us      43.00 us            194
ENTRYLK
      4.00      25.57 us      16.00 us      55.00 us            188
 FINODELK
      5.65      72.37 us      57.00 us     177.00 us             94
WRITE
      7.38      47.22 us      38.00 us      60.00 us            188
REMOVEXATTR
      8.28      50.55 us      27.00 us     108.00 us            197
SETATTR
      8.50      54.41 us      37.00 us     112.00 us            188
 FXATTROP
     13.04     166.86 us      78.00 us    1050.00 us             94
 CREATE
     17.43      26.27 us      11.00 us      85.00 us            798
INODELK
     28.95      68.04 us      14.00 us     233.00 us            512
 LOOKUP

    Duration: 29 seconds
   Data Read: 78602 bytes
Data Written: 365315 bytes

Interval 28 Stats:
   Block Size:                  4b+                  32b+
 64b+
 No. of Reads:                    0                     0
  0
No. of Writes:                    1                     5
 28

   Block Size:                128b+                 256b+
512b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   59                   164
305

   Block Size:               1024b+                2048b+
 4096b+
 No. of Reads:                    0                     0
  1
No. of Writes:                  232                   171
165

   Block Size:               8192b+               16384b+
32768b+
 No. of Reads:                    0                     0
  2
No. of Writes:                  117                    96
 39

   Block Size:              65536b+              131072b+
 No. of Reads:                   11                     8
No. of Writes:                   24                    22
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us           1406
RELEASE
      0.00       0.00 us       0.00 us       0.00 us            265
 RELEASEDIR
      0.00      22.50 us      13.00 us      32.00 us              2
READDIR
      0.00      30.50 us      19.00 us      42.00 us              2
 GETXATTR
      0.02      35.12 us      28.00 us      45.00 us             16
FSTAT
      0.06      75.86 us      27.00 us     232.00 us             22
 READ
      0.08      21.99 us      15.00 us      37.00 us            102
 STAT
      0.31      32.43 us       2.00 us      55.00 us            265
OPENDIR
      0.43      44.88 us      33.00 us      69.00 us            264
 SETXATTR
      0.54      24.82 us      15.00 us      91.00 us            598
 STATFS
      0.84      16.54 us      10.00 us      74.00 us           1406
FLUSH
      1.89     199.11 us      99.00 us    6678.00 us            263
MKDIR
      2.39      23.27 us      14.00 us      91.00 us           2840
 FINODELK
      2.58      21.39 us      11.00 us      62.00 us           3339
ENTRYLK
      2.83     288.06 us      25.00 us    2315.00 us            272
 READDIRP
      3.93      76.22 us      54.00 us     355.00 us           1428
WRITE
      6.05      59.59 us      36.00 us   42015.00 us           2812
REMOVEXATTR
      6.34      61.60 us      24.00 us   28264.00 us           2850
 FXATTROP
      6.40      45.78 us      25.00 us     146.00 us           3867
SETATTR
     10.96     215.75 us      79.00 us   14633.00 us           1406
 CREATE
     12.49      23.78 us      10.00 us     100.00 us          14545
INODELK
     41.85      71.82 us      14.00 us   55712.00 us          16131
 LOOKUP

    Duration: 598 seconds
   Data Read: 2150643 bytes
Data Written: 12210039 bytes

Do let me know if you need any other details

Thanks
Srikanth


On Fri, Dec 11, 2015 at 4:15 PM, Anuradha Talur <atalur at redhat.com>
wrote:
> Response inline.
>
> ----- Original Message -----
> > From: "Srikanth Mampilakal" <shrikanth1980 at
gmail.com>
> > To: gluster-users at gluster.org
> > Sent: Thursday, December 10, 2015 7:59:04 PM
> > Subject: Re: [Gluster-users] Gluster - Performance issue while copying
> bulk   files/folders
> >
> >
> >
> > Hi members,
> >
> > Really appreciate if you can share your thoughts or any feedback for
> > resolving the slow copy issue
> >
> > Regards
> > Srikanth
> > On 10-Dec-2015 2:12 AM, "Srikanth Mampilakal" <
> srikanth.mampilakal at gmail.com
> > > wrote:
> >
> >
> >
> > Hi,
> >
> >
> > I have production gluster file service used as a shared storage where
the
> > content management system uses it as document root. I have run in to a
> > performance issue with the gluster/fuse client.
> >
> > Looking for your thoughts and experience in resolving Gluster
performance
> > issues:
> >
> > Gluster Infrastructure
> >
> > Gluster version :GlusterFS 3.7.6
> >
> > 2 gluster nodes of the same config below
> >
> > Redhat EL7.0-64
> > Memory : 4GB
> > Processor : 2 x 2.0 Ghz
> > Network : 100 Mbps
> > File Storage Volume : NETAPP Storage LUN with 2.0 IOPS/GB
> >
> > Gluster Volume information:
> >
> > [root at GlusterFileServe1 ~]# gluster volume info
> >
> > Volume Name: prodcmsroot
> > Type: Replicate
> > Volume ID: f1284bf0-1939-46f9-a672-a7716e362947
> > Status: Started
> > Number of Bricks: 1 x 2 = 2
> > Transport-type: tcp
> > Bricks:
> > Brick1: Server1:/glusterfs/brick1/prodcmsroot
> > Brick2: Server2:/glusterfs/brick1/prodcmsroot
> > Options Reconfigured:
> > performance.io-thread-count: 64
> > performance.cache-size: 1073741824
> > performance.readdir-ahead: on
> > performance.write-behind-window-size: 524288
> >
> > [root at GlusterFileServe1 ~]#
> >
> > The replication between Gluster node are quick and consistent.
> >
> > The apache webservers are accessing the Gluster volume using native
> gluster
> > fuse client and located in the same VLAN as the Gluster Server.
> >
> > GlusterFileServe1:/prodcmsroot /mnt/glusterfs glusterfs
> > direct-io-mode=disable,defaults,_netdev 0 0
> >
> > The server utilization (memory,cpu,network and disk 1/0) is relatively
> low
> >
> > I am experiencing very slow performance while copying multiple
> file/folders
> > (approx 75 MB) and it takes atleast approx 35 min. Even copy a folder
> (with
> > multiple files/subfolders) within the Gluster volume take the same
time.
> >
> > However, if I do dd to check the copy speed, I get the below result.
> >
> > [root at ClientServer ~]# time sh -c "dd if=/dev/zero
> of=/mnt/testmount/test.tmp
> > bs=4k count=20000 && sync"
> > 20000+0 records in
> > 20000+0 records out
> > 81920000 bytes (82 MB) copied, 17.1357 s, 4.8 MB/s
> >
> > real 0m17.337s
> > user 0m0.031s
> > sys 0m0.317s
> >
> >
> > Anyone experience the same kind of performance issue, please let me
know
> your
> > thoughts.
> >
> Hi Srikanth,
>
> Could you please provide the following information so that the reason
> behind
> slow copy can be deduced?
>
> 1) Profile info of the volume when you copy dirs/files into glusterfs.
> 2) Profile info of the volume when you copy dirs/files within glusterfs.
>
> The following steps should help you with profile info:
> 1) gluster volume profile <VOLNAME> start
> 2) Perform copy operations
> 3) gluster volume profile <VOLNAME> info (you will get stats of the
FOPs
> at this point)
> 4) gluster volume profile <VOLNAME> stop
>
> Please follow steps 1 through 4 twice. Once for copy into glusterfs and
> once for copy
> within.
>
> > Cheers
> > Srikanth
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
>
> --
> Thanks,
> Anuradha.
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>


-- 
Cheers
Shrikanth
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20151215/ebb2d04d/attachment.html>

Gluster users - Dec 2015 - Gluster - Performance issue while copying bulk files/folders

[Gluster-users] Gluster - Performance issue while copying bulk files/folders

[Gluster-users] Gluster - Performance issue while copying bulk files/folders