thr3ads.net - Gluster users - [Gluster-users] High latency on FSYNC, INODELK, FINODELK [Jan 2015]

If this information is useful, please help other people find it:
Share via:

Pranith Kumar Karampuri

2015-Jan-28 09:16 UTC

[Gluster-users] High latency on FSYNC, INODELK, FINODELK

On 01/27/2015 11:29 PM, Paul E Stallworth wrote:> Hello,
>
> I ran a gluster volume profile while doing a dd to read from /dev/zero and
write to a file and noticed unusually high latency on the FSYNC, INODELK, and
FINODELK operations.  This latency seems to correspond with very slow page loads
of our website.  I'm looking for any help in finding where this latency
might be coming from and how we might mitigate it.FSYNC shouldn't happen in the workload you specified. Are you sure this 
is the only workload on the volume when you executed 'dd'? What is the 
version of gluster you are using?

Pranith>
> Here are the values from the profile
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 110601 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 104872 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 269678 RELEASEDIR
> 0.00 50.00 us 45.00 us 55.00 us 2 OPEN
> 0.00 73.50 us 32.00 us 115.00 us 2 FLUSH
> 0.00 47.00 us 33.00 us 57.00 us 4 ENTRYLK
> 0.00 226.00 us 226.00 us 226.00 us 1 CREATE
> 0.00 83.75 us 42.00 us 136.00 us 4 STATFS
> 0.00 72.80 us 23.00 us 175.00 us 5 READDIR
> 0.00 54.90 us 15.00 us 97.00 us 10 GETXATTR
> 0.00 79.19 us 26.00 us 193.00 us 32 FXATTROP
> 0.00 45.36 us 14.00 us 271.00 us 206 STAT
> 0.01 66.26 us 2.00 us 586.00 us 352 OPENDIR
> 0.09 252262.00 us 252262.00 us 252262.00 us 1 UNLINK
> 1.02 139.54 us 29.00 us 10777.00 us 20263 WRITE
> 7.98 1843032.83 us 22261.00 us 16023044.00 us 12 FSYNC
> 8.19 1620932.64 us 27.00 us 17171453.00 us 14 INODELK
> 9.03 94.52 us 22.00 us 9533.00 us 264526 LOOKUP
> 73.67 563743.05 us 14.00 us 17173239.00 us 362 FINODELK
>
> Thanks,
> Paul
>
> Paul Stallworth
> Housing Information Technology
> University of Colorado Boulder
> Boulder, Colorado 80309
> T 303.735.6623
> M 303.735.6623
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

Paul E Stallworth

2015-Jan-28 15:54 UTC

head link

[Gluster-users] High latency on FSYNC, INODELK, FINODELK

We are running version 3.6.1 on Ubuntu 12.04.5 LTS.  The file system for the
volume is ext4 and is mounted on clients as glusterfs.

I verified apache wasn't running, and there shouldn't be any other
workload on the web server and ran another test using the following:
"dd if=/dev/zero of=/web/ddtest bs=8k count=250000 conv=fdatasync"
with the profiler running.

The dd operation took 38 seconds and the two big results from the profiler for
this particular brick were:
     
42.79     202.15 us      27.00 us 1008658.00 us          27649       WRITE
56.14 7332621.00 us 7332621.00 us 7332621.00 us              1       FSYNC

I repeated the test and the results for this brick are similar: 

29.60     176.56 us      27.00 us 1008658.00 us          55273       WRITE
62.41 1714773.67 us   10098.00 us 10996032.00 us             12       FSYNC

Thanks,
Paul

Paul Stallworth
Housing Information Technology
University of Colorado Boulder
Boulder, Colorado 80309
T 303.735.6623
M 303.735.6623



-----Original Message-----
From: Pranith Kumar Karampuri [mailto:pkarampu at redhat.com] 
Sent: Wednesday, January 28, 2015 2:16 AM
To: Paul E Stallworth; gluster-users at gluster.org
Subject: Re: [Gluster-users] High latency on FSYNC, INODELK, FINODELK


On 01/27/2015 11:29 PM, Paul E Stallworth wrote:> Hello,
>
> I ran a gluster volume profile while doing a dd to read from /dev/zero and
write to a file and noticed unusually high latency on the FSYNC, INODELK, and
FINODELK operations.  This latency seems to correspond with very slow page loads
of our website.  I'm looking for any help in finding where this latency
might be coming from and how we might mitigate it.FSYNC shouldn't happen in the workload you specified. Are you sure this is
the only workload on the volume when you executed 'dd'? What is the
version of gluster you are using?

Pranith>
> Here are the values from the profile
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 110601 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 104872 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 269678 RELEASEDIR
> 0.00 50.00 us 45.00 us 55.00 us 2 OPEN
> 0.00 73.50 us 32.00 us 115.00 us 2 FLUSH
> 0.00 47.00 us 33.00 us 57.00 us 4 ENTRYLK
> 0.00 226.00 us 226.00 us 226.00 us 1 CREATE
> 0.00 83.75 us 42.00 us 136.00 us 4 STATFS
> 0.00 72.80 us 23.00 us 175.00 us 5 READDIR
> 0.00 54.90 us 15.00 us 97.00 us 10 GETXATTR
> 0.00 79.19 us 26.00 us 193.00 us 32 FXATTROP
> 0.00 45.36 us 14.00 us 271.00 us 206 STAT
> 0.01 66.26 us 2.00 us 586.00 us 352 OPENDIR
> 0.09 252262.00 us 252262.00 us 252262.00 us 1 UNLINK
> 1.02 139.54 us 29.00 us 10777.00 us 20263 WRITE
> 7.98 1843032.83 us 22261.00 us 16023044.00 us 12 FSYNC
> 8.19 1620932.64 us 27.00 us 17171453.00 us 14 INODELK
> 9.03 94.52 us 22.00 us 9533.00 us 264526 LOOKUP
> 73.67 563743.05 us 14.00 us 17173239.00 us 362 FINODELK
>
> Thanks,
> Paul
>
> Paul Stallworth
> Housing Information Technology
> University of Colorado Boulder
> Boulder, Colorado 80309
> T 303.735.6623
> M 303.735.6623
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

Gluster users - Jan 2015 - High latency on FSYNC, INODELK, FINODELK

[Gluster-users] High latency on FSYNC, INODELK, FINODELK

[Gluster-users] High latency on FSYNC, INODELK, FINODELK