Paul E Stallworth
2015-Jan-27 17:59 UTC
[Gluster-users] High latency on FSYNC, INODELK, FINODELK
Hello, I ran a gluster volume profile while doing a dd to read from /dev/zero and write to a file and noticed unusually high latency on the FSYNC, INODELK, and FINODELK operations. This latency seems to correspond with very slow page loads of our website. I'm looking for any help in finding where this latency might be coming from and how we might mitigate it. Here are the values from the profile %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 110601 FORGET 0.00 0.00 us 0.00 us 0.00 us 104872 RELEASE 0.00 0.00 us 0.00 us 0.00 us 269678 RELEASEDIR 0.00 50.00 us 45.00 us 55.00 us 2 OPEN 0.00 73.50 us 32.00 us 115.00 us 2 FLUSH 0.00 47.00 us 33.00 us 57.00 us 4 ENTRYLK 0.00 226.00 us 226.00 us 226.00 us 1 CREATE 0.00 83.75 us 42.00 us 136.00 us 4 STATFS 0.00 72.80 us 23.00 us 175.00 us 5 READDIR 0.00 54.90 us 15.00 us 97.00 us 10 GETXATTR 0.00 79.19 us 26.00 us 193.00 us 32 FXATTROP 0.00 45.36 us 14.00 us 271.00 us 206 STAT 0.01 66.26 us 2.00 us 586.00 us 352 OPENDIR 0.09 252262.00 us 252262.00 us 252262.00 us 1 UNLINK 1.02 139.54 us 29.00 us 10777.00 us 20263 WRITE 7.98 1843032.83 us 22261.00 us 16023044.00 us 12 FSYNC 8.19 1620932.64 us 27.00 us 17171453.00 us 14 INODELK 9.03 94.52 us 22.00 us 9533.00 us 264526 LOOKUP 73.67 563743.05 us 14.00 us 17173239.00 us 362 FINODELK Thanks, Paul Paul Stallworth Housing Information Technology University of Colorado Boulder Boulder, Colorado 80309 T 303.735.6623 M 303.735.6623
Pranith Kumar Karampuri
2015-Jan-28 09:16 UTC
[Gluster-users] High latency on FSYNC, INODELK, FINODELK
On 01/27/2015 11:29 PM, Paul E Stallworth wrote:> Hello, > > I ran a gluster volume profile while doing a dd to read from /dev/zero and write to a file and noticed unusually high latency on the FSYNC, INODELK, and FINODELK operations. This latency seems to correspond with very slow page loads of our website. I'm looking for any help in finding where this latency might be coming from and how we might mitigate it.FSYNC shouldn't happen in the workload you specified. Are you sure this is the only workload on the volume when you executed 'dd'? What is the version of gluster you are using? Pranith> > Here are the values from the profile > > %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop > --------- ----------- ----------- ----------- ------------ ---- > 0.00 0.00 us 0.00 us 0.00 us 110601 FORGET > 0.00 0.00 us 0.00 us 0.00 us 104872 RELEASE > 0.00 0.00 us 0.00 us 0.00 us 269678 RELEASEDIR > 0.00 50.00 us 45.00 us 55.00 us 2 OPEN > 0.00 73.50 us 32.00 us 115.00 us 2 FLUSH > 0.00 47.00 us 33.00 us 57.00 us 4 ENTRYLK > 0.00 226.00 us 226.00 us 226.00 us 1 CREATE > 0.00 83.75 us 42.00 us 136.00 us 4 STATFS > 0.00 72.80 us 23.00 us 175.00 us 5 READDIR > 0.00 54.90 us 15.00 us 97.00 us 10 GETXATTR > 0.00 79.19 us 26.00 us 193.00 us 32 FXATTROP > 0.00 45.36 us 14.00 us 271.00 us 206 STAT > 0.01 66.26 us 2.00 us 586.00 us 352 OPENDIR > 0.09 252262.00 us 252262.00 us 252262.00 us 1 UNLINK > 1.02 139.54 us 29.00 us 10777.00 us 20263 WRITE > 7.98 1843032.83 us 22261.00 us 16023044.00 us 12 FSYNC > 8.19 1620932.64 us 27.00 us 17171453.00 us 14 INODELK > 9.03 94.52 us 22.00 us 9533.00 us 264526 LOOKUP > 73.67 563743.05 us 14.00 us 17173239.00 us 362 FINODELK > > Thanks, > Paul > > Paul Stallworth > Housing Information Technology > University of Colorado Boulder > Boulder, Colorado 80309 > T 303.735.6623 > M 303.735.6623 > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users