Kyle Harris
2014-Dec-22 05:00 UTC
[Gluster-users] glusterfs and glusterfsd process utilization
Hi Krutika,
Thank you so much for the response. I'm starting to get a bit desperate :-)
I'm not sure which file is the brick log but the client log (I assume that
is the one that start with mnt-) is far too large to include. If you can
tell me what it is, perhaps we can arrange to get it to you using some
other method?
Anyway, I primarily see the following messages in what I believe to be the
client log.
The first is as mentioned earlier and repeated again here:
[2014-12-21 05:33:49.873989] I [dict.c:370:dict_get] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7fe3bd0d2420] (-->
/usr/lib64/libglusterfs.so.0(dict_get+0x4f)[0x7fe3bd0cbbdf] (-->
/usr/lib64/glusterfs/3.6.1/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x1e1)[0x7fe3aad3ada1]
(-->
/usr/lib64/glusterfs/3.6.1/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x113)[0x7fe3aaf524e3]
(-->
/usr/lib64/glusterfs/3.6.1/xlator/performance/md-cache.so(mdc_lookup+0x300)[0x7fe3ab167310]
))))) 0-dict: !this || key=system.posix_acl_access
This seems to be the one I see the most when it is starting up and having
this issue but I can't be certain. By the way, can you tell me what that
means?
The second entry I see often in the log files is in regards to missing
files such as follows:
[2014-12-22 02:33:32.062741] I [dht-common.c:1822:dht_lookup_cbk]
0-gv0-dht: Entry /html/some_dir/some_file/asdf.php missing on subvol
gv0-replicate-0
I thought these might be causing my problems but I see them all the time,
including right now and process utilization on the servers in this cluster
is currently way down next to nothing so I no longer think that is my
problem. I also checked while to see what, if any files needed healing
during the last time this issue occurred using the following command:
gluster volume heal gv0 info
It showed no files in need of healing.
In the log file etc-glusterfs-glusterd.vol.log I see many of the following
entries:
[2014-12-21 19:13:54.137263] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/095271447d0b71369a479e6211bbe6e5.socket failed (Invalid
argument)
I have a bricks dir with the file export-brick1.log and it is empty.
gluster volume info as requested:
Volume Name: gv0
Type: Replicate
Volume ID: 3565c78b-2d23-42e7-8cd2-1724fe671610
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 192.168.10.90:/export/brick1
Brick2: 192.168.10.91:/export/brick1
Brick3: 192.168.10.92:/export/brick1
Options Reconfigured:
cluster.data-self-heal-algorithm: full
cluster.entry-self-heal: off
I also found the following in a log file and perhaps it will help:
Final graph:
+------------------------------------------------------------------------------+
1: volume gv0-client-0
2: type protocol/client
3: option ping-timeout 42
4: option remote-host 192.168.10.90
5: option remote-subvolume /export/brick1
6: option transport-type socket
7: option username {removed}
8: option password {removed}
9: option send-gids true
10: end-volume
11:
12: volume gv0-client-1
13: type protocol/client
14: option ping-timeout 42
15: option remote-host 192.168.10.91
16: option remote-subvolume /export/brick1
17: option transport-type socket
18: option username {removed}
19: option password {removed}
20: option send-gids true
21: end-volume
22:
23: volume gv0-client-2
24: type protocol/client
25: option ping-timeout 42
26: option remote-host 192.168.10.92
27: option remote-subvolume /export/brick1
28: option transport-type socket
29: option username {removed}
30: option password {removed}
31: option send-gids true
32: end-volume
33:
34: volume gv0-replicate-0
35: type cluster/replicate
36: option entry-self-heal off
37: option data-self-heal-algorithm full
38: subvolumes gv0-client-0 gv0-client-1 gv0-client-2
39: end-volume
40:
41: volume gv0-dht
42: type cluster/distribute
43: subvolumes gv0-replicate-0
44: end-volume
45:
46: volume gv0-write-behind
47: type performance/write-behind
48: subvolumes gv0-dht
49: end-volume
50:
51: volume gv0
52: type debug/io-stats
53: option latency-measurement off
54: option count-fop-hits off
55: subvolumes gv0-write-behind
56: end-volume
57:
58: volume nfs-server
59: type nfs/server
60: option rpc-auth.auth-glusterfs on
61: option rpc-auth.auth-unix on
62: option rpc-auth.auth-null on
63: option rpc-auth.ports.insecure on
64: option rpc-auth-allow-insecure on
65: option transport-type socket
66: option transport.socket.listen-port 2049
67: option nfs.dynamic-volumes on
68: option nfs.nlm on
69: option nfs.drc off
70: option rpc-auth.addr.gv0.allow *
71: option nfs3.gv0.volume-id 3565c78b-2d23-42e7-8cd2-1724fe671610
72: subvolumes gv0
73: end-volume
74:
With regard to the average number of directory entries, I don't know for
sure but I'm sure some of them get pretty deep.
Finally, I do not have quota enabled.
Again, thank you very much and regards,
Kyle
On Sun, Dec 21, 2014 at 10:19 PM, Krutika Dhananjay <kdhananj at
redhat.com>
wrote:
> Hi,
>
> Could you provide the following pieces of information?
>
> 1. The brick and client logs
> 2. output of `gluster volume info`
> 3. average number of entries under directories
>
> Just to be clear, you don't have quota enabled on the volume, do you?
>
> -Krutika
> ------------------------------
>
> *From: *"Kyle Harris" <kyle.harris98 at gmail.com>
> *To: *gluster-users at gluster.org
> *Sent: *Monday, December 22, 2014 8:27:09 AM
> *Subject: *Re: [Gluster-users] glusterfs and glusterfsd process
> utilization
>
> I haven't heard anything from anyone re this post yet so I have a bit
more
> info to add. The last time I posted about extremely high process
> utilization for glusterfs and glusterfsd, I was asked to see if I could
> capture the output of a profile. I did so and here is the output of 2
> captures (they appear very similar). Perhaps this will assist. Note that
> previously I was mounting with the fuse client. This time I tried NFS.
>
> Profile Output 1:
> Brick: 192.168.10.92:/export/brick1
> -----------------------------------
> Cumulative Stats:
> Block Size: 1b+ 2b+
> 4b+
> No. of Reads: 0 0
> 0
> No. of Writes: 133 6
> 37
>
> Block Size: 8b+ 16b+
> 32b+
> No. of Reads: 28 93
> 31
> No. of Writes: 4576 6494
> 22228
>
> Block Size: 64b+ 128b+
> 256b+
> No. of Reads: 217 216
> 254
> No. of Writes: 37627 41711
> 20519
>
> Block Size: 512b+ 1024b+
> 2048b+
> No. of Reads: 484 978
> 684
> No. of Writes: 39094 42589
> 203747
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 1073 1800
> 2838
> No. of Writes: 220154 229725
> 215083
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 4224 14941
> 57812
> No. of Writes: 60938 61967
> 793400
>
> Block Size: 262144b+ 524288b+
> 1048576b+
> No. of Reads: 4 4
> 2
> No. of Writes: 5 0
> 0
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 1835064
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 2639321
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 1439759
> RELEASEDIR
> 0.00 153.50 us 142.00 us 165.00 us 2
> XATTROP
> 0.00 102.67 us 97.00 us 110.00 us 3
> SETATTR
> 0.00 93.20 us 53.00 us 129.00 us 5
> GETXATTR
> 0.00 55.07 us 33.00 us 100.00 us 15
> INODELK
> 0.00 46.61 us 35.00 us 69.00 us 18
> STATFS
> 0.01 48.40 us 31.00 us 64.00 us 25
> LK
> 0.01 50.72 us 32.00 us 88.00 us 47
> FINODELK
> 0.01 35.49 us 17.00 us 68.00 us 86
> FLUSH
> 0.02 50.59 us 27.00 us 76.00 us 68
> ENTRYLK
> 0.03 75.72 us 47.00 us 112.00 us 74
> OPEN
> 0.06 58.79 us 32.00 us 100.00 us 222
> OPENDIR
> 0.08 1535.92 us 129.00 us 16583.00 us 12
> UNLINK
> 0.11 24814.00 us 24814.00 us 24814.00 us 1
> TRUNCATE
> 0.12 34.25 us 11.00 us 1950.00 us 784
> ACCESS
> 0.13 603.50 us 90.00 us 21829.00 us 46
> FXATTROP
> 0.16 7225.40 us 63.00 us 14563.00 us 5
> READLINK
> 0.17 437.90 us 54.00 us 19042.00 us 83
> READ
> 0.36 85.92 us 56.00 us 313.00 us 912
> FSTAT
> 0.42 59.86 us 22.00 us 1443.00 us 1533
> STAT
> 0.45 4318.65 us 70.00 us 56839.00 us 23
> WRITE
> 0.84 8405.18 us 161.00 us 37688.00 us 22
> CREATE
> 8.33 2005.77 us 361.00 us 2796.00 us 912
> READDIRP
> 88.68 921.10 us 34.00 us 159393.00 us 21137
> LOOKUP
>
> Duration: 338151 seconds
> Data Read: 9290348715 bytes
> Data Written: 122127653133 bytes
>
> Interval 1 Stats:
> Block Size: 8b+ 32b+
> 64b+
> No. of Reads: 2 0
> 1
> No. of Writes: 0 1
> 4
>
> Block Size: 128b+ 256b+
> 512b+
> No. of Reads: 1 2
> 5
> No. of Writes: 3 0
> 0
>
> Block Size: 1024b+ 2048b+
> 4096b+
> No. of Reads: 7 13
> 11
> No. of Writes: 6 0
> 0
>
> Block Size: 8192b+ 16384b+
> 32768b+
> No. of Reads: 10 2
> 9
> No. of Writes: 0 0
> 0
>
> Block Size: 65536b+ 131072b+
> 262144b+
> No. of Reads: 1 1
> 1
> No. of Writes: 0 0
> 0
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 7
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 45
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 177
> RELEASEDIR
> 0.00 109.00 us 109.00 us 109.00 us 1
> GETXATTR
> 0.00 110.00 us 110.00 us 110.00 us 1
> SETATTR
> 0.00 52.78 us 33.00 us 81.00 us 9
> INODELK
> 0.00 48.92 us 36.00 us 69.00 us 12
> STATFS
> 0.01 146.80 us 129.00 us 171.00 us 5
> UNLINK
> 0.01 47.89 us 31.00 us 63.00 us 18
> LK
> 0.01 34.25 us 17.00 us 61.00 us 40
> FLUSH
> 0.01 49.89 us 35.00 us 64.00 us 28
> FINODELK
> 0.02 49.12 us 27.00 us 76.00 us 42
> ENTRYLK
> 0.02 78.14 us 47.00 us 99.00 us 29
> OPEN
> 0.08 58.01 us 34.00 us 100.00 us 177
> OPENDIR
> 0.13 241.08 us 54.00 us 10214.00 us 66
> READ
> 0.18 34.76 us 11.00 us 1950.00 us 649
> ACCESS
> 0.21 906.82 us 94.00 us 21829.00 us 28
> FXATTROP
> 0.29 9014.75 us 63.00 us 14563.00 us 4
> READLINK
> 0.32 85.78 us 56.00 us 313.00 us 455
> FSTAT
> 0.59 60.01 us 22.00 us 1443.00 us 1215
> STAT
> 0.67 5906.21 us 70.00 us 56839.00 us 14
> WRITE
> 1.02 7854.81 us 161.00 us 27303.00 us 16
> CREATE
> 7.42 2008.99 us 361.00 us 2796.00 us 455
> READDIRP
> 89.00 1052.36 us 37.00 us 132924.00 us 10418
> LOOKUP
>
> Duration: 18 seconds
> Data Read: 1285736 bytes
> Data Written: 8465 bytes
>
> Brick: 192.168.10.90:/export/brick1
> -----------------------------------
> Cumulative Stats:
> Block Size: 2b+ 8b+
> 16b+
> No. of Reads: 0 6
> 32
> No. of Writes: 1 1928
> 2604
>
> Block Size: 32b+ 64b+
> 128b+
> No. of Reads: 21 112
> 169
> No. of Writes: 6983 12066
> 26008
>
> Block Size: 256b+ 512b+
> 1024b+
> No. of Reads: 85 299
> 809
> No. of Writes: 743 18584
> 7209
>
> Block Size: 2048b+ 4096b+
> 8192b+
> No. of Reads: 790 1418
> 2884
> No. of Writes: 1650 10647
> 16901
>
> Block Size: 16384b+ 32768b+
> 65536b+
> No. of Reads: 4588 5856
> 32428
> No. of Writes: 2974 5758
> 3387
>
> Block Size: 131072b+
> No. of Reads: 455199
> No. of Writes: 4601
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 604780
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 1410794
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 989700
> RELEASEDIR
> 0.00 45.50 us 29.00 us 62.00 us 2
> READDIR
> 0.00 58.00 us 44.00 us 77.00 us 4
> GETXATTR
> 0.00 92.00 us 86.00 us 101.00 us 3
> SETATTR
> 0.00 147.50 us 136.00 us 159.00 us 2
> XATTROP
> 0.00 55.89 us 41.00 us 84.00 us 9
> FSTAT
> 0.00 54.22 us 33.00 us 76.00 us 18
> STATFS
> 0.00 46.04 us 30.00 us 60.00 us 25
> LK
> 0.01 172.17 us 115.00 us 222.00 us 12
> UNLINK
> 0.01 38.75 us 15.00 us 150.00 us 56
> FLUSH
> 0.01 54.46 us 30.00 us 101.00 us 46
> FINODELK
> 0.01 52.78 us 27.00 us 91.00 us 68
> ENTRYLK
> 0.01 93.74 us 49.00 us 352.00 us 43
> OPEN
> 0.01 204.50 us 150.00 us 511.00 us 22
> CREATE
> 0.03 58.25 us 3.00 us 94.00 us 183
> OPENDIR
> 0.03 247.33 us 73.00 us 5551.00 us 45
> FXATTROP
> 0.11 35786.00 us 35786.00 us 35786.00 us 1
> TRUNCATE
> 0.15 669.42 us 41.00 us 23631.00 us 76
> READLINK
> 0.17 3501.75 us 60.00 us 27637.00 us 16
> READ
> 0.34 5093.86 us 60.00 us 69136.00 us 22
> WRITE
> 4.56 100248.87 us 25.00 us 1502989.00 us 15
> INODELK
> 9.64 1930.07 us 30.00 us 3404.00 us 1646
> READDIRP
> 84.89 1367.04 us 34.00 us 828308.00 us 20463
> LOOKUP
>
> Duration: 76033 seconds
> Data Read: 63112532714 bytes
> Data Written: 1489661828 bytes
>
> Interval 1 Stats:
> Block Size: 32b+ 64b+
> 128b+
> No. of Reads: 0 0
> 0
> No. of Writes: 1 4
> 3
>
> Block Size: 1024b+ 65536b+
> 131072b+
> No. of Reads: 0 2
> 1
> No. of Writes: 6 0
> 0
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 9
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 45
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 177
> RELEASEDIR
> 0.00 86.00 us 86.00 us 86.00 us 1
> SETATTR
> 0.00 55.89 us 41.00 us 84.00 us 9
> FSTAT
> 0.00 53.50 us 40.00 us 76.00 us 12
> STATFS
> 0.00 160.00 us 130.00 us 184.00 us 5
> UNLINK
> 0.00 46.83 us 30.00 us 60.00 us 18
> LK
> 0.01 51.11 us 30.00 us 88.00 us 28
> FINODELK
> 0.01 37.88 us 15.00 us 150.00 us 40
> FLUSH
> 0.01 52.14 us 27.00 us 83.00 us 42
> ENTRYLK
> 0.01 101.34 us 52.00 us 352.00 us 29
> OPEN
> 0.02 209.44 us 150.00 us 511.00 us 16
> CREATE
> 0.04 319.57 us 73.00 us 5551.00 us 28
> FXATTROP
> 0.05 58.58 us 29.00 us 94.00 us 177
> OPENDIR
> 0.23 669.42 us 41.00 us 23631.00 us 76
> READLINK
> 0.26 3728.27 us 60.00 us 27637.00 us 15
> READ
> 0.37 5742.93 us 60.00 us 69136.00 us 14
> WRITE
> 6.88 167043.89 us 25.00 us 1502989.00 us 9
> INODELK
> 10.95 1908.32 us 30.00 us 3404.00 us 1254
> READDIRP
> 81.16 1701.62 us 34.00 us 828308.00 us 10421
> LOOKUP
>
> Duration: 18 seconds
> Data Read: 310621 bytes
> Data Written: 8465 bytes
>
> Brick: 192.168.10.91:/export/brick1
> -----------------------------------
> Cumulative Stats:
> Block Size: 1b+ 2b+
> 4b+
> No. of Reads: 0 0
> 0
> No. of Writes: 3 3
> 6
>
> Block Size: 8b+ 16b+
> 32b+
> No. of Reads: 13 173
> 58
> No. of Writes: 3370 5495
> 13458
>
> Block Size: 64b+ 128b+
> 256b+
> No. of Reads: 194 280
> 500
> No. of Writes: 33703 34265
> 7020
>
> Block Size: 512b+ 1024b+
> 2048b+
> No. of Reads: 1302 1888
> 3596
> No. of Writes: 26022 17086
> 12495
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 5031 13204
> 21838
> No. of Writes: 21889 32294
> 18289
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 73578 190508
> 2710663
> No. of Writes: 17017 13980
> 144272
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 817553
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 1663792
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 1438610
> RELEASEDIR
> 0.00 85.00 us 85.00 us 85.00 us 1
> READDIR
> 0.00 58.50 us 56.00 us 60.00 us 4
> FSTAT
> 0.00 34.86 us 24.00 us 46.00 us 7
> STAT
> 0.00 89.67 us 81.00 us 100.00 us 3
> SETATTR
> 0.00 136.50 us 108.00 us 165.00 us 2
> XATTROP
> 0.00 63.20 us 35.00 us 87.00 us 5
> GETXATTR
> 0.00 54.73 us 39.00 us 66.00 us 15
> INODELK
> 0.01 53.39 us 33.00 us 88.00 us 18
> STATFS
> 0.01 58.00 us 32.00 us 259.00 us 25
> LK
> 0.01 208.50 us 112.00 us 691.00 us 12
> UNLINK
> 0.02 60.76 us 40.00 us 125.00 us 46
> FINODELK
> 0.02 176.63 us 74.00 us 302.00 us 19
> READ
> 0.02 52.51 us 23.00 us 97.00 us 68
> ENTRYLK
> 0.02 48.00 us 18.00 us 579.00 us 81
> FLUSH
> 0.03 206.95 us 163.00 us 497.00 us 22
> CREATE
> 0.04 85.16 us 51.00 us 168.00 us 70
> OPEN
> 0.04 145.02 us 92.00 us 344.00 us 45
> FXATTROP
> 0.07 59.91 us 3.00 us 145.00 us 184
> OPENDIR
> 0.13 1024.41 us 61.00 us 12251.00 us 22
> WRITE
> 0.14 444.38 us 44.00 us 11184.00 us 52
> READLINK
> 0.27 45104.00 us 45104.00 us 45104.00 us 1
> TRUNCATE
> 20.33 1895.33 us 27.00 us 6083.00 us 1819
> READDIRP
> 78.84 645.59 us 35.00 us 209082.00 us 20706
> LOOKUP
>
> Duration: 269212 seconds
> Data Read: 376538217873 bytes
> Data Written: 21968843126 bytes
>
> Interval 1 Stats:
> Block Size: 32b+ 64b+
> 128b+
> No. of Reads: 0 0
> 0
> No. of Writes: 1 4
> 3
>
> Block Size: 1024b+ 32768b+
> 131072b+
> No. of Reads: 0 1
> 10
> No. of Writes: 6 0
> 0
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 8
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 45
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 178
> RELEASEDIR
> 0.00 81.00 us 81.00 us 81.00 us 1
> SETATTR
> 0.00 85.00 us 85.00 us 85.00 us 1
> READDIR
> 0.00 63.33 us 35.00 us 87.00 us 3
> GETXATTR
> 0.00 58.50 us 56.00 us 60.00 us 4
> FSTAT
> 0.00 34.86 us 24.00 us 46.00 us 7
> STAT
> 0.00 54.00 us 47.00 us 66.00 us 9
> INODELK
> 0.01 51.25 us 33.00 us 78.00 us 12
> STATFS
> 0.01 156.20 us 112.00 us 202.00 us 5
> UNLINK
> 0.01 61.33 us 34.00 us 259.00 us 18
> LK
> 0.02 61.64 us 40.00 us 125.00 us 28
> FINODELK
> 0.02 54.65 us 21.00 us 579.00 us 40
> FLUSH
> 0.02 54.12 us 28.00 us 97.00 us 42
> ENTRYLK
> 0.02 161.93 us 74.00 us 302.00 us 15
> READ
> 0.02 83.97 us 68.00 us 107.00 us 29
> OPEN
> 0.03 214.88 us 163.00 us 497.00 us 16
> CREATE
> 0.04 144.21 us 92.00 us 344.00 us 28
> FXATTROP
> 0.10 60.01 us 3.00 us 145.00 us 178
> OPENDIR
> 0.10 316.62 us 44.00 us 8512.00 us 34
> READLINK
> 0.15 1121.14 us 61.00 us 12251.00 us 14
> WRITE
> 32.97 1895.33 us 27.00 us 6083.00 us 1819
> READDIRP
> 66.46 667.09 us 35.00 us 67882.00 us 10418
> LOOKUP
>
> Duration: 18 seconds
> Data Read: 1352163 bytes
> Data Written: 8465 bytes
>
> Profile 2:
> Brick: 192.168.10.92:/export/brick1
> -----------------------------------
> Cumulative Stats:
> Block Size: 1b+ 2b+
> 4b+
> No. of Reads: 0 0
> 0
> No. of Writes: 133 6
> 37
>
> Block Size: 8b+ 16b+
> 32b+
> No. of Reads: 28 93
> 31
> No. of Writes: 4577 6495
> 22231
>
> Block Size: 64b+ 128b+
> 256b+
> No. of Reads: 219 218
> 254
> No. of Writes: 37641 41723
> 20519
>
> Block Size: 512b+ 1024b+
> 2048b+
> No. of Reads: 484 981
> 696
> No. of Writes: 39094 42613
> 203748
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 1079 1802
> 2840
> No. of Writes: 220154 229725
> 215083
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 4227 14944
> 57816
> No. of Writes: 60938 61967
> 793400
>
> Block Size: 262144b+ 524288b+
> 1048576b+
> No. of Reads: 6 5
> 2
> No. of Writes: 5 0
> 0
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 1835089
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 2639439
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 1440418
> RELEASEDIR
> 0.00 53.50 us 43.00 us 64.00 us 2
> READDIR
> 0.00 153.50 us 142.00 us 165.00 us 2
> XATTROP
> 0.00 113.00 us 97.00 us 144.00 us 4
> SETATTR
> 0.00 114.60 us 49.00 us 339.00 us 10
> GETXATTR
> 0.00 55.00 us 33.00 us 100.00 us 24
> INODELK
> 0.00 48.85 us 25.00 us 81.00 us 71
> STATFS
> 0.01 49.47 us 31.00 us 76.00 us 92
> LK
> 0.01 36.07 us 16.00 us 68.00 us 180
> FLUSH
> 0.01 52.45 us 32.00 us 88.00 us 159
> FINODELK
> 0.02 79.16 us 47.00 us 112.00 us 137
> OPEN
> 0.02 51.06 us 26.00 us 95.00 us 216
> ENTRYLK
> 0.03 24814.00 us 24814.00 us 24814.00 us 1
> TRUNCATE
> 0.06 271.36 us 86.00 us 21829.00 us 158
> FXATTROP
> 0.07 59.02 us 2.00 us 100.00 us 881
> OPENDIR
> 0.07 5910.00 us 56.00 us 16873.00 us 9
> READLINK
> 0.11 33.81 us 11.00 us 1950.00 us 2348
> ACCESS
> 0.17 4181.53 us 129.00 us 41541.00 us 30
> UNLINK
> 0.31 2788.77 us 59.00 us 56839.00 us 79
> WRITE
> 0.37 2096.99 us 54.00 us 61234.00 us 125
> READ
> 0.38 84.58 us 53.00 us 313.00 us 3191
> FSTAT
> 0.50 61.21 us 20.00 us 14904.00 us 5873
> STAT
> 0.76 7023.69 us 145.00 us 102353.00 us 78
> CREATE
> 8.90 2002.80 us 361.00 us 3545.00 us 3191
> READDIRP
> 88.19 1112.81 us 34.00 us 307904.00 us 56876
> LOOKUP
>
> Duration: 338222 seconds
> Data Read: 9292898491 bytes
> Data Written: 122127690284 bytes
>
> Interval 4 Stats:
> Block Size: 16b+ 32b+
> 64b+
> No. of Reads: 0 0
> 2
> No. of Writes: 1 3
> 12
>
> Block Size: 128b+ 1024b+
> 2048b+
> No. of Reads: 2 3
> 10
> No. of Writes: 8 16
> 0
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 4 2
> 1
> No. of Writes: 0 0
> 0
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 2 3
> 1
> No. of Writes: 0 0
> 0
>
> Block Size: 262144b+
> No. of Reads: 2
> No. of Writes: 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 21
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 76
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 407
> RELEASEDIR
> 0.00 144.00 us 144.00 us 144.00 us 1
> SETATTR
> 0.00 63.33 us 56.00 us 69.00 us 3
> READLINK
> 0.00 226.00 us 113.00 us 339.00 us 2
> GETXATTR
> 0.00 54.89 us 39.00 us 70.00 us 9
> INODELK
> 0.01 48.03 us 25.00 us 72.00 us 37
> STATFS
> 0.01 36.75 us 16.00 us 65.00 us 59
> FLUSH
> 0.01 50.44 us 33.00 us 76.00 us 48
> LK
> 0.01 83.73 us 55.00 us 111.00 us 37
> OPEN
> 0.01 53.79 us 32.00 us 80.00 us 80
> FINODELK
> 0.02 50.55 us 26.00 us 83.00 us 104
> ENTRYLK
> 0.03 135.21 us 86.00 us 228.00 us 80
> FXATTROP
> 0.07 58.75 us 35.00 us 98.00 us 407
> OPENDIR
> 0.10 1044.22 us 55.00 us 30268.00 us 32
> READ
> 0.14 33.74 us 13.00 us 1044.00 us 1384
> ACCESS
> 0.19 5076.42 us 141.00 us 41541.00 us 12
> UNLINK
> 0.36 2860.70 us 59.00 us 29520.00 us 40
> WRITE
> 0.36 84.16 us 53.00 us 182.00 us 1369
> FSTAT
> 0.65 5225.18 us 145.00 us 28783.00 us 40
> CREATE
> 0.73 61.98 us 20.00 us 14904.00 us 3771
> STAT
> 8.53 2009.18 us 518.00 us 2532.00 us 1369
> READDIRP
> 88.77 1156.14 us 36.00 us 232288.00 us 24747
> LOOKUP
>
> Duration: 48 seconds
> Data Read: 1424095 bytes
> Data Written: 22682 bytes
>
> Brick: 192.168.10.91:/export/brick1
> -----------------------------------
> Cumulative Stats:
> Block Size: 1b+ 2b+
> 4b+
> No. of Reads: 0 0
> 0
> No. of Writes: 3 3
> 6
>
> Block Size: 8b+ 16b+
> 32b+
> No. of Reads: 13 173
> 58
> No. of Writes: 3371 5496
> 13461
>
> Block Size: 64b+ 128b+
> 256b+
> No. of Reads: 194 280
> 500
> No. of Writes: 33717 34277
> 7020
>
> Block Size: 512b+ 1024b+
> 2048b+
> No. of Reads: 1302 1888
> 3596
> No. of Writes: 26022 17110
> 12496
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 5031 13204
> 21838
> No. of Writes: 21889 32294
> 18289
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 73579 190511
> 2710676
> No. of Writes: 17017 13980
> 144272
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 817592
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 1663910
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 1439269
> RELEASEDIR
> 0.00 61.67 us 39.00 us 85.00 us 3
> READDIR
> 0.00 136.50 us 108.00 us 165.00 us 2
> XATTROP
> 0.00 91.00 us 81.00 us 100.00 us 4
> SETATTR
> 0.00 58.86 us 35.00 us 87.00 us 7
> GETXATTR
> 0.00 60.57 us 55.00 us 69.00 us 7
> FSTAT
> 0.00 35.53 us 24.00 us 53.00 us 15
> STAT
> 0.00 57.12 us 39.00 us 72.00 us 24
> INODELK
> 0.01 51.25 us 33.00 us 90.00 us 71
> STATFS
> 0.01 52.09 us 32.00 us 259.00 us 92
> LK
> 0.01 175.87 us 112.00 us 691.00 us 30
> UNLINK
> 0.01 41.84 us 16.00 us 579.00 us 175
> FLUSH
> 0.02 56.23 us 25.00 us 125.00 us 158
> FINODELK
> 0.02 51.59 us 23.00 us 97.00 us 216
> ENTRYLK
> 0.02 85.10 us 51.00 us 168.00 us 133
> OPEN
> 0.03 200.55 us 150.00 us 554.00 us 78
> CREATE
> 0.06 163.03 us 38.00 us 11184.00 us 204
> READLINK
> 0.07 1022.60 us 67.00 us 24446.00 us 40
> READ
> 0.08 45104.00 us 45104.00 us 45104.00 us 1
> TRUNCATE
> 0.09 59.52 us 3.00 us 145.00 us 843
> OPENDIR
> 0.13 936.73 us 56.00 us 12251.00 us 78
> WRITE
> 0.19 687.17 us 88.00 us 85925.00 us 157
> FXATTROP
> 22.66 1881.29 us 23.00 us 6083.00 us 6698
> READDIRP
> 76.59 754.63 us 33.00 us 209082.00 us 56449
> LOOKUP
>
> Duration: 269283 seconds
> Data Read: 376540270509 bytes
> Data Written: 21968880277 bytes
>
> Interval 4 Stats:
> Block Size: 16b+ 32b+
> 64b+
> No. of Reads: 0 0
> 0
> No. of Writes: 1 3
> 12
>
> Block Size: 128b+ 1024b+
> 32768b+
> No. of Reads: 0 0
> 1
> No. of Writes: 8 16
> 0
>
> Block Size: 65536b+ 131072b+
> No. of Reads: 2 9
> No. of Writes: 0 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 36
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 76
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 408
> RELEASEDIR
> 0.00 95.00 us 95.00 us 95.00 us 1
> SETATTR
> 0.00 48.00 us 39.00 us 57.00 us 2
> GETXATTR
> 0.00 50.00 us 39.00 us 61.00 us 2
> READDIR
> 0.00 35.50 us 27.00 us 53.00 us 4
> STAT
> 0.00 63.33 us 55.00 us 69.00 us 3
> FSTAT
> 0.00 61.11 us 50.00 us 72.00 us 9
> INODELK
> 0.01 50.41 us 34.00 us 90.00 us 37
> STATFS
> 0.01 160.08 us 125.00 us 213.00 us 12
> UNLINK
> 0.01 37.19 us 18.00 us 71.00 us 59
> FLUSH
> 0.01 48.46 us 32.00 us 78.00 us 48
> LK
> 0.01 84.92 us 57.00 us 118.00 us 37
> OPEN
> 0.02 54.89 us 25.00 us 89.00 us 80
> FINODELK
> 0.02 50.57 us 27.00 us 74.00 us 104
> ENTRYLK
> 0.03 66.88 us 38.00 us 100.00 us 113
> READLINK
> 0.03 196.90 us 150.00 us 554.00 us 40
> CREATE
> 0.04 139.06 us 88.00 us 233.00 us 80
> FXATTROP
> 0.10 58.95 us 8.00 us 93.00 us 408
> OPENDIR
> 0.11 1675.56 us 67.00 us 24446.00 us 16
> READ
> 0.17 1066.62 us 62.00 us 8506.00 us 40
> WRITE
> 23.14 1889.02 us 23.00 us 3775.00 us 3073
> READDIRP
> 76.29 773.40 us 33.00 us 135928.00 us 24746
> LOOKUP
>
> Duration: 48 seconds
> Data Read: 1446611 bytes
> Data Written: 22682 bytes
>
> Brick: 192.168.10.90:/export/brick1
> -----------------------------------
> Cumulative Stats:
> Block Size: 2b+ 8b+
> 16b+
> No. of Reads: 0 6
> 32
> No. of Writes: 1 1929
> 2605
>
> Block Size: 32b+ 64b+
> 128b+
> No. of Reads: 21 112
> 169
> No. of Writes: 6986 12080
> 26020
>
> Block Size: 256b+ 512b+
> 1024b+
> No. of Reads: 85 299
> 809
> No. of Writes: 743 18584
> 7233
>
> Block Size: 2048b+ 4096b+
> 8192b+
> No. of Reads: 790 1419
> 2885
> No. of Writes: 1651 10647
> 16901
>
> Block Size: 16384b+ 32768b+
> 65536b+
> No. of Reads: 4588 5858
> 32434
> No. of Writes: 2974 5758
> 3387
>
> Block Size: 131072b+
> No. of Reads: 455218
> No. of Writes: 4601
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 604813
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 1410912
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 990360
> RELEASEDIR
> 0.00 44.33 us 16.00 us 62.00 us 6
> READDIR
> 0.00 147.50 us 136.00 us 159.00 us 2
> XATTROP
> 0.00 100.25 us 86.00 us 125.00 us 4
> SETATTR
> 0.00 60.11 us 39.00 us 92.00 us 9
> GETXATTR
> 0.00 60.19 us 41.00 us 84.00 us 16
> FSTAT
> 0.00 45.50 us 17.00 us 182.00 us 38
> STAT
> 0.00 51.37 us 31.00 us 81.00 us 71
> STATFS
> 0.00 49.05 us 30.00 us 79.00 us 92
> LK
> 0.00 168.87 us 115.00 us 287.00 us 30
> UNLINK
> 0.01 38.74 us 15.00 us 150.00 us 150
> FLUSH
> 0.01 54.76 us 30.00 us 236.00 us 158
> FINODELK
> 0.01 87.17 us 49.00 us 352.00 us 106
> OPEN
> 0.01 52.56 us 27.00 us 91.00 us 216
> ENTRYLK
> 0.02 174.25 us 73.00 us 5551.00 us 157
> FXATTROP
> 0.03 35786.00 us 35786.00 us 35786.00 us 1
> TRUNCATE
> 0.05 62.12 us 3.00 us 1628.00 us 843
> OPENDIR
> 0.10 685.58 us 41.00 us 51432.00 us 157
> READLINK
> 0.36 6483.54 us 59.00 us 47225.00 us 63
> READ
> 0.58 8304.77 us 60.00 us 185503.00 us 78
> WRITE
> 0.67 9646.06 us 150.00 us 686579.00 us 78
> CREATE
> 1.82 85338.88 us 25.00 us 1502989.00 us 24
> INODELK
> 14.62 1920.48 us 23.00 us 9822.00 us 8567
> READDIRP
> 81.71 1635.58 us 30.00 us 828308.00 us 56208
> LOOKUP
>
> Duration: 76104 seconds
> Data Read: 63115678019 bytes
> Data Written: 1489698979 bytes
>
> Interval 4 Stats:
> Block Size: 16b+ 32b+
> 64b+
> No. of Reads: 0 0
> 0
> No. of Writes: 1 3
> 12
>
> Block Size: 128b+ 1024b+
> 4096b+
> No. of Reads: 0 0
> 1
> No. of Writes: 8 16
> 0
>
> Block Size: 8192b+ 32768b+
> 65536b+
> No. of Reads: 1 1
> 6
> No. of Writes: 0 0
> 0
>
> Block Size: 131072b+
> No. of Reads: 10
> No. of Writes: 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 29
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 76
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 409
> RELEASEDIR
> 0.00 125.00 us 125.00 us 125.00 us 1
> SETATTR
> 0.00 43.75 us 16.00 us 60.00 us 4
> READDIR
> 0.00 61.80 us 39.00 us 92.00 us 5
> GETXATTR
> 0.00 65.71 us 44.00 us 76.00 us 7
> FSTAT
> 0.00 48.50 us 24.00 us 182.00 us 28
> STAT
> 0.00 50.08 us 31.00 us 70.00 us 37
> STATFS
> 0.00 165.25 us 119.00 us 238.00 us 12
> UNLINK
> 0.00 39.66 us 19.00 us 62.00 us 59
> FLUSH
> 0.00 51.12 us 33.00 us 79.00 us 48
> LK
> 0.01 82.78 us 64.00 us 104.00 us 37
> OPEN
> 0.01 56.45 us 30.00 us 236.00 us 80
> FINODELK
> 0.01 52.17 us 28.00 us 84.00 us 104
> ENTRYLK
> 0.02 154.12 us 88.00 us 1577.00 us 80
> FXATTROP
> 0.05 65.00 us 3.00 us 1628.00 us 409
> OPENDIR
> 0.09 909.64 us 43.00 us 51432.00 us 61
> READLINK
> 0.60 9748.22 us 59.00 us 47225.00 us 36
> READ
> 0.86 12688.17 us 60.00 us 185503.00 us 40
> WRITE
> 0.92 60488.89 us 50.00 us 543898.00 us 9
> INODELK
> 1.26 18623.40 us 161.00 us 686579.00 us 40
> CREATE
> 19.51 1951.60 us 25.00 us 9822.00 us 5889
> READDIRP
> 76.64 1823.96 us 30.00 us 318395.00 us 24747
> LOOKUP
>
> Duration: 48 seconds
> Data Read: 1931486 bytes
> Data Written: 22682 bytes
>
> Again, any assistance on this issue would be much appreciated.
>
> Thank you.
>
>
>
> On Sat, Dec 20, 2014 at 11:58 PM, Kyle Harris <kyle.harris98 at
gmail.com>
> wrote:
>
>> Hello All,
>>
>> I have been experiencing difficulty with high process utilization on
the
>> glusterfs and glusterfsd processes. I thought I had it tracked down to
>> Apache looking for files on the cluster drive that don't exist. I
have
>> been working to clean that up and now I have discovered another
problem.
>>
>> When I reboot one of the nodes in my 3-node replicated cluster, as soon
>> as it comes back up I see a bunch of entries such as follows:
>>
>> [2014-12-21 05:33:49.873989] I [dict.c:370:dict_get] (-->
>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7fe3bd0d2420]
(-->
>> /usr/lib64/libglusterfs.so.0(dict_get+0x4f)[0x7fe3bd0cbbdf] (-->
>>
/usr/lib64/glusterfs/3.6.1/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x1e1)[0x7fe3aad3ada1]
>> (-->
>>
/usr/lib64/glusterfs/3.6.1/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x113)[0x7fe3aaf524e3]
>> (-->
>>
/usr/lib64/glusterfs/3.6.1/xlator/performance/md-cache.so(mdc_lookup+0x300)[0x7fe3ab167310]
>> ))))) 0-dict: !this || key=system.posix_acl_access
>>
>> While this is occurring, the process utilization for glusterfs and
>> glusterfsd go sky high. Eventually they calm down on their own but the
>> servers are virtually unusable until they do. Note this is on Centos
6.6
>> with Gluster 3.6.1 and kernel 2.6.32 using a fuse mount with each of
the 3
>> servers having the client mounted for hosting web sites via Apache.
There
>> are many small files and some are php. I have not done any tuning to
the
>> default gluster parameters.
>>
>> Can someone tell me why this occurs after one of the nodes is rebooted
>> and perhaps lend some guidance on what can be done to resolve this
issue?
>>
>> Thank you.
>>
>> --
>> Kyle
>>
>>
>
>
> --
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
--
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20141221/22b6efbe/attachment.html>
Krutika Dhananjay
2014-Dec-22 07:31 UTC
[Gluster-users] glusterfs and glusterfsd process utilization
----- Original Message -----> From: "Kyle Harris" <kyle.harris98 at gmail.com> > To: "Krutika Dhananjay" <kdhananj at redhat.com> > Cc: gluster-users at gluster.org > Sent: Monday, December 22, 2014 10:30:14 AM > Subject: Re: [Gluster-users] glusterfs and glusterfsd process utilization> Hi Krutika,> Thank you so much for the response. I'm starting to get a bit desperate :-)> I'm not sure which file is the brick log but the client log (I assume that is > the one that start with mnt-) is far too large to include. If you can tell > me what it is, perhaps we can arrange to get it to you using some other > method?> Anyway, I primarily see the following messages in what I believe to be the > client log.> The first is as mentioned earlier and repeated again here: > [2014-12-21 05:33:49.873989] I [dict.c:370:dict_get] (--> > /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7fe3bd0d2420] (--> > /usr/lib64/libglusterfs.so.0(dict_get+0x4f)[0x7fe3bd0cbbdf] (--> > /usr/lib64/glusterfs/3.6.1/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x1e1)[0x7fe3aad3ada1] > (--> > /usr/lib64/glusterfs/3.6.1/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x113)[0x7fe3aaf524e3] > (--> > /usr/lib64/glusterfs/3.6.1/xlator/performance/md-cache.so(mdc_lookup+0x300)[0x7fe3ab167310] > ))))) 0-dict: !this || key=system.posix_acl_access> This seems to be the one I see the most when it is starting up and having > this issue but I can't be certain. By the way, can you tell me what that > means?> The second entry I see often in the log files is in regards to missing files > such as follows: > [2014-12-22 02:33:32.062741] I [dht-common.c:1822:dht_lookup_cbk] 0-gv0-dht: > Entry /html/some_dir/some_file/asdf.php missing on subvol gv0-replicate-0> I thought these might be causing my problems but I see them all the time, > including right now and process utilization on the servers in this cluster > is currently way down next to nothing so I no longer think that is my > problem. I also checked while to see what, if any files needed healing > during the last time this issue occurred using the following command: > gluster volume heal gv0 info> It showed no files in need of healing.Hi, So it does look like it was self-heal that was causing the CPU usage to go up because ever since 'heal info' started reporting no entries to be healed, the CPU usage was also under control. We also found a bug in the code where disabling entry-self-heal does not stop the normal clients from performing self-heal. This will be fixed soon. For now, to disable self-heal on the client side, you should disable all the three forms of self-healing. You can do that using the following commands: #gluster volume set <volname> cluster.data-self-heal off #gluster volume set <volname> cluster.entry-self-heal off #gluster volume set <volname> cluster.metadata-self-heal off And if you want to disable self-healing from self-heal-daemon, you can set cluster.self-heal-daemon to off. And you can re-enable it when you want self-heal to happen at a time of your choice, so that it does not come in the way of the normal clients' operation. -Krutika> In the log file etc-glusterfs-glusterd.vol.log I see many of the following > entries: > [2014-12-21 19:13:54.137263] W [socket.c:611:__socket_rwv] 0-management: > readv on /var/run/095271447d0b71369a479e6211bbe6e5.socket failed (Invalid > argument)> I have a bricks dir with the file export-brick1.log and it is empty.> gluster volume info as requested:> Volume Name: gv0 > Type: Replicate > Volume ID: 3565c78b-2d23-42e7-8cd2-1724fe671610 > Status: Started > Number of Bricks: 1 x 3 = 3 > Transport-type: tcp > Bricks: > Brick1: 192.168.10.90:/export/brick1 > Brick2: 192.168.10.91:/export/brick1 > Brick3: 192.168.10.92:/export/brick1 > Options Reconfigured: > cluster.data-self-heal-algorithm: full > cluster.entry-self-heal: off> I also found the following in a log file and perhaps it will help: > Final graph: > +------------------------------------------------------------------------------+ > 1: volume gv0-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 192.168.10.90 > 5: option remote-subvolume /export/brick1 > 6: option transport-type socket > 7: option username {removed} > 8: option password {removed} > 9: option send-gids true > 10: end-volume > 11: > 12: volume gv0-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 192.168.10.91 > 16: option remote-subvolume /export/brick1 > 17: option transport-type socket > 18: option username {removed} > 19: option password {removed} > 20: option send-gids true > 21: end-volume > 22: > 23: volume gv0-client-2 > 24: type protocol/client > 25: option ping-timeout 42 > 26: option remote-host 192.168.10.92 > 27: option remote-subvolume /export/brick1 > 28: option transport-type socket > 29: option username {removed} > 30: option password {removed} > 31: option send-gids true > 32: end-volume > 33: > 34: volume gv0-replicate-0 > 35: type cluster/replicate > 36: option entry-self-heal off > 37: option data-self-heal-algorithm full > 38: subvolumes gv0-client-0 gv0-client-1 gv0-client-2 > 39: end-volume > 40: > 41: volume gv0-dht > 42: type cluster/distribute > 43: subvolumes gv0-replicate-0 > 44: end-volume > 45: > 46: volume gv0-write-behind > 47: type performance/write-behind > 48: subvolumes gv0-dht > 49: end-volume > 50: > 51: volume gv0 > 52: type debug/io-stats > 53: option latency-measurement off > 54: option count-fop-hits off > 55: subvolumes gv0-write-behind > 56: end-volume > 57: > 58: volume nfs-server > 59: type nfs/server > 60: option rpc-auth.auth-glusterfs on > 61: option rpc-auth.auth-unix on > 62: option rpc-auth.auth-null on > 63: option rpc-auth.ports.insecure on > 64: option rpc-auth-allow-insecure on > 65: option transport-type socket > 66: option transport.socket.listen-port 2049 > 67: option nfs.dynamic-volumes on > 68: option nfs.nlm on > 69: option nfs.drc off > 70: option rpc-auth.addr.gv0.allow * > 71: option nfs3.gv0.volume-id 3565c78b-2d23-42e7-8cd2-1724fe671610 > 72: subvolumes gv0 > 73: end-volume > 74:> With regard to the average number of directory entries, I don't know for sure > but I'm sure some of them get pretty deep.> Finally, I do not have quota enabled.> Again, thank you very much and regards,> Kyle> On Sun, Dec 21, 2014 at 10:19 PM, Krutika Dhananjay < kdhananj at redhat.com > > wrote:> > Hi, >> > Could you provide the following pieces of information? >> > 1. The brick and client logs > > > 2. output of `gluster volume info` > > > 3. average number of entries under directories >> > Just to be clear, you don't have quota enabled on the volume, do you? >> > -Krutika >> > > From: "Kyle Harris" < kyle.harris98 at gmail.com > > > > > > > To: gluster-users at gluster.org > > > > > > Sent: Monday, December 22, 2014 8:27:09 AM > > > > > > Subject: Re: [Gluster-users] glusterfs and glusterfsd process utilization > > >> > > I haven't heard anything from anyone re this post yet so I have a bit > > > more > > > info to add. The last time I posted about extremely high process > > > utilization > > > for glusterfs and glusterfsd, I was asked to see if I could capture the > > > output of a profile. I did so and here is the output of 2 captures (they > > > appear very similar). Perhaps this will assist. Note that previously I > > > was > > > mounting with the fuse client. This time I tried NFS. > > >> > > Profile Output 1: > > > > > > Brick: 192.168.10.92:/export/brick1 > > > > > > ----------------------------------- > > > > > > Cumulative Stats: > > > > > > Block Size: 1b+ 2b+ 4b+ > > > > > > No. of Reads: 0 0 0 > > > > > > No. of Writes: 133 6 37 > > > > > > Block Size: 8b+ 16b+ 32b+ > > > > > > No. of Reads: 28 93 31 > > > > > > No. of Writes: 4576 6494 22228 > > > > > > Block Size: 64b+ 128b+ 256b+ > > > > > > No. of Reads: 217 216 254 > > > > > > No. of Writes: 37627 41711 20519 > > > > > > Block Size: 512b+ 1024b+ 2048b+ > > > > > > No. of Reads: 484 978 684 > > > > > > No. of Writes: 39094 42589 203747 > > > > > > Block Size: 4096b+ 8192b+ 16384b+ > > > > > > No. of Reads: 1073 1800 2838 > > > > > > No. of Writes: 220154 229725 215083 > > > > > > Block Size: 32768b+ 65536b+ 131072b+ > > > > > > No. of Reads: 4224 14941 57812 > > > > > > No. of Writes: 60938 61967 793400 > > > > > > Block Size: 262144b+ 524288b+ 1048576b+ > > > > > > No. of Reads: 4 4 2 > > > > > > No. of Writes: 5 0 0 > > > > > > %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop > > > > > > --------- ----------- ----------- ----------- ------------ ---- > > > > > > 0.00 0.00 us 0.00 us 0.00 us 1835064 FORGET > > > > > > 0.00 0.00 us 0.00 us 0.00 us 2639321 RELEASE > > > > > > 0.00 0.00 us 0.00 us 0.00 us 1439759 RELEASEDIR > > > > > > 0.00 153.50 us 142.00 us 165.00 us 2 XATTROP > > > > > > 0.00 102.67 us 97.00 us 110.00 us 3 SETATTR > > > > > > 0.00 93.20 us 53.00 us 129.00 us 5 GETXATTR > > > > > > 0.00 55.07 us 33.00 us 100.00 us 15 INODELK > > > > > > 0.00 46.61 us 35.00 us 69.00 us 18 STATFS > > > > > > 0.01 48.40 us 31.00 us 64.00 us 25 LK > > > > > > 0.01 50.72 us 32.00 us 88.00 us 47 FINODELK > > > > > > 0.01 35.49 us 17.00 us 68.00 us 86 FLUSH > > > > > > 0.02 50.59 us 27.00 us 76.00 us 68 ENTRYLK > > > > > > 0.03 75.72 us 47.00 us 112.00 us 74 OPEN > > > > > > 0.06 58.79 us 32.00 us 100.00 us 222 OPENDIR > > > > > > 0.08 1535.92 us 129.00 us 16583.00 us 12 UNLINK > > > > > > 0.11 24814.00 us 24814.00 us 24814.00 us 1 TRUNCATE > > > > > > 0.12 34.25 us 11.00 us 1950.00 us 784 ACCESS > > > > > > 0.13 603.50 us 90.00 us 21829.00 us 46 FXATTROP > > > > > > 0.16 7225.40 us 63.00 us 14563.00 us 5 READLINK > > > > > > 0.17 437.90 us 54.00 us 19042.00 us 83 READ > > > > > > 0.36 85.92 us 56.00 us 313.00 us 912 FSTAT > > > > > > 0.42 59.86 us 22.00 us 1443.00 us 1533 STAT > > > > > > 0.45 4318.65 us 70.00 us 56839.00 us 23 WRITE > > > > > > 0.84 8405.18 us 161.00 us 37688.00 us 22 CREATE > > > > > > 8.33 2005.77 us 361.00 us 2796.00 us 912 READDIRP > > > > > > 88.68 921.10 us 34.00 us 159393.00 us 21137 LOOKUP > > > > > > Duration: 338151 seconds > > > > > > Data Read: 9290348715 bytes > > > > > > Data Written: 122127653133 bytes > > > > > > Interval 1 Stats: > > > > > > Block Size: 8b+ 32b+ 64b+ > > > > > > No. of Reads: 2 0 1 > > > > > > No. of Writes: 0 1 4 > > > > > > Block Size: 128b+ 256b+ 512b+ > > > > > > No. of Reads: 1 2 5 > > > > > > No. of Writes: 3 0 0 > > > > > > Block Size: 1024b+ 2048b+ 4096b+ > > > > > > No. of Reads: 7 13 11 > > > > > > No. of Writes: 6 0 0 > > > > > > Block Size: 8192b+ 16384b+ 32768b+ > > > > > > No. of Reads: 10 2 9 > > > > > > No. of Writes: 0 0 0 > > > > > > Block Size: 65536b+ 131072b+ 262144b+ > > > > > > No. of Reads: 1 1 1 > > > > > > No. of Writes: 0 0 0 > > > > > > %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop > > > > > > --------- ----------- ----------- ----------- ------------ ---- > > > > > > 0.00 0.00 us 0.00 us 0.00 us 7 FORGET > > > > > > 0.00 0.00 us 0.00 us 0.00 us 45 RELEASE > > > > > > 0.00 0.00 us 0.00 us 0.00 us 177 RELEASEDIR > > > > > > 0.00 109.00 us 109.00 us 109.00 us 1 GETXATTR > > > > > > 0.00 110.00 us 110.00 us 110.00 us 1 SETATTR > > > > > > 0.00 52.78 us 33.00 us 81.00 us 9 INODELK > > > > > > 0.00 48.92 us 36.00 us 69.00 us 12 STATFS > > > > > > 0.01 146.80 us 129.00 us 171.00 us 5 UNLINK > > > > > > 0.01 47.89 us 31.00 us 63.00 us 18 LK > > > > > > 0.01 34.25 us 17.00 us 61.00 us 40 FLUSH > > > > > > 0.01 49.89 us 35.00 us 64.00 us 28 FINODELK > > > > > > 0.02 49.12 us 27.00 us 76.00 us 42 ENTRYLK > > > > > > 0.02 78.14 us 47.00 us 99.00 us 29 OPEN > > > > > > 0.08 58.01 us 34.00 us 100.00 us 177 OPENDIR > > > > > > 0.13 241.08 us 54.00 us 10214.00 us 66 READ > > > > > > 0.18 34.76 us 11.00 us 1950.00 us 649 ACCESS > > > > > > 0.21 906.82 us 94.00 us 21829.00 us 28 FXATTROP > > > > > > 0.29 9014.75 us 63.00 us 14563.00 us 4 READLINK > > > > > > 0.32 85.78 us 56.00 us 313.00 us 455 FSTAT > > > > > > 0.59 60.01 us 22.00 us 1443.00 us 1215 STAT > > > > > > 0.67 5906.21 us 70.00 us 56839.00 us 14 WRITE > > > > > > 1.02 7854.81 us 161.00 us 27303.00 us 16 CREATE > > > > > > 7.42 2008.99 us 361.00 us 2796.00 us 455 READDIRP > > > > > > 89.00 1052.36 us 37.00 us 132924.00 us 10418 LOOKUP > > > > > > Duration: 18 seconds > > > > > > Data Read: 1285736 bytes > > > > > > Data Written: 8465 bytes > > > > > > Brick: 192.168.10.90:/export/brick1 > > > > > > ----------------------------------- > > > > > > Cumulative Stats: > > > > > > Block Size: 2b+ 8b+ 16b+ > > > > > > No. of Reads: 0 6 32 > > > > > > No. of Writes: 1 1928 2604 > > > > > > Block Size: 32b+ 64b+ 128b+ > > > > > > No. of Reads: 21 112 169 > > > > > > No. of Writes: 6983 12066 26008 > > > > > > Block Size: 256b+ 512b+ 1024b+ > > > > > > No. of Reads: 85 299 809 > > > > > > No. of Writes: 743 18584 7209 > > > > > > Block Size: 2048b+ 4096b+ 8192b+ > > > > > > No. of Reads: 790 1418 2884 > > > > > > No. of Writes: 1650 10647 16901 > > > > > > Block Size: 16384b+ 32768b+ 65536b+ > > > > > > No. of Reads: 4588 5856 32428 > > > > > > No. of Writes: 2974 5758 3387 > > > > > > Block Size: 131072b+ > > > > > > No. of Reads: 455199 > > > > > > No. of Writes: 4601 > > > > > > %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop > > > > > > --------- ----------- ----------- ----------- ------------ ---- > > > > > > 0.00 0.00 us 0.00 us 0.00 us 604780 FORGET > > > > > > 0.00 0.00 us 0.00 us 0.00 us 1410794 RELEASE > > > > > > 0.00 0.00 us 0.00 us 0.00 us 989700 RELEASEDIR > > > > > > 0.00 45.50 us 29.00 us 62.00 us 2 READDIR > > > > > > 0.00 58.00 us 44.00 us 77.00 us 4 GETXATTR > > > > > > 0.00 92.00 us 86.00 us 101.00 us 3 SETATTR > > > > > > 0.00 147.50 us 136.00 us 159.00 us 2 XATTROP > > > > > > 0.00 55.89 us 41.00 us 84.00 us 9 FSTAT > > > > > > 0.00 54.22 us 33.00 us 76.00 us 18 STATFS > > > > > > 0.00 46.04 us 30.00 us 60.00 us 25 LK > > > > > > 0.01 172.17 us 115.00 us 222.00 us 12 UNLINK > > > > > > 0.01 38.75 us 15.00 us 150.00 us 56 FLUSH > > > > > > 0.01 54.46 us 30.00 us 101.00 us 46 FINODELK > > > > > > 0.01 52.78 us 27.00 us 91.00 us 68 ENTRYLK > > > > > > 0.01 93.74 us 49.00 us 352.00 us 43 OPEN > > > > > > 0.01 204.50 us 150.00 us 511.00 us 22 CREATE > > > > > > 0.03 58.25 us 3.00 us 94.00 us 183 OPENDIR > > > > > > 0.03 247.33 us 73.00 us 5551.00 us 45 FXATTROP > > > > > > 0.11 35786.00 us 35786.00 us 35786.00 us 1 TRUNCATE > > > > > > 0.15 669.42 us 41.00 us 23631.00 us 76 READLINK > > > > > > 0.17 3501.75 us 60.00 us 27637.00 us 16 READ > > > > > > 0.34 5093.86 us 60.00 us 69136.00 us 22 WRITE > > > > > > 4.56 100248.87 us 25.00 us 1502989.00 us 15 INODELK > > > > > > 9.64 1930.07 us 30.00 us 3404.00 us 1646 READDIRP > > > > > > 84.89 1367.04 us 34.00 us 828308.00 us 20463 LOOKUP > > > > > > Duration: 76033 seconds > > > > > > Data Read: 63112532714 bytes > > > > > > Data Written: 1489661828 bytes > > > > > > Interval 1 Stats: > > > > > > Block Size: 32b+ 64b+ 128b+ > > > > > > No. of Reads: 0 0 0 > > > > > > No. of Writes: 1 4 3 > > > > > > Block Size: 1024b+ 65536b+ 131072b+ > > > > > > No. of Reads: 0 2 1 > > > > > > No. of Writes: 6 0 0 > > > > > > %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop > > > > > > --------- ----------- ----------- ----------- ------------ ---- > > > > > > 0.00 0.00 us 0.00 us 0.00 us 9 FORGET > > > > > > 0.00 0.00 us 0.00 us 0.00 us 45 RELEASE > > > > > > 0.00 0.00 us 0.00 us 0.00 us 177 RELEASEDIR > > > > > > 0.00 86.00 us 86.00 us 86.00 us 1 SETATTR > > > > > > 0.00 55.89 us 41.00 us 84.00 us 9 FSTAT > > > > > > 0.00 53.50 us 40.00 us 76.00 us 12 STATFS > > > > > > 0.00 160.00 us 130.00 us 184.00 us 5 UNLINK > > > > > > 0.00 46.83 us 30.00 us 60.00 us 18 LK > > > > > > 0.01 51.11 us 30.00 us 88.00 us 28 FINODELK > > > > > > 0.01 37.88 us 15.00 us 150.00 us 40 FLUSH > > > > > > 0.01 52.14 us 27.00 us 83.00 us 42 ENTRYLK > > > > > > 0.01 101.34 us 52.00 us 352.00 us 29 OPEN > > > > > > 0.02 209.44 us 150.00 us 511.00 us 16 CREATE > > > > > > 0.04 319.57 us 73.00 us 5551.00 us 28 FXATTROP > > > > > > 0.05 58.58 us 29.00 us 94.00 us 177 OPENDIR > > > > > > 0.23 669.42 us 41.00 us 23631.00 us 76 READLINK > > > > > > 0.26 3728.27 us 60.00 us 27637.00 us 15 READ > > > > > > 0.37 5742.93 us 60.00 us 69136.00 us 14 WRITE > > > > > > 6.88 167043.89 us 25.00 us 1502989.00 us 9 INODELK > > > > > > 10.95 1908.32 us 30.00 us 3404.00 us 1254 READDIRP > > > > > > 81.16 1701.62 us 34.00 us 828308.00 us 10421 LOOKUP > > > > > > Duration: 18 seconds > > > > > > Data Read: 310621 bytes > > > > > > Data Written: 8465 bytes > > > > > > Brick: 192.168.10.91:/export/brick1 > > > > > > ----------------------------------- > > > > > > Cumulative Stats: > > > > > > Block Size: 1b+ 2b+ 4b+ > > > > > > No. of Reads: 0 0 0 > > > > > > No. of Writes: 3 3 6 > > > > > > Block Size: 8b+ 16b+ 32b+ > > > > > > No. of Reads: 13 173 58 > > > > > > No. of Writes: 3370 5495 13458 > > > > > > Block Size: 64b+ 128b+ 256b+ > > > > > > No. of Reads: 194 280 500 > > > > > > No. of Writes: 33703 34265 7020 > > > > > > Block Size: 512b+ 1024b+ 2048b+ > > > > > > No. of Reads: 1302 1888 3596 > > > > > > No. of Writes: 26022 17086 12495 > > > > > > Block Size: 4096b+ 8192b+ 16384b+ > > > > > > No. of Reads: 5031 13204 21838 > > > > > > No. of Writes: 21889 32294 18289 > > > > > > Block Size: 32768b+ 65536b+ 131072b+ > > > > > > No. of Reads: 73578 190508 2710663 > > > > > > No. of Writes: 17017 13980 144272 > > > > > > %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop > > > > > > --------- ----------- ----------- ----------- ------------ ---- > > > > > > 0.00 0.00 us 0.00 us 0.00 us 817553 FORGET > > > > > > 0.00 0.00 us 0.00 us 0.00 us 1663792 RELEASE > > > > > > 0.00 0.00 us 0.00 us 0.00 us 1438610 RELEASEDIR > > > > > > 0.00 85.00 us 85.00 us 85.00 us 1 READDIR > > > > > > 0.00 58.50 us 56.00 us 60.00 us 4 FSTAT > > > > > > 0.00 34.86 us 24.00 us 46.00 us 7 STAT > > > > > > 0.00 89.67 us 81.00 us 100.00 us 3 SETATTR > > > > > > 0.00 136.50 us 108.00 us 165.00 us 2 XATTROP > > > > > > 0.00 63.20 us 35.00 us 87.00 us 5 GETXATTR > > > > > > 0.00 54.73 us 39.00 us 66.00 us 15 INODELK > > > > > > 0.01 53.39 us 33.00 us 88.00 us 18 STATFS > > > > > > 0.01 58.00 us 32.00 us 259.00 us 25 LK > > > > > > 0.01 208.50 us 112.00 us 691.00 us 12 UNLINK > > > > > > 0.02 60.76 us 40.00 us 125.00 us 46 FINODELK > > > > > > 0.02 176.63 us 74.00 us 302.00 us 19 READ > > > > > > 0.02 52.51 us 23.00 us 97.00 us 68 ENTRYLK > > > > > > 0.02 48.00 us 18.00 us 579.00 us 81 FLUSH > > > > > > 0.03 206.95 us 163.00 us 497.00 us 22 CREATE > > > > > > 0.04 85.16 us 51.00 us 168.00 us 70 OPEN > > > > > > 0.04 145.02 us 92.00 us 344.00 us 45 FXATTROP > > > > > > 0.07 59.91 us 3.00 us 145.00 us 184 OPENDIR > > > > > > 0.13 1024.41 us 61.00 us 12251.00 us 22 WRITE > > > > > > 0.14 444.38 us 44.00 us 11184.00 us 52 READLINK > > > > > > 0.27 45104.00 us 45104.00 us 45104.00 us 1 TRUNCATE > > > > > > 20.33 1895.33 us 27.00 us 6083.00 us 1819 READDIRP > > > > > > 78.84 645.59 us 35.00 us 209082.00 us 20706 LOOKUP > > > > > > Duration: 269212 seconds > > > > > > Data Read: 376538217873 bytes > > > > > > Data Written: 21968843126 bytes > > > > > > Interval 1 Stats: > > > > > > Block Size: 32b+ 64b+ 128b+ > > > > > > No. of Reads: 0 0 0 > > > > > > No. of Writes: 1 4 3 > > > > > > Block Size: 1024b+ 32768b+ 131072b+ > > > > > > No. of Reads: 0 1 10 > > > > > > No. of Writes: 6 0 0 > > > > > > %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop > > > > > > --------- ----------- ----------- ----------- ------------ ---- > > > > > > 0.00 0.00 us 0.00 us 0.00 us 8 FORGET > > > > > > 0.00 0.00 us 0.00 us 0.00 us 45 RELEASE > > > > > > 0.00 0.00 us 0.00 us 0.00 us 178 RELEASEDIR > > > > > > 0.00 81.00 us 81.00 us 81.00 us 1 SETATTR > > > > > > 0.00 85.00 us 85.00 us 85.00 us 1 READDIR > > > > > > 0.00 63.33 us 35.00 us 87.00 us 3 GETXATTR > > > > > > 0.00 58.50 us 56.00 us 60.00 us 4 FSTAT > > > > > > 0.00 34.86 us 24.00 us 46.00 us 7 STAT > > > > > > 0.00 54.00 us 47.00 us 66.00 us 9 INODELK > > > > > > 0.01 51.25 us 33.00 us 78.00 us 12 STATFS > > > > > > 0.01 156.20 us 112.00 us 202.00 us 5 UNLINK > > > > > > 0.01 61.33 us 34.00 us 259.00 us 18 LK > > > > > > 0.02 61.64 us 40.00 us 125.00 us 28 FINODELK > > > > > > 0.02 54.65 us 21.00 us 579.00 us 40 FLUSH > > > > > > 0.02 54.12 us 28.00 us 97.00 us 42 ENTRYLK > > > > > > 0.02 161.93 us 74.00 us 302.00 us 15 READ > > > > > > 0.02 83.97 us 68.00 us 107.00 us 29 OPEN > > > > > > 0.03 214.88 us 163.00 us 497.00 us 16 CREATE > > > > > > 0.04 144.21 us 92.00 us 344.00 us 28 FXATTROP > > > > > > 0.10 60.01 us 3.00 us 145.00 us 178 OPENDIR > > > > > > 0.10 316.62 us 44.00 us 8512.00 us 34 READLINK > > > > > > 0.15 1121.14 us 61.00 us 12251.00 us 14 WRITE > > > > > > 32.97 1895.33 us 27.00 us 6083.00 us 1819 READDIRP > > > > > > 66.46 667.09 us 35.00 us 67882.00 us 10418 LOOKUP > > > > > > Duration: 18 seconds > > > > > > Data Read: 1352163 bytes > > > > > > Data Written: 8465 bytes > > > > > > Profile 2: > > > > > > Brick: 192.168.10.92:/export/brick1 > > > > > > ----------------------------------- > > > > > > Cumulative Stats: > > > > > > Block Size: 1b+ 2b+ 4b+ > > > > > > No. of Reads: 0 0 0 > > > > > > No. of Writes: 133 6 37 > > > > > > Block Size: 8b+ 16b+ 32b+ > > > > > > No. of Reads: 28 93 31 > > > > > > No. of Writes: 4577 6495 22231 > > > > > > Block Size: 64b+ 128b+ 256b+ > > > > > > No. of Reads: 219 218 254 > > > > > > No. of Writes: 37641 41723 20519 > > > > > > Block Size: 512b+ 1024b+ 2048b+ > > > > > > No. of Reads: 484 981 696 > > > > > > No. of Writes: 39094 42613 203748 > > > > > > Block Size: 4096b+ 8192b+ 16384b+ > > > > > > No. of Reads: 1079 1802 2840 > > > > > > No. of Writes: 220154 229725 215083 > > > > > > Block Size: 32768b+ 65536b+ 131072b+ > > > > > > No. of Reads: 4227 14944 57816 > > > > > > No. of Writes: 60938 61967 793400 > > > > > > Block Size: 262144b+ 524288b+ 1048576b+ > > > > > > No. of Reads: 6 5 2 > > > > > > No. of Writes: 5 0 0 > > > > > > %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop > > > > > > --------- ----------- ----------- ----------- ------------ ---- > > > > > > 0.00 0.00 us 0.00 us 0.00 us 1835089 FORGET > > > > > > 0.00 0.00 us 0.00 us 0.00 us 2639439 RELEASE > > > > > > 0.00 0.00 us 0.00 us 0.00 us 1440418 RELEASEDIR > > > > > > 0.00 53.50 us 43.00 us 64.00 us 2 READDIR > > > > > > 0.00 153.50 us 142.00 us 165.00 us 2 XATTROP > > > > > > 0.00 113.00 us 97.00 us 144.00 us 4 SETATTR > > > > > > 0.00 114.60 us 49.00 us 339.00 us 10 GETXATTR > > > > > > 0.00 55.00 us 33.00 us 100.00 us 24 INODELK > > > > > > 0.00 48.85 us 25.00 us 81.00 us 71 STATFS > > > > > > 0.01 49.47 us 31.00 us 76.00 us 92 LK > > > > > > 0.01 36.07 us 16.00 us 68.00 us 180 FLUSH > > > > > > 0.01 52.45 us 32.00 us 88.00 us 159 FINODELK > > > > > > 0.02 79.16 us 47.00 us 112.00 us 137 OPEN > > > > > > 0.02 51.06 us 26.00 us 95.00 us 216 ENTRYLK > > > > > > 0.03 24814.00 us 24814.00 us 24814.00 us 1 TRUNCATE > > > > > > 0.06 271.36 us 86.00 us 21829.00 us 158 FXATTROP > > > > > > 0.07 59.02 us 2.00 us 100.00 us 881 OPENDIR > > > > > > 0.07 5910.00 us 56.00 us 16873.00 us 9 READLINK > > > > > > 0.11 33.81 us 11.00 us 1950.00 us 2348 ACCESS > > > > > > 0.17 4181.53 us 129.00 us 41541.00 us 30 UNLINK > > > > > > 0.31 2788.77 us 59.00 us 56839.00 us 79 WRITE > > > > > > 0.37 2096.99 us 54.00 us 61234.00 us 125 READ > > > > > > 0.38 84.58 us 53.00 us 313.00 us 3191 FSTAT > > > > > > 0.50 61.21 us 20.00 us 14904.00 us 5873 STAT > > > > > > 0.76 7023.69 us 145.00 us 102353.00 us 78 CREATE > > > > > > 8.90 2002.80 us 361.00 us 3545.00 us 3191 READDIRP > > > > > > 88.19 1112.81 us 34.00 us 307904.00 us 56876 LOOKUP > > > > > > Duration: 338222 seconds > > > > > > Data Read: 9292898491 bytes > > > > > > Data Written: 122127690284 bytes > > > > > > Interval 4 Stats: > > > > > > Block Size: 16b+ 32b+ 64b+ > > > > > > No. of Reads: 0 0 2 > > > > > > No. of Writes: 1 3 12 > > > > > > Block Size: 128b+ 1024b+ 2048b+ > > > > > > No. of Reads: 2 3 10 > > > > > > No. of Writes: 8 16 0 > > > > > > Block Size: 4096b+ 8192b+ 16384b+ > > > > > > No. of Reads: 4 2 1 > > > > > > No. of Writes: 0 0 0 > > > > > > Block Size: 32768b+ 65536b+ 131072b+ > > > > > > No. of Reads: 2 3 1 > > > > > > No. of Writes: 0 0 0 > > > > > > Block Size: 262144b+ > > > > > > No. of Reads: 2 > > > > > > No. of Writes: 0 > > > > > > %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop > > > > > > --------- ----------- ----------- ----------- ------------ ---- > > > > > > 0.00 0.00 us 0.00 us 0.00 us 21 FORGET > > > > > > 0.00 0.00 us 0.00 us 0.00 us 76 RELEASE > > > > > > 0.00 0.00 us 0.00 us 0.00 us 407 RELEASEDIR > > > > > > 0.00 144.00 us 144.00 us 144.00 us 1 SETATTR > > > > > > 0.00 63.33 us 56.00 us 69.00 us 3 READLINK > > > > > > 0.00 226.00 us 113.00 us 339.00 us 2 GETXATTR > > > > > > 0.00 54.89 us 39.00 us 70.00 us 9 INODELK > > > > > > 0.01 48.03 us 25.00 us 72.00 us 37 STATFS > > > > > > 0.01 36.75 us 16.00 us 65.00 us 59 FLUSH > > > > > > 0.01 50.44 us 33.00 us 76.00 us 48 LK > > > > > > 0.01 83.73 us 55.00 us 111.00 us 37 OPEN > > > > > > 0.01 53.79 us 32.00 us 80.00 us 80 FINODELK > > > > > > 0.02 50.55 us 26.00 us 83.00 us 104 ENTRYLK > > > > > > 0.03 135.21 us 86.00 us 228.00 us 80 FXATTROP > > > > > > 0.07 58.75 us 35.00 us 98.00 us 407 OPENDIR > > > > > > 0.10 1044.22 us 55.00 us 30268.00 us 32 READ > > > > > > 0.14 33.74 us 13.00 us 1044.00 us 1384 ACCESS > > > > > > 0.19 5076.42 us 141.00 us 41541.00 us 12 UNLINK > > > > > > 0.36 2860.70 us 59.00 us 29520.00 us 40 WRITE > > > > > > 0.36 84.16 us 53.00 us 182.00 us 1369 FSTAT > > > > > > 0.65 5225.18 us 145.00 us 28783.00 us 40 CREATE > > > > > > 0.73 61.98 us 20.00 us 14904.00 us 3771 STAT > > > > > > 8.53 2009.18 us 518.00 us 2532.00 us 1369 READDIRP > > > > > > 88.77 1156.14 us 36.00 us 232288.00 us 24747 LOOKUP > > > > > > Duration: 48 seconds > > > > > > Data Read: 1424095 bytes > > > > > > Data Written: 22682 bytes > > > > > > Brick: 192.168.10.91:/export/brick1 > > > > > > ----------------------------------- > > > > > > Cumulative Stats: > > > > > > Block Size: 1b+ 2b+ 4b+ > > > > > > No. of Reads: 0 0 0 > > > > > > No. of Writes: 3 3 6 > > > > > > Block Size: 8b+ 16b+ 32b+ > > > > > > No. of Reads: 13 173 58 > > > > > > No. of Writes: 3371 5496 13461 > > > > > > Block Size: 64b+ 128b+ 256b+ > > > > > > No. of Reads: 194 280 500 > > > > > > No. of Writes: 33717 34277 7020 > > > > > > Block Size: 512b+ 1024b+ 2048b+ > > > > > > No. of Reads: 1302 1888 3596 > > > > > > No. of Writes: 26022 17110 12496 > > > > > > Block Size: 4096b+ 8192b+ 16384b+ > > > > > > No. of Reads: 5031 13204 21838 > > > > > > No. of Writes: 21889 32294 18289 > > > > > > Block Size: 32768b+ 65536b+ 131072b+ > > > > > > No. of Reads: 73579 190511 2710676 > > > > > > No. of Writes: 17017 13980 144272 > > > > > > %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop > > > > > > --------- ----------- ----------- ----------- ------------ ---- > > > > > > 0.00 0.00 us 0.00 us 0.00 us 817592 FORGET > > > > > > 0.00 0.00 us 0.00 us 0.00 us 1663910 RELEASE > > > > > > 0.00 0.00 us 0.00 us 0.00 us 1439269 RELEASEDIR > > > > > > 0.00 61.67 us 39.00 us 85.00 us 3 READDIR > > > > > > 0.00 136.50 us 108.00 us 165.00 us 2 XATTROP > > > > > > 0.00 91.00 us 81.00 us 100.00 us 4 SETATTR > > > > > > 0.00 58.86 us 35.00 us 87.00 us 7 GETXATTR > > > > > > 0.00 60.57 us 55.00 us 69.00 us 7 FSTAT > > > > > > 0.00 35.53 us 24.00 us 53.00 us 15 STAT > > > > > > 0.00 57.12 us 39.00 us 72.00 us 24 INODELK > > > > > > 0.01 51.25 us 33.00 us 90.00 us 71 STATFS > > > > > > 0.01 52.09 us 32.00 us 259.00 us 92 LK > > > > > > 0.01 175.87 us 112.00 us 691.00 us 30 UNLINK > > > > > > 0.01 41.84 us 16.00 us 579.00 us 175 FLUSH > > > > > > 0.02 56.23 us 25.00 us 125.00 us 158 FINODELK > > > > > > 0.02 51.59 us 23.00 us 97.00 us 216 ENTRYLK > > > > > > 0.02 85.10 us 51.00 us 168.00 us 133 OPEN > > > > > > 0.03 200.55 us 150.00 us 554.00 us 78 CREATE > > > > > > 0.06 163.03 us 38.00 us 11184.00 us 204 READLINK > > > > > > 0.07 1022.60 us 67.00 us 24446.00 us 40 READ > > > > > > 0.08 45104.00 us 45104.00 us 45104.00 us 1 TRUNCATE > > > > > > 0.09 59.52 us 3.00 us 145.00 us 843 OPENDIR > > > > > > 0.13 936.73 us 56.00 us 12251.00 us 78 WRITE > > > > > > 0.19 687.17 us 88.00 us 85925.00 us 157 FXATTROP > > > > > > 22.66 1881.29 us 23.00 us 6083.00 us 6698 READDIRP > > > > > > 76.59 754.63 us 33.00 us 209082.00 us 56449 LOOKUP > > > > > > Duration: 269283 seconds > > > > > > Data Read: 376540270509 bytes > > > > > > Data Written: 21968880277 bytes > > > > > > Interval 4 Stats: > > > > > > Block Size: 16b+ 32b+ 64b+ > > > > > > No. of Reads: 0 0 0 > > > > > > No. of Writes: 1 3 12 > > > > > > Block Size: 128b+ 1024b+ 32768b+ > > > > > > No. of Reads: 0 0 1 > > > > > > No. of Writes: 8 16 0 > > > > > > Block Size: 65536b+ 131072b+ > > > > > > No. of Reads: 2 9 > > > > > > No. of Writes: 0 0 > > > > > > %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop > > > > > > --------- ----------- ----------- ----------- ------------ ---- > > > > > > 0.00 0.00 us 0.00 us 0.00 us 36 FORGET > > > > > > 0.00 0.00 us 0.00 us 0.00 us 76 RELEASE > > > > > > 0.00 0.00 us 0.00 us 0.00 us 408 RELEASEDIR > > > > > > 0.00 95.00 us 95.00 us 95.00 us 1 SETATTR > > > > > > 0.00 48.00 us 39.00 us 57.00 us 2 GETXATTR > > > > > > 0.00 50.00 us 39.00 us 61.00 us 2 READDIR > > > > > > 0.00 35.50 us 27.00 us 53.00 us 4 STAT > > > > > > 0.00 63.33 us 55.00 us 69.00 us 3 FSTAT > > > > > > 0.00 61.11 us 50.00 us 72.00 us 9 INODELK > > > > > > 0.01 50.41 us 34.00 us 90.00 us 37 STATFS > > > > > > 0.01 160.08 us 125.00 us 213.00 us 12 UNLINK > > > > > > 0.01 37.19 us 18.00 us 71.00 us 59 FLUSH > > > > > > 0.01 48.46 us 32.00 us 78.00 us 48 LK > > > > > > 0.01 84.92 us 57.00 us 118.00 us 37 OPEN > > > > > > 0.02 54.89 us 25.00 us 89.00 us 80 FINODELK > > > > > > 0.02 50.57 us 27.00 us 74.00 us 104 ENTRYLK > > > > > > 0.03 66.88 us 38.00 us 100.00 us 113 READLINK > > > > > > 0.03 196.90 us 150.00 us 554.00 us 40 CREATE > > > > > > 0.04 139.06 us 88.00 us 233.00 us 80 FXATTROP > > > > > > 0.10 58.95 us 8.00 us 93.00 us 408 OPENDIR > > > > > > 0.11 1675.56 us 67.00 us 24446.00 us 16 READ > > > > > > 0.17 1066.62 us 62.00 us 8506.00 us 40 WRITE > > > > > > 23.14 1889.02 us 23.00 us 3775.00 us 3073 READDIRP > > > > > > 76.29 773.40 us 33.00 us 135928.00 us 24746 LOOKUP > > > > > > Duration: 48 seconds > > > > > > Data Read: 1446611 bytes > > > > > > Data Written: 22682 bytes > > > > > > Brick: 192.168.10.90:/export/brick1 > > > > > > ----------------------------------- > > > > > > Cumulative Stats: > > > > > > Block Size: 2b+ 8b+ 16b+ > > > > > > No. of Reads: 0 6 32 > > > > > > No. of Writes: 1 1929 2605 > > > > > > Block Size: 32b+ 64b+ 128b+ > > > > > > No. of Reads: 21 112 169 > > > > > > No. of Writes: 6986 12080 26020 > > > > > > Block Size: 256b+ 512b+ 1024b+ > > > > > > No. of Reads: 85 299 809 > > > > > > No. of Writes: 743 18584 7233 > > > > > > Block Size: 2048b+ 4096b+ 8192b+ > > > > > > No. of Reads: 790 1419 2885 > > > > > > No. of Writes: 1651 10647 16901 > > > > > > Block Size: 16384b+ 32768b+ 65536b+ > > > > > > No. of Reads: 4588 5858 32434 > > > > > > No. of Writes: 2974 5758 3387 > > > > > > Block Size: 131072b+ > > > > > > No. of Reads: 455218 > > > > > > No. of Writes: 4601 > > > > > > %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop > > > > > > --------- ----------- ----------- ----------- ------------ ---- > > > > > > 0.00 0.00 us 0.00 us 0.00 us 604813 FORGET > > > > > > 0.00 0.00 us 0.00 us 0.00 us 1410912 RELEASE > > > > > > 0.00 0.00 us 0.00 us 0.00 us 990360 RELEASEDIR > > > > > > 0.00 44.33 us 16.00 us 62.00 us 6 READDIR > > > > > > 0.00 147.50 us 136.00 us 159.00 us 2 XATTROP > > > > > > 0.00 100.25 us 86.00 us 125.00 us 4 SETATTR > > > > > > 0.00 60.11 us 39.00 us 92.00 us 9 GETXATTR > > > > > > 0.00 60.19 us 41.00 us 84.00 us 16 FSTAT > > > > > > 0.00 45.50 us 17.00 us 182.00 us 38 STAT > > > > > > 0.00 51.37 us 31.00 us 81.00 us 71 STATFS > > > > > > 0.00 49.05 us 30.00 us 79.00 us 92 LK > > > > > > 0.00 168.87 us 115.00 us 287.00 us 30 UNLINK > > > > > > 0.01 38.74 us 15.00 us 150.00 us 150 FLUSH > > > > > > 0.01 54.76 us 30.00 us 236.00 us 158 FINODELK > > > > > > 0.01 87.17 us 49.00 us 352.00 us 106 OPEN > > > > > > 0.01 52.56 us 27.00 us 91.00 us 216 ENTRYLK > > > > > > 0.02 174.25 us 73.00 us 5551.00 us 157 FXATTROP > > > > > > 0.03 35786.00 us 35786.00 us 35786.00 us 1 TRUNCATE > > > > > > 0.05 62.12 us 3.00 us 1628.00 us 843 OPENDIR > > > > > > 0.10 685.58 us 41.00 us 51432.00 us 157 READLINK > > > > > > 0.36 6483.54 us 59.00 us 47225.00 us 63 READ > > > > > > 0.58 8304.77 us 60.00 us 185503.00 us 78 WRITE > > > > > > 0.67 9646.06 us 150.00 us 686579.00 us 78 CREATE > > > > > > 1.82 85338.88 us 25.00 us 1502989.00 us 24 INODELK > > > > > > 14.62 1920.48 us 23.00 us 9822.00 us 8567 READDIRP > > > > > > 81.71 1635.58 us 30.00 us 828308.00 us 56208 LOOKUP > > > > > > Duration: 76104 seconds > > > > > > Data Read: 63115678019 bytes > > > > > > Data Written: 1489698979 bytes > > > > > > Interval 4 Stats: > > > > > > Block Size: 16b+ 32b+ 64b+ > > > > > > No. of Reads: 0 0 0 > > > > > > No. of Writes: 1 3 12 > > > > > > Block Size: 128b+ 1024b+ 4096b+ > > > > > > No. of Reads: 0 0 1 > > > > > > No. of Writes: 8 16 0 > > > > > > Block Size: 8192b+ 32768b+ 65536b+ > > > > > > No. of Reads: 1 1 6 > > > > > > No. of Writes: 0 0 0 > > > > > > Block Size: 131072b+ > > > > > > No. of Reads: 10 > > > > > > No. of Writes: 0 > > > > > > %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop > > > > > > --------- ----------- ----------- ----------- ------------ ---- > > > > > > 0.00 0.00 us 0.00 us 0.00 us 29 FORGET > > > > > > 0.00 0.00 us 0.00 us 0.00 us 76 RELEASE > > > > > > 0.00 0.00 us 0.00 us 0.00 us 409 RELEASEDIR > > > > > > 0.00 125.00 us 125.00 us 125.00 us 1 SETATTR > > > > > > 0.00 43.75 us 16.00 us 60.00 us 4 READDIR > > > > > > 0.00 61.80 us 39.00 us 92.00 us 5 GETXATTR > > > > > > 0.00 65.71 us 44.00 us 76.00 us 7 FSTAT > > > > > > 0.00 48.50 us 24.00 us 182.00 us 28 STAT > > > > > > 0.00 50.08 us 31.00 us 70.00 us 37 STATFS > > > > > > 0.00 165.25 us 119.00 us 238.00 us 12 UNLINK > > > > > > 0.00 39.66 us 19.00 us 62.00 us 59 FLUSH > > > > > > 0.00 51.12 us 33.00 us 79.00 us 48 LK > > > > > > 0.01 82.78 us 64.00 us 104.00 us 37 OPEN > > > > > > 0.01 56.45 us 30.00 us 236.00 us 80 FINODELK > > > > > > 0.01 52.17 us 28.00 us 84.00 us 104 ENTRYLK > > > > > > 0.02 154.12 us 88.00 us 1577.00 us 80 FXATTROP > > > > > > 0.05 65.00 us 3.00 us 1628.00 us 409 OPENDIR > > > > > > 0.09 909.64 us 43.00 us 51432.00 us 61 READLINK > > > > > > 0.60 9748.22 us 59.00 us 47225.00 us 36 READ > > > > > > 0.86 12688.17 us 60.00 us 185503.00 us 40 WRITE > > > > > > 0.92 60488.89 us 50.00 us 543898.00 us 9 INODELK > > > > > > 1.26 18623.40 us 161.00 us 686579.00 us 40 CREATE > > > > > > 19.51 1951.60 us 25.00 us 9822.00 us 5889 READDIRP > > > > > > 76.64 1823.96 us 30.00 us 318395.00 us 24747 LOOKUP > > > > > > Duration: 48 seconds > > > > > > Data Read: 1931486 bytes > > > > > > Data Written: 22682 bytes > > > > > > Again, any assistance on this issue would be much appreciated. > > >> > > Thank you. > > >> > > On Sat, Dec 20, 2014 at 11:58 PM, Kyle Harris < kyle.harris98 at gmail.com > > > > wrote: > > >> > > > Hello All, > > > > > >> > > > I have been experiencing difficulty with high process utilization on > > > > the > > > > glusterfs and glusterfsd processes. I thought I had it tracked down to > > > > Apache looking for files on the cluster drive that don't exist. I have > > > > been > > > > working to clean that up and now I have discovered another problem. > > > > > >> > > > When I reboot one of the nodes in my 3-node replicated cluster, as soon > > > > as > > > > it > > > > comes back up I see a bunch of entries such as follows: > > > > > >> > > > [2014-12-21 05:33:49.873989] I [dict.c:370:dict_get] (--> > > > > /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7fe3bd0d2420] > > > > (--> > > > > /usr/lib64/libglusterfs.so.0(dict_get+0x4f)[0x7fe3bd0cbbdf] (--> > > > > /usr/lib64/glusterfs/3.6.1/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x1e1)[0x7fe3aad3ada1] > > > > (--> > > > > /usr/lib64/glusterfs/3.6.1/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x113)[0x7fe3aaf524e3] > > > > (--> > > > > /usr/lib64/glusterfs/3.6.1/xlator/performance/md-cache.so(mdc_lookup+0x300)[0x7fe3ab167310] > > > > ))))) 0-dict: !this || key=system.posix_acl_access > > > > > >> > > > While this is occurring, the process utilization for glusterfs and > > > > glusterfsd > > > > go sky high. Eventually they calm down on their own but the servers are > > > > virtually unusable until they do. Note this is on Centos 6.6 with > > > > Gluster > > > > 3.6.1 and kernel 2.6.32 using a fuse mount with each of the 3 servers > > > > having > > > > the client mounted for hosting web sites via Apache. There are many > > > > small > > > > files and some are php. I have not done any tuning to the default > > > > gluster > > > > parameters. > > > > > >> > > > Can someone tell me why this occurs after one of the nodes is rebooted > > > > and > > > > perhaps lend some guidance on what can be done to resolve this issue? > > > > > >> > > > Thank you. > > > > > >> > > > -- > > > > > > > > > > Kyle > > > > > >> > > -- > > >> > > _______________________________________________ > > > > > > Gluster-users mailing list > > > > > > Gluster-users at gluster.org > > > > > > http://www.gluster.org/mailman/listinfo/gluster-users > > > > ---------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20141222/5cb169e1/attachment-0001.html>