All the volume commands are taking several minutes to complete. Here's the
profiler's output:
Node3's brick is the one that was replaced. It's replica is node6
Brick: node1:/bricks/brickname
---------------------------------------------------
Cumulative Stats:
Block Size: 8b+ 16b+
32b+
No. of Reads: 0 13
2
No. of Writes: 60 0
141
Block Size: 64b+ 128b+
256b+
No. of Reads: 3 26
17
No. of Writes: 325 87
738
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 99 114
222
No. of Writes: 877 343
128
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 110 5
2
No. of Writes: 29401 78829
1448
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 19 33
34679
No. of Writes: 6233 22903
41202
Block Size: 262144b+ 524288b+
1048576b+
No. of Reads: 0 1
513
No. of Writes: 1 0
105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 126138
FORGET
0.00 0.00 us 0.00 us 0.00 us 141671
RELEASE
0.00 0.00 us 0.00 us 0.00 us 117718
RELEASEDIR
0.02 43.00 us 21.00 us 80.00 us 17
STAT
0.02 51.62 us 27.00 us 131.00 us 21
STATFS
0.09 42.78 us 11.00 us 1640.00 us 95
FLUSH
0.21 50.01 us 23.00 us 567.00 us 189
ENTRYLK
0.21 50.01 us 19.00 us 291.00 us 190
FINODELK
0.56 8578.33 us 53.00 us 25625.00 us 3
GETXATTR
0.62 148.89 us 71.00 us 2761.00 us 190
XATTROP
0.70 168.11 us 83.00 us 1019.00 us 190
FXATTROP
0.91 219.84 us 47.00 us 13732.00 us 190
SETATTR
1.21 71.74 us 16.00 us 11516.00 us 775
INODELK
1.47 354.88 us 56.00 us 22669.00 us 190
REMOVEXATTR
2.60 1254.69 us 122.00 us 12514.00 us 95
WRITE
3.25 194.10 us 51.00 us 48823.00 us 770
LOOKUP
88.15 43068.81 us 265.00 us 418819.00 us 94
CREATE
Duration: 644070 seconds
Data Read: 5089537361 bytes
Data Written: 9083513756 bytes
Interval 0 Stats:
Block Size: 8b+ 16b+
32b+
No. of Reads: 0 13
2
No. of Writes: 60 0
141
Block Size: 64b+ 128b+
256b+
No. of Reads: 3 26
17
No. of Writes: 325 87
738
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 99 114
222
No. of Writes: 877 343
128
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 110 5
2
No. of Writes: 29401 78829
1448
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 19 33
34679
No. of Writes: 6233 22903
41202
Block Size: 262144b+ 524288b+
1048576b+
No. of Reads: 0 1
513
No. of Writes: 1 0
105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 126138
FORGET
0.00 0.00 us 0.00 us 0.00 us 141671
RELEASE
0.00 0.00 us 0.00 us 0.00 us 117718
RELEASEDIR
0.02 43.00 us 21.00 us 80.00 us 17
STAT
0.02 51.62 us 27.00 us 131.00 us 21
STATFS
0.09 42.78 us 11.00 us 1640.00 us 95
FLUSH
0.21 50.01 us 23.00 us 567.00 us 189
ENTRYLK
0.21 50.01 us 19.00 us 291.00 us 190
FINODELK
0.56 8578.33 us 53.00 us 25625.00 us 3
GETXATTR
0.62 148.89 us 71.00 us 2761.00 us 190
XATTROP
0.70 168.11 us 83.00 us 1019.00 us 190
FXATTROP
0.91 219.84 us 47.00 us 13732.00 us 190
SETATTR
1.21 71.74 us 16.00 us 11516.00 us 775
INODELK
1.47 354.88 us 56.00 us 22669.00 us 190
REMOVEXATTR
2.60 1254.69 us 122.00 us 12514.00 us 95
WRITE
3.25 194.10 us 51.00 us 48823.00 us 770
LOOKUP
88.15 43068.81 us 265.00 us 418819.00 us 94
CREATE
Duration: 644070 seconds
Data Read: 5089537361 bytes
Data Written: 9083513756 bytes
Brick: node2:/bricks/brickname
-----------------------------------------------
Cumulative Stats:
Block Size: 8b+ 16b+
32b+
No. of Reads: 0 1
23
No. of Writes: 60 0
141
Block Size: 64b+ 128b+
256b+
No. of Reads: 45 47
363
No. of Writes: 325 87
738
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 515 37
42
No. of Writes: 877 343
128
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 17 0
1
No. of Writes: 29401 78829
1448
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 15 39
67031
No. of Writes: 6233 22903
41202
Block Size: 262144b+ 1048576b+
No. of Reads: 1 105
No. of Writes: 1 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 126136
FORGET
0.00 0.00 us 0.00 us 0.00 us 141671
RELEASE
0.00 0.00 us 0.00 us 0.00 us 117718
RELEASEDIR
0.02 74.00 us 74.00 us 74.00 us 1
STAT
0.15 196.00 us 141.00 us 288.00 us 3
GETXATTR
0.59 105.23 us 44.00 us 146.00 us 22
STATFS
2.05 83.88 us 11.00 us 137.00 us 96
FLUSH
4.79 98.61 us 20.00 us 146.00 us 191
ENTRYLK
5.03 102.93 us 22.00 us 158.00 us 192
FINODELK
5.52 226.07 us 136.00 us 295.00 us 96
WRITE
6.31 261.08 us 150.00 us 345.00 us 95
CREATE
6.72 137.53 us 50.00 us 214.00 us 192
SETATTR
7.68 157.14 us 75.00 us 237.00 us 192
REMOVEXATTR
8.13 166.49 us 81.00 us 282.00 us 192
XATTROP
8.26 169.01 us 76.00 us 275.00 us 192
FXATTROP
17.22 86.46 us 16.00 us 216.00 us 783
INODELK
27.54 138.09 us 43.00 us 266.00 us 784
LOOKUP
Duration: 644071 seconds
Data Read: 8902589511 bytes
Data Written: 9083513756 bytes
Interval 0 Stats:
Block Size: 8b+ 16b+
32b+
No. of Reads: 0 1
23
No. of Writes: 60 0
141
Block Size: 64b+ 128b+
256b+
No. of Reads: 45 47
363
No. of Writes: 325 87
738
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 515 37
42
No. of Writes: 877 343
128
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 17 0
1
No. of Writes: 29401 78829
1448
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 15 39
67031
No. of Writes: 6233 22903
41202
Block Size: 262144b+ 1048576b+
No. of Reads: 1 105
No. of Writes: 1 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 126136
FORGET
0.00 0.00 us 0.00 us 0.00 us 141671
RELEASE
0.00 0.00 us 0.00 us 0.00 us 117718
RELEASEDIR
0.02 74.00 us 74.00 us 74.00 us 1
STAT
0.15 196.00 us 141.00 us 288.00 us 3
GETXATTR
0.59 105.23 us 44.00 us 146.00 us 22
STATFS
2.05 83.88 us 11.00 us 137.00 us 96
FLUSH
4.79 98.61 us 20.00 us 146.00 us 191
ENTRYLK
5.03 102.93 us 22.00 us 158.00 us 192
FINODELK
5.52 226.07 us 136.00 us 295.00 us 96
WRITE
6.31 261.08 us 150.00 us 345.00 us 95
CREATE
6.72 137.53 us 50.00 us 214.00 us 192
SETATTR
7.68 157.14 us 75.00 us 237.00 us 192
REMOVEXATTR
8.13 166.49 us 81.00 us 282.00 us 192
XATTROP
8.26 169.01 us 76.00 us 275.00 us 192
FXATTROP
17.22 86.46 us 16.00 us 216.00 us 783
INODELK
27.54 138.09 us 43.00 us 266.00 us 784
LOOKUP
Duration: 644071 seconds
Data Read: 8902589511 bytes
Data Written: 9083513756 bytes
Brick: node3(sink):/bricks/brickname
------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+
4b+
No. of Reads: 0 0
0
No. of Writes: 11 26
125
Block Size: 8b+ 16b+
32b+
No. of Reads: 0 0
0
No. of Writes: 829 2341
9599
Block Size: 64b+ 128b+
256b+
No. of Reads: 0 0
0
No. of Writes: 12674 9229
27346
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 2 10
0
No. of Writes: 23414 28727
18372
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 1 0
0
No. of Writes: 48347 92134
9675
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 2 11
50
No. of Writes: 11717 24948
1022216
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 13805186
FORGET
0.00 0.00 us 0.00 us 0.00 us 17674891
RELEASE
0.00 0.00 us 0.00 us 0.00 us 218068
RELEASEDIR
0.00 24.00 us 24.00 us 24.00 us 1
OPENDIR
0.00 13.06 us 8.00 us 36.00 us 16
STAT
0.00 19.22 us 9.00 us 46.00 us 18
STATFS
0.00 45.54 us 22.00 us 82.00 us 13
SETXATTR
0.00 120.93 us 77.00 us 156.00 us 14
XATTROP
0.00 11.01 us 7.00 us 68.00 us 156
ENTRYLK
0.01 283.15 us 246.00 us 504.00 us 59
READDIR
0.02 899.19 us 39.00 us 17518.00 us 26
SETATTR
0.02 2004.85 us 38.00 us 10406.00 us 13
WRITE
0.02 2022.77 us 24.00 us 21677.00 us 13
REMOVEXATTR
0.03 2965.85 us 34.00 us 37695.00 us 13
FTRUNCATE
0.04 3691.62 us 31.00 us 18386.00 us 13
FLUSH
0.31 2105.65 us 23.00 us 57417.00 us 177
OPEN
0.43 2603.12 us 57.00 us 73929.00 us 202
FXATTROP
0.46 3030.94 us 7.00 us 87892.00 us 186
FSTAT
1.07 33.17 us 18.00 us 17545.00 us 39491
GETXATTR
1.31 123033.46 us 75610.00 us 269227.00 us 13
FSYNC
46.14 704.24 us 6.00 us 268597.00 us 79866
INODELK
50.13 699.93 us 20.00 us 267607.00 us 87307
LOOKUP
Duration: 112674 seconds
Data Read: 7441454 bytes
Data Written: 138577629032 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+
4b+
No. of Reads: 0 0
0
No. of Writes: 11 26
125
Block Size: 8b+ 16b+
32b+
No. of Reads: 0 0
0
No. of Writes: 829 2341
9599
Block Size: 64b+ 128b+
256b+
No. of Reads: 0 0
0
No. of Writes: 12674 9229
27346
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 2 10
0
No. of Writes: 23414 28727
18372
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 1 0
0
No. of Writes: 48347 92134
9675
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 2 11
50
No. of Writes: 11717 24948
1022216
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 13805186
FORGET
0.00 0.00 us 0.00 us 0.00 us 17674862
RELEASE
0.00 0.00 us 0.00 us 0.00 us 218068
RELEASEDIR
0.00 24.00 us 24.00 us 24.00 us 1
OPENDIR
0.00 13.06 us 8.00 us 36.00 us 16
STAT
0.00 19.22 us 9.00 us 46.00 us 18
STATFS
0.00 45.54 us 22.00 us 82.00 us 13
SETXATTR
0.00 120.93 us 77.00 us 156.00 us 14
XATTROP
0.00 11.01 us 7.00 us 68.00 us 156
ENTRYLK
0.01 283.15 us 246.00 us 504.00 us 59
READDIR
0.02 899.19 us 39.00 us 17518.00 us 26
SETATTR
0.02 2004.85 us 38.00 us 10406.00 us 13
WRITE
0.02 2022.77 us 24.00 us 21677.00 us 13
REMOVEXATTR
0.03 2965.85 us 34.00 us 37695.00 us 13
FTRUNCATE
0.04 3691.62 us 31.00 us 18386.00 us 13
FLUSH
0.31 2105.65 us 23.00 us 57417.00 us 177
OPEN
0.43 2603.12 us 57.00 us 73929.00 us 202
FXATTROP
0.46 3030.94 us 7.00 us 87892.00 us 186
FSTAT
1.07 33.17 us 18.00 us 17545.00 us 39491
GETXATTR
1.31 123033.46 us 75610.00 us 269227.00 us 13
FSYNC
46.14 704.24 us 6.00 us 268597.00 us 79866
INODELK
50.13 699.93 us 20.00 us 267607.00 us 87307
LOOKUP
Duration: 112674 seconds
Data Read: 7441454 bytes
Data Written: 138577629032 bytes
Brick: node4:/bricks/brickname
-----------------------------------------------
Cumulative Stats:
Block Size: 8b+ 32b+
64b+
No. of Reads: 0 9
24
No. of Writes: 62 128
335
Block Size: 128b+ 256b+
512b+
No. of Reads: 21 177
257
No. of Writes: 186 779
885
Block Size: 1024b+ 2048b+
4096b+
No. of Reads: 30 14
7
No. of Writes: 286 101
29410
Block Size: 8192b+ 16384b+
32768b+
No. of Reads: 0 9
0
No. of Writes: 79662 1379
6187
Block Size: 65536b+ 131072b+
262144b+
No. of Reads: 29 3924
0
No. of Writes: 22467 32424
1
Block Size: 1048576b+
No. of Reads: 0
No. of Writes: 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 126295
FORGET
0.00 0.00 us 0.00 us 0.00 us 141875
RELEASE
0.00 0.00 us 0.00 us 0.00 us 117220
RELEASEDIR
0.12 119.50 us 102.00 us 147.00 us 4
GETXATTR
0.19 68.18 us 42.00 us 109.00 us 11
STAT
0.44 92.16 us 19.00 us 141.00 us 19
STATFS
1.20 68.03 us 11.00 us 120.00 us 71
FLUSH
2.93 83.02 us 18.00 us 136.00 us 142
ENTRYLK
3.15 89.18 us 16.00 us 160.00 us 142
FINODELK
3.40 192.82 us 76.00 us 271.00 us 71
WRITE
4.04 114.43 us 35.00 us 204.00 us 142
SETATTR
4.87 138.05 us 49.00 us 222.00 us 142
REMOVEXATTR
5.63 159.52 us 56.00 us 262.00 us 142
FXATTROP
10.68 73.81 us 11.00 us 202.00 us 582
INODELK
19.90 1127.35 us 116.00 us 27717.00 us 71
CREATE
21.67 613.82 us 46.00 us 65260.00 us 142
XATTROP
21.80 130.68 us 35.00 us 241.00 us 671
LOOKUP
Duration: 458509 seconds
Data Read: 517180943 bytes
Data Written: 7895152670 bytes
Interval 0 Stats:
Block Size: 8b+ 32b+
64b+
No. of Reads: 0 9
24
No. of Writes: 62 128
335
Block Size: 128b+ 256b+
512b+
No. of Reads: 21 177
257
No. of Writes: 186 779
885
Block Size: 1024b+ 2048b+
4096b+
No. of Reads: 30 14
7
No. of Writes: 286 101
29410
Block Size: 8192b+ 16384b+
32768b+
No. of Reads: 0 9
0
No. of Writes: 79662 1379
6187
Block Size: 65536b+ 131072b+
262144b+
No. of Reads: 29 3924
0
No. of Writes: 22467 32424
1
Block Size: 1048576b+
No. of Reads: 0
No. of Writes: 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 126295
FORGET
0.00 0.00 us 0.00 us 0.00 us 141875
RELEASE
0.00 0.00 us 0.00 us 0.00 us 117220
RELEASEDIR
0.12 119.50 us 102.00 us 147.00 us 4
GETXATTR
0.19 68.18 us 42.00 us 109.00 us 11
STAT
0.44 92.16 us 19.00 us 141.00 us 19
STATFS
1.20 68.03 us 11.00 us 120.00 us 71
FLUSH
2.93 83.02 us 18.00 us 136.00 us 142
ENTRYLK
3.15 89.18 us 16.00 us 160.00 us 142
FINODELK
3.40 192.82 us 76.00 us 271.00 us 71
WRITE
4.04 114.43 us 35.00 us 204.00 us 142
SETATTR
4.87 138.05 us 49.00 us 222.00 us 142
REMOVEXATTR
5.63 159.52 us 56.00 us 262.00 us 142
FXATTROP
10.68 73.81 us 11.00 us 202.00 us 582
INODELK
19.90 1127.35 us 116.00 us 27717.00 us 71
CREATE
21.67 613.82 us 46.00 us 65260.00 us 142
XATTROP
21.80 130.68 us 35.00 us 241.00 us 671
LOOKUP
Duration: 458509 seconds
Data Read: 517180943 bytes
Data Written: 7895152670 bytes
Brick: node5:/bricks/brickname
------------------------------------------------
Cumulative Stats:
Block Size: 8b+ 16b+
32b+
No. of Reads: 0 14
9
No. of Writes: 62 0
128
Block Size: 64b+ 128b+
256b+
No. of Reads: 23 56
225
No. of Writes: 335 186
779
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 357 106
233
No. of Writes: 885 286
102
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 128 11
15
No. of Writes: 29410 79662
1379
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 16 34
28965
No. of Writes: 6191 22467
32424
Block Size: 262144b+ 524288b+
1048576b+
No. of Reads: 5 3
984
No. of Writes: 1 0
105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 126301
FORGET
0.00 0.00 us 0.00 us 0.00 us 141880
RELEASE
0.00 0.00 us 0.00 us 0.00 us 117718
RELEASEDIR
0.01 51.75 us 41.00 us 69.00 us 4
GETXATTR
0.02 59.50 us 35.00 us 108.00 us 4
STAT
0.05 44.06 us 28.00 us 118.00 us 16
STATFS
0.10 24.78 us 15.00 us 99.00 us 59
FLUSH
0.34 41.01 us 24.00 us 107.00 us 118
ENTRYLK
0.37 44.43 us 25.00 us 156.00 us 118
FINODELK
0.76 90.88 us 70.00 us 183.00 us 118
SETATTR
0.81 193.88 us 162.00 us 283.00 us 59
WRITE
0.98 116.89 us 85.00 us 212.00 us 118
REMOVEXATTR
1.10 131.18 us 86.00 us 219.00 us 118
FXATTROP
1.52 44.02 us 24.00 us 1004.00 us 487
INODELK
3.35 86.29 us 52.00 us 183.00 us 549
LOOKUP
4.21 504.03 us 83.00 us 43099.00 us 118
XATTROP
86.39 20696.02 us 207.00 us 69802.00 us 59
CREATE
Duration: 644071 seconds
Data Read: 4837930222 bytes
Data Written: 7895351133 bytes
Interval 0 Stats:
Block Size: 8b+ 16b+
32b+
No. of Reads: 0 14
9
No. of Writes: 62 0
128
Block Size: 64b+ 128b+
256b+
No. of Reads: 23 56
225
No. of Writes: 335 186
779
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 357 106
233
No. of Writes: 885 286
102
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 128 11
15
No. of Writes: 29410 79662
1379
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 16 34
28965
No. of Writes: 6191 22467
32424
Block Size: 262144b+ 524288b+
1048576b+
No. of Reads: 5 3
984
No. of Writes: 1 0
105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 126301
FORGET
0.00 0.00 us 0.00 us 0.00 us 141880
RELEASE
0.00 0.00 us 0.00 us 0.00 us 117718
RELEASEDIR
0.01 51.75 us 41.00 us 69.00 us 4
GETXATTR
0.02 59.50 us 35.00 us 108.00 us 4
STAT
0.05 44.06 us 28.00 us 118.00 us 16
STATFS
0.10 24.78 us 15.00 us 99.00 us 59
FLUSH
0.34 41.01 us 24.00 us 107.00 us 118
ENTRYLK
0.37 44.43 us 25.00 us 156.00 us 118
FINODELK
0.76 90.88 us 70.00 us 183.00 us 118
SETATTR
0.81 193.88 us 162.00 us 283.00 us 59
WRITE
0.98 116.89 us 85.00 us 212.00 us 118
REMOVEXATTR
1.10 131.18 us 86.00 us 219.00 us 118
FXATTROP
1.52 44.02 us 24.00 us 1004.00 us 487
INODELK
3.35 86.29 us 52.00 us 183.00 us 549
LOOKUP
4.21 504.03 us 83.00 us 43099.00 us 118
XATTROP
86.39 20696.02 us 207.00 us 69802.00 us 59
CREATE
Duration: 644071 seconds
Data Read: 4837930222 bytes
Data Written: 7895351133 bytes
Brick: node6(source):/bricks/brickname
--------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+
4b+
No. of Reads: 7 18
89
No. of Writes: 4 8
37
Block Size: 8b+ 16b+
32b+
No. of Reads: 727 2325
9459
No. of Writes: 108 54
188
Block Size: 64b+ 128b+
256b+
No. of Reads: 12419 9313
27616
No. of Writes: 360 85
772
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 23708 28691
18594
No. of Writes: 847 313
138
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 19484 12596
8458
No. of Writes: 29185 79632
1431
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 5695 5755
1062899
No. of Writes: 6168 19435
32017
Block Size: 262144b+ 1048576b+
No. of Reads: 0 0
No. of Writes: 1 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 13806534
FORGET
0.00 0.00 us 0.00 us 0.00 us 17813646
RELEASE
0.00 0.00 us 0.00 us 0.00 us 223324
RELEASEDIR
0.00 560.00 us 560.00 us 560.00 us 1
ENTRYLK
0.00 3901.00 us 3901.00 us 3901.00 us 1
SETXATTR
0.00 4010.00 us 4010.00 us 4010.00 us 1
REMOVEXATTR
0.01 62446.08 us 8.00 us 365433.00 us 13
FLUSH
0.01 93887.77 us 52.00 us 588566.00 us 13
SETATTR
0.03 10772.83 us 28.00 us 1121761.00 us 253
GETXATTR
0.04 3190096.00 us 3190096.00 us 3190096.00 us 1
READDIR
0.09 558307.69 us 179931.00 us 3188951.00 us 13
READ
0.11 616756.00 us 74.00 us 7307745.00 us 14
XATTROP
0.12 4754785.50 us 48.00 us 9509523.00 us 2
OPENDIR
0.15 1799185.00 us 2310.00 us 5023537.00 us 7
STATFS
0.16 68757.98 us 10.00 us 872148.00 us 189
FSTAT
0.31 143533.93 us 42.00 us 7002195.00 us 174
OPEN
0.40 160262.95 us 661.00 us 2825083.00 us 202
READDIRP
1.55 624450.87 us 31.00 us 7397432.00 us 203
FXATTROP
22.43 212161.62 us 12.00 us 7397413.00 us 8639
INODELK
74.60 541421.09 us 63.00 us 14463033.00 us 11261
LOOKUP
Duration: 644071 seconds
Data Read: 140706386722 bytes
Data Written: 7549422894 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+
4b+
No. of Reads: 7 18
89
No. of Writes: 4 8
37
Block Size: 8b+ 16b+
32b+
No. of Reads: 727 2325
9459
No. of Writes: 108 54
188
Block Size: 64b+ 128b+
256b+
No. of Reads: 12419 9313
27616
No. of Writes: 360 85
772
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 23708 28691
18594
No. of Writes: 847 313
138
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 19484 12596
8458
No. of Writes: 29185 79632
1431
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 5695 5755
1062899
No. of Writes: 6168 19435
32017
Block Size: 262144b+ 1048576b+
No. of Reads: 0 0
No. of Writes: 1 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 13806534
FORGET
0.00 0.00 us 0.00 us 0.00 us 17813657
RELEASE
0.00 0.00 us 0.00 us 0.00 us 223324
RELEASEDIR
0.00 560.00 us 560.00 us 560.00 us 1
ENTRYLK
0.00 3901.00 us 3901.00 us 3901.00 us 1
SETXATTR
0.00 4010.00 us 4010.00 us 4010.00 us 1
REMOVEXATTR
0.01 62446.08 us 8.00 us 365433.00 us 13
FLUSH
0.01 93887.77 us 52.00 us 588566.00 us 13
SETATTR
0.03 10772.83 us 28.00 us 1121761.00 us 253
GETXATTR
0.04 3190096.00 us 3190096.00 us 3190096.00 us 1
READDIR
0.09 558307.69 us 179931.00 us 3188951.00 us 13
READ
0.11 616756.00 us 74.00 us 7307745.00 us 14
XATTROP
0.12 4754785.50 us 48.00 us 9509523.00 us 2
OPENDIR
0.15 1799185.00 us 2310.00 us 5023537.00 us 7
STATFS
0.16 68757.98 us 10.00 us 872148.00 us 189
FSTAT
0.31 143533.93 us 42.00 us 7002195.00 us 174
OPEN
0.40 160262.95 us 661.00 us 2825083.00 us 202
READDIRP
1.55 624450.87 us 31.00 us 7397432.00 us 203
FXATTROP
22.43 212161.62 us 12.00 us 7397413.00 us 8639
INODELK
74.60 541421.09 us 63.00 us 14463033.00 us 11261
LOOKUP
Duration: 644071 seconds
Data Read: 140706386722 bytes
Data Written: 7549422894 bytes
On Fri, Aug 7, 2015 at 12:17 AM, Ravishankar N <ravishankar at redhat.com>
wrote:
>
>
> On 08/07/2015 12:11 PM, Prasun Gera wrote:
>
> No, no noticeable difference. Still very high, possibly higher than
> before.
>
>
> I was guessing that the cpu usage could be because of the diff algorithm
> which computes checksums (which is a cpu intensive task). That doesn't
> seem to be the case. Could you do a volume profile and see the FOPS that
> are happening on the bricks and share the result?
> 1.gluster volume profile <volname> start
> 2. gluster volume profile <volname> info
> 3. wait 10-15 seconds
> 4.gluster volume profile <volname> info
>
>
>
> The system has come down to a crawl. It's difficult to even ssh or run
any
> commands on the terminal. Do you make anything of the logs ? The brick log
> is just a giant alternating stream of those two lines I mentioned earlier.
>
>
>
>
> On Thu, Aug 6, 2015 at 10:10 PM, Ravishankar N <ravishankar at
redhat.com>
> wrote:
>
>>
>>
>> On 08/07/2015 01:33 AM, Prasun Gera wrote:
>>
>>> I replaced the brick in a node in my 3x2 dist+repl volume (RHS 3).
I'm
>>> seeing that the heal process, which should essentially be a dump
from the
>>> working replica to the newly added one is taking exceptionally
long. It has
>>> moved ~100 G over a day on a 1Gigabit network. The CPU usage on
both the
>>> nodes of the replica has been pretty high.
>>>
>>
>> Does setting `cluster.data-self-heal-algorithm` to full make a
difference
>> in the cpu usage?
>>
>>
>> I also think that nagios is making it worse. The heal is slow enough as
>>> it is, and nagios keeps triggering heal info, which I think never
>>> completes. I also see my logs filling up These are some of the log
contents
>>> which I got by running tail on them:
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150807/e1290e72/attachment.html>
So the nodes 3 and 6 seem to indicate inode-locks and lookups are of the highest latency. This only seems to confirm self-heals are happening. If you are unable to use the system because of this, you could try killing the self-heal daemons on both these nodes (kill `pgrep -f glustershd`) to stop heals. You can then do a lookup of the files from the mount, which will also trigger heals. Restart the selfheal daemons (with `gluster vol start volname force`) when you think you can spare the volume for heals again. The sooner the better though. For the brick log errors, we are suspecting it could be something related to selinux. Can you tell what kind of data is there in your volume? - no. of files, avg. file size, brick size, network connection speed etc? Perhaps we can try to reproduce the issue and identify the bottle neck. Thanks, Ravi On 08/07/2015 01:27 PM, Prasun Gera wrote:> All the volume commands are taking several minutes to complete. Here's > the profiler's output: > Node3's brick is the one that was replaced. It's replica is node6 > > > Brick: node1:/bricks/brickname > --------------------------------------------------- > Cumulative Stats: > Block Size: 8b+ 16b+ 32b+ > No. of Reads: 0 13 2 > No. of Writes: 60 0 141 > Block Size: 64b+ 128b+ 256b+ > No. of Reads: 3 26 17 > No. of Writes: 325 87 738 > Block Size: 512b+ 1024b+ 2048b+ > No. of Reads: 99 114 222 > No. of Writes: 877 343 128 > Block Size: 4096b+ 8192b+ 16384b+ > No. of Reads: 110 5 2 > No. of Writes: 29401 78829 1448 > Block Size: 32768b+ 65536b+ 131072b+ > No. of Reads: 19 33 34679 > No. of Writes: 6233 22903 41202 > Block Size: 262144b+ 524288b+ 1048576b+ > No. of Reads: 0 1 513 > No. of Writes: 1 0 105 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 126138 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 141671 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 117718 > RELEASEDIR > 0.02 43.00 us 21.00 us 80.00 us 17 > STAT > 0.02 51.62 us 27.00 us 131.00 us 21 > STATFS > 0.09 42.78 us 11.00 us 1640.00 us 95 > FLUSH > 0.21 50.01 us 23.00 us 567.00 us 189 > ENTRYLK > 0.21 50.01 us 19.00 us 291.00 us 190 > FINODELK > 0.56 8578.33 us 53.00 us 25625.00 us 3 > GETXATTR > 0.62 148.89 us 71.00 us 2761.00 us 190 > XATTROP > 0.70 168.11 us 83.00 us 1019.00 us 190 > FXATTROP > 0.91 219.84 us 47.00 us 13732.00 us 190 > SETATTR > 1.21 71.74 us 16.00 us 11516.00 us 775 > INODELK > 1.47 354.88 us 56.00 us 22669.00 us 190 > REMOVEXATTR > 2.60 1254.69 us 122.00 us 12514.00 us 95 > WRITE > 3.25 194.10 us 51.00 us 48823.00 us 770 > LOOKUP > 88.15 43068.81 us 265.00 us 418819.00 us 94 > CREATE > Duration: 644070 seconds > Data Read: 5089537361 bytes > Data Written: 9083513756 bytes > Interval 0 Stats: > Block Size: 8b+ 16b+ 32b+ > No. of Reads: 0 13 2 > No. of Writes: 60 0 141 > Block Size: 64b+ 128b+ 256b+ > No. of Reads: 3 26 17 > No. of Writes: 325 87 738 > Block Size: 512b+ 1024b+ 2048b+ > No. of Reads: 99 114 222 > No. of Writes: 877 343 128 > Block Size: 4096b+ 8192b+ 16384b+ > No. of Reads: 110 5 2 > No. of Writes: 29401 78829 1448 > Block Size: 32768b+ 65536b+ 131072b+ > No. of Reads: 19 33 34679 > No. of Writes: 6233 22903 41202 > Block Size: 262144b+ 524288b+ 1048576b+ > No. of Reads: 0 1 513 > No. of Writes: 1 0 105 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 126138 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 141671 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 117718 > RELEASEDIR > 0.02 43.00 us 21.00 us 80.00 us 17 > STAT > 0.02 51.62 us 27.00 us 131.00 us 21 > STATFS > 0.09 42.78 us 11.00 us 1640.00 us 95 > FLUSH > 0.21 50.01 us 23.00 us 567.00 us 189 > ENTRYLK > 0.21 50.01 us 19.00 us 291.00 us 190 > FINODELK > 0.56 8578.33 us 53.00 us 25625.00 us 3 > GETXATTR > 0.62 148.89 us 71.00 us 2761.00 us 190 > XATTROP > 0.70 168.11 us 83.00 us 1019.00 us 190 > FXATTROP > 0.91 219.84 us 47.00 us 13732.00 us 190 > SETATTR > 1.21 71.74 us 16.00 us 11516.00 us 775 > INODELK > 1.47 354.88 us 56.00 us 22669.00 us 190 > REMOVEXATTR > 2.60 1254.69 us 122.00 us 12514.00 us 95 > WRITE > 3.25 194.10 us 51.00 us 48823.00 us 770 > LOOKUP > 88.15 43068.81 us 265.00 us 418819.00 us 94 > CREATE > Duration: 644070 seconds > Data Read: 5089537361 bytes > Data Written: 9083513756 bytes > Brick: node2:/bricks/brickname > ----------------------------------------------- > Cumulative Stats: > Block Size: 8b+ 16b+ 32b+ > No. of Reads: 0 1 23 > No. of Writes: 60 0 141 > Block Size: 64b+ 128b+ 256b+ > No. of Reads: 45 47 363 > No. of Writes: 325 87 738 > Block Size: 512b+ 1024b+ 2048b+ > No. of Reads: 515 37 42 > No. of Writes: 877 343 128 > Block Size: 4096b+ 8192b+ 16384b+ > No. of Reads: 17 0 1 > No. of Writes: 29401 78829 1448 > Block Size: 32768b+ 65536b+ 131072b+ > No. of Reads: 15 39 67031 > No. of Writes: 6233 22903 41202 > Block Size: 262144b+ 1048576b+ > No. of Reads: 1 105 > No. of Writes: 1 105 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 126136 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 141671 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 117718 > RELEASEDIR > 0.02 74.00 us 74.00 us 74.00 us 1 > STAT > 0.15 196.00 us 141.00 us 288.00 us 3 > GETXATTR > 0.59 105.23 us 44.00 us 146.00 us 22 > STATFS > 2.05 83.88 us 11.00 us 137.00 us 96 > FLUSH > 4.79 98.61 us 20.00 us 146.00 us 191 > ENTRYLK > 5.03 102.93 us 22.00 us 158.00 us 192 > FINODELK > 5.52 226.07 us 136.00 us 295.00 us 96 > WRITE > 6.31 261.08 us 150.00 us 345.00 us 95 > CREATE > 6.72 137.53 us 50.00 us 214.00 us 192 > SETATTR > 7.68 157.14 us 75.00 us 237.00 us 192 > REMOVEXATTR > 8.13 166.49 us 81.00 us 282.00 us 192 > XATTROP > 8.26 169.01 us 76.00 us 275.00 us 192 > FXATTROP > 17.22 86.46 us 16.00 us 216.00 us 783 > INODELK > 27.54 138.09 us 43.00 us 266.00 us 784 > LOOKUP > Duration: 644071 seconds > Data Read: 8902589511 bytes > Data Written: 9083513756 bytes > Interval 0 Stats: > Block Size: 8b+ 16b+ 32b+ > No. of Reads: 0 1 23 > No. of Writes: 60 0 141 > Block Size: 64b+ 128b+ 256b+ > No. of Reads: 45 47 363 > No. of Writes: 325 87 738 > Block Size: 512b+ 1024b+ 2048b+ > No. of Reads: 515 37 42 > No. of Writes: 877 343 128 > Block Size: 4096b+ 8192b+ 16384b+ > No. of Reads: 17 0 1 > No. of Writes: 29401 78829 1448 > Block Size: 32768b+ 65536b+ 131072b+ > No. of Reads: 15 39 67031 > No. of Writes: 6233 22903 41202 > Block Size: 262144b+ 1048576b+ > No. of Reads: 1 105 > No. of Writes: 1 105 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 126136 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 141671 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 117718 > RELEASEDIR > 0.02 74.00 us 74.00 us 74.00 us 1 > STAT > 0.15 196.00 us 141.00 us 288.00 us 3 > GETXATTR > 0.59 105.23 us 44.00 us 146.00 us 22 > STATFS > 2.05 83.88 us 11.00 us 137.00 us 96 > FLUSH > 4.79 98.61 us 20.00 us 146.00 us 191 > ENTRYLK > 5.03 102.93 us 22.00 us 158.00 us 192 > FINODELK > 5.52 226.07 us 136.00 us 295.00 us 96 > WRITE > 6.31 261.08 us 150.00 us 345.00 us 95 > CREATE > 6.72 137.53 us 50.00 us 214.00 us 192 > SETATTR > 7.68 157.14 us 75.00 us 237.00 us 192 > REMOVEXATTR > 8.13 166.49 us 81.00 us 282.00 us 192 > XATTROP > 8.26 169.01 us 76.00 us 275.00 us 192 > FXATTROP > 17.22 86.46 us 16.00 us 216.00 us 783 > INODELK > 27.54 138.09 us 43.00 us 266.00 us 784 > LOOKUP > Duration: 644071 seconds > Data Read: 8902589511 bytes > Data Written: 9083513756 bytes > Brick: node3(sink):/bricks/brickname > ------------------------------------------------ > Cumulative Stats: > Block Size: 1b+ 2b+ 4b+ > No. of Reads: 0 0 0 > No. of Writes: 11 26 125 > Block Size: 8b+ 16b+ 32b+ > No. of Reads: 0 0 0 > No. of Writes: 829 2341 9599 > Block Size: 64b+ 128b+ 256b+ > No. of Reads: 0 0 0 > No. of Writes: 12674 9229 27346 > Block Size: 512b+ 1024b+ 2048b+ > No. of Reads: 2 10 0 > No. of Writes: 23414 28727 18372 > Block Size: 4096b+ 8192b+ 16384b+ > No. of Reads: 1 0 0 > No. of Writes: 48347 92134 9675 > Block Size: 32768b+ 65536b+ 131072b+ > No. of Reads: 2 11 50 > No. of Writes: 11717 24948 1022216 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 13805186 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 17674891 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 218068 > RELEASEDIR > 0.00 24.00 us 24.00 us 24.00 us 1 > OPENDIR > 0.00 13.06 us 8.00 us 36.00 us 16 > STAT > 0.00 19.22 us 9.00 us 46.00 us 18 > STATFS > 0.00 45.54 us 22.00 us 82.00 us 13 > SETXATTR > 0.00 120.93 us 77.00 us 156.00 us 14 > XATTROP > 0.00 11.01 us 7.00 us 68.00 us 156 > ENTRYLK > 0.01 283.15 us 246.00 us 504.00 us 59 > READDIR > 0.02 899.19 us 39.00 us 17518.00 us 26 > SETATTR > 0.02 2004.85 us 38.00 us 10406.00 us 13 > WRITE > 0.02 2022.77 us 24.00 us 21677.00 us 13 > REMOVEXATTR > 0.03 2965.85 us 34.00 us 37695.00 us 13 > FTRUNCATE > 0.04 3691.62 us 31.00 us 18386.00 us 13 > FLUSH > 0.31 2105.65 us 23.00 us 57417.00 us 177 > OPEN > 0.43 2603.12 us 57.00 us 73929.00 us 202 > FXATTROP > 0.46 3030.94 us 7.00 us 87892.00 us 186 > FSTAT > 1.07 33.17 us 18.00 us 17545.00 us 39491 > GETXATTR > 1.31 123033.46 us 75610.00 us 269227.00 us 13 > FSYNC > 46.14 704.24 us 6.00 us 268597.00 us 79866 > INODELK > 50.13 699.93 us 20.00 us 267607.00 us 87307 > LOOKUP > Duration: 112674 seconds > Data Read: 7441454 bytes > Data Written: 138577629032 bytes > Interval 0 Stats: > Block Size: 1b+ 2b+ 4b+ > No. of Reads: 0 0 0 > No. of Writes: 11 26 125 > Block Size: 8b+ 16b+ 32b+ > No. of Reads: 0 0 0 > No. of Writes: 829 2341 9599 > Block Size: 64b+ 128b+ 256b+ > No. of Reads: 0 0 0 > No. of Writes: 12674 9229 27346 > Block Size: 512b+ 1024b+ 2048b+ > No. of Reads: 2 10 0 > No. of Writes: 23414 28727 18372 > Block Size: 4096b+ 8192b+ 16384b+ > No. of Reads: 1 0 0 > No. of Writes: 48347 92134 9675 > Block Size: 32768b+ 65536b+ 131072b+ > No. of Reads: 2 11 50 > No. of Writes: 11717 24948 1022216 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 13805186 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 17674862 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 218068 > RELEASEDIR > 0.00 24.00 us 24.00 us 24.00 us 1 > OPENDIR > 0.00 13.06 us 8.00 us 36.00 us 16 > STAT > 0.00 19.22 us 9.00 us 46.00 us 18 > STATFS > 0.00 45.54 us 22.00 us 82.00 us 13 > SETXATTR > 0.00 120.93 us 77.00 us 156.00 us 14 > XATTROP > 0.00 11.01 us 7.00 us 68.00 us 156 > ENTRYLK > 0.01 283.15 us 246.00 us 504.00 us 59 > READDIR > 0.02 899.19 us 39.00 us 17518.00 us 26 > SETATTR > 0.02 2004.85 us 38.00 us 10406.00 us 13 > WRITE > 0.02 2022.77 us 24.00 us 21677.00 us 13 > REMOVEXATTR > 0.03 2965.85 us 34.00 us 37695.00 us 13 > FTRUNCATE > 0.04 3691.62 us 31.00 us 18386.00 us 13 > FLUSH > 0.31 2105.65 us 23.00 us 57417.00 us 177 > OPEN > 0.43 2603.12 us 57.00 us 73929.00 us 202 > FXATTROP > 0.46 3030.94 us 7.00 us 87892.00 us 186 > FSTAT > 1.07 33.17 us 18.00 us 17545.00 us 39491 > GETXATTR > 1.31 123033.46 us 75610.00 us 269227.00 us 13 > FSYNC > 46.14 704.24 us 6.00 us 268597.00 us 79866 > INODELK > 50.13 699.93 us 20.00 us 267607.00 us 87307 > LOOKUP > Duration: 112674 seconds > Data Read: 7441454 bytes > Data Written: 138577629032 bytes > Brick: node4:/bricks/brickname > ----------------------------------------------- > Cumulative Stats: > Block Size: 8b+ 32b+ 64b+ > No. of Reads: 0 9 24 > No. of Writes: 62 128 335 > Block Size: 128b+ 256b+ 512b+ > No. of Reads: 21 177 257 > No. of Writes: 186 779 885 > Block Size: 1024b+ 2048b+ 4096b+ > No. of Reads: 30 14 7 > No. of Writes: 286 101 29410 > Block Size: 8192b+ 16384b+ 32768b+ > No. of Reads: 0 9 0 > No. of Writes: 79662 1379 6187 > Block Size: 65536b+ 131072b+ 262144b+ > No. of Reads: 29 3924 0 > No. of Writes: 22467 32424 1 > Block Size: 1048576b+ > No. of Reads: 0 > No. of Writes: 105 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 126295 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 141875 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 117220 > RELEASEDIR > 0.12 119.50 us 102.00 us 147.00 us 4 > GETXATTR > 0.19 68.18 us 42.00 us 109.00 us 11 > STAT > 0.44 92.16 us 19.00 us 141.00 us 19 > STATFS > 1.20 68.03 us 11.00 us 120.00 us 71 > FLUSH > 2.93 83.02 us 18.00 us 136.00 us 142 > ENTRYLK > 3.15 89.18 us 16.00 us 160.00 us 142 > FINODELK > 3.40 192.82 us 76.00 us 271.00 us 71 > WRITE > 4.04 114.43 us 35.00 us 204.00 us 142 > SETATTR > 4.87 138.05 us 49.00 us 222.00 us 142 > REMOVEXATTR > 5.63 159.52 us 56.00 us 262.00 us 142 > FXATTROP > 10.68 73.81 us 11.00 us 202.00 us 582 > INODELK > 19.90 1127.35 us 116.00 us 27717.00 us 71 > CREATE > 21.67 613.82 us 46.00 us 65260.00 us 142 > XATTROP > 21.80 130.68 us 35.00 us 241.00 us 671 > LOOKUP > Duration: 458509 seconds > Data Read: 517180943 bytes > Data Written: 7895152670 bytes > Interval 0 Stats: > Block Size: 8b+ 32b+ 64b+ > No. of Reads: 0 9 24 > No. of Writes: 62 128 335 > Block Size: 128b+ 256b+ 512b+ > No. of Reads: 21 177 257 > No. of Writes: 186 779 885 > Block Size: 1024b+ 2048b+ 4096b+ > No. of Reads: 30 14 7 > No. of Writes: 286 101 29410 > Block Size: 8192b+ 16384b+ 32768b+ > No. of Reads: 0 9 0 > No. of Writes: 79662 1379 6187 > Block Size: 65536b+ 131072b+ 262144b+ > No. of Reads: 29 3924 0 > No. of Writes: 22467 32424 1 > Block Size: 1048576b+ > No. of Reads: 0 > No. of Writes: 105 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 126295 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 141875 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 117220 > RELEASEDIR > 0.12 119.50 us 102.00 us 147.00 us 4 > GETXATTR > 0.19 68.18 us 42.00 us 109.00 us 11 > STAT > 0.44 92.16 us 19.00 us 141.00 us 19 > STATFS > 1.20 68.03 us 11.00 us 120.00 us 71 > FLUSH > 2.93 83.02 us 18.00 us 136.00 us 142 > ENTRYLK > 3.15 89.18 us 16.00 us 160.00 us 142 > FINODELK > 3.40 192.82 us 76.00 us 271.00 us 71 > WRITE > 4.04 114.43 us 35.00 us 204.00 us 142 > SETATTR > 4.87 138.05 us 49.00 us 222.00 us 142 > REMOVEXATTR > 5.63 159.52 us 56.00 us 262.00 us 142 > FXATTROP > 10.68 73.81 us 11.00 us 202.00 us 582 > INODELK > 19.90 1127.35 us 116.00 us 27717.00 us 71 > CREATE > 21.67 613.82 us 46.00 us 65260.00 us 142 > XATTROP > 21.80 130.68 us 35.00 us 241.00 us 671 > LOOKUP > Duration: 458509 seconds > Data Read: 517180943 bytes > Data Written: 7895152670 bytes > Brick: node5:/bricks/brickname > ------------------------------------------------ > Cumulative Stats: > Block Size: 8b+ 16b+ 32b+ > No. of Reads: 0 14 9 > No. of Writes: 62 0 128 > Block Size: 64b+ 128b+ 256b+ > No. of Reads: 23 56 225 > No. of Writes: 335 186 779 > Block Size: 512b+ 1024b+ 2048b+ > No. of Reads: 357 106 233 > No. of Writes: 885 286 102 > Block Size: 4096b+ 8192b+ 16384b+ > No. of Reads: 128 11 15 > No. of Writes: 29410 79662 1379 > Block Size: 32768b+ 65536b+ 131072b+ > No. of Reads: 16 34 28965 > No. of Writes: 6191 22467 32424 > Block Size: 262144b+ 524288b+ 1048576b+ > No. of Reads: 5 3 984 > No. of Writes: 1 0 105 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 126301 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 141880 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 117718 > RELEASEDIR > 0.01 51.75 us 41.00 us 69.00 us 4 > GETXATTR > 0.02 59.50 us 35.00 us 108.00 us 4 > STAT > 0.05 44.06 us 28.00 us 118.00 us 16 > STATFS > 0.10 24.78 us 15.00 us 99.00 us 59 > FLUSH > 0.34 41.01 us 24.00 us 107.00 us 118 > ENTRYLK > 0.37 44.43 us 25.00 us 156.00 us 118 > FINODELK > 0.76 90.88 us 70.00 us 183.00 us 118 > SETATTR > 0.81 193.88 us 162.00 us 283.00 us 59 > WRITE > 0.98 116.89 us 85.00 us 212.00 us 118 > REMOVEXATTR > 1.10 131.18 us 86.00 us 219.00 us 118 > FXATTROP > 1.52 44.02 us 24.00 us 1004.00 us 487 > INODELK > 3.35 86.29 us 52.00 us 183.00 us 549 > LOOKUP > 4.21 504.03 us 83.00 us 43099.00 us 118 > XATTROP > 86.39 20696.02 us 207.00 us 69802.00 us 59 > CREATE > Duration: 644071 seconds > Data Read: 4837930222 bytes > Data Written: 7895351133 bytes > Interval 0 Stats: > Block Size: 8b+ 16b+ 32b+ > No. of Reads: 0 14 9 > No. of Writes: 62 0 128 > Block Size: 64b+ 128b+ 256b+ > No. of Reads: 23 56 225 > No. of Writes: 335 186 779 > Block Size: 512b+ 1024b+ 2048b+ > No. of Reads: 357 106 233 > No. of Writes: 885 286 102 > Block Size: 4096b+ 8192b+ 16384b+ > No. of Reads: 128 11 15 > No. of Writes: 29410 79662 1379 > Block Size: 32768b+ 65536b+ 131072b+ > No. of Reads: 16 34 28965 > No. of Writes: 6191 22467 32424 > Block Size: 262144b+ 524288b+ 1048576b+ > No. of Reads: 5 3 984 > No. of Writes: 1 0 105 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 126301 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 141880 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 117718 > RELEASEDIR > 0.01 51.75 us 41.00 us 69.00 us 4 > GETXATTR > 0.02 59.50 us 35.00 us 108.00 us 4 > STAT > 0.05 44.06 us 28.00 us 118.00 us 16 > STATFS > 0.10 24.78 us 15.00 us 99.00 us 59 > FLUSH > 0.34 41.01 us 24.00 us 107.00 us 118 > ENTRYLK > 0.37 44.43 us 25.00 us 156.00 us 118 > FINODELK > 0.76 90.88 us 70.00 us 183.00 us 118 > SETATTR > 0.81 193.88 us 162.00 us 283.00 us 59 > WRITE > 0.98 116.89 us 85.00 us 212.00 us 118 > REMOVEXATTR > 1.10 131.18 us 86.00 us 219.00 us 118 > FXATTROP > 1.52 44.02 us 24.00 us 1004.00 us 487 > INODELK > 3.35 86.29 us 52.00 us 183.00 us 549 > LOOKUP > 4.21 504.03 us 83.00 us 43099.00 us 118 > XATTROP > 86.39 20696.02 us 207.00 us 69802.00 us 59 > CREATE > Duration: 644071 seconds > Data Read: 4837930222 bytes > Data Written: 7895351133 bytes > Brick: node6(source):/bricks/brickname > -------------------------------------------------- > Cumulative Stats: > Block Size: 1b+ 2b+ 4b+ > No. of Reads: 7 18 89 > No. of Writes: 4 8 37 > Block Size: 8b+ 16b+ 32b+ > No. of Reads: 727 2325 9459 > No. of Writes: 108 54 188 > Block Size: 64b+ 128b+ 256b+ > No. of Reads: 12419 9313 27616 > No. of Writes: 360 85 772 > Block Size: 512b+ 1024b+ 2048b+ > No. of Reads: 23708 28691 18594 > No. of Writes: 847 313 138 > Block Size: 4096b+ 8192b+ 16384b+ > No. of Reads: 19484 12596 8458 > No. of Writes: 29185 79632 1431 > Block Size: 32768b+ 65536b+ 131072b+ > No. of Reads: 5695 5755 1062899 > No. of Writes: 6168 19435 32017 > Block Size: 262144b+ 1048576b+ > No. of Reads: 0 0 > No. of Writes: 1 105 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 13806534 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 17813646 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 223324 > RELEASEDIR > 0.00 560.00 us 560.00 us 560.00 us 1 > ENTRYLK > 0.00 3901.00 us 3901.00 us 3901.00 us 1 > SETXATTR > 0.00 4010.00 us 4010.00 us 4010.00 us 1 > REMOVEXATTR > 0.01 62446.08 us 8.00 us 365433.00 us 13 > FLUSH > 0.01 93887.77 us 52.00 us 588566.00 us 13 > SETATTR > 0.03 10772.83 us 28.00 us 1121761.00 us 253 > GETXATTR > 0.04 3190096.00 us 3190096.00 us 3190096.00 us 1 > READDIR > 0.09 558307.69 us 179931.00 us 3188951.00 us 13 > READ > 0.11 616756.00 us 74.00 us 7307745.00 us 14 > XATTROP > 0.12 4754785.50 us 48.00 us 9509523.00 us 2 > OPENDIR > 0.15 1799185.00 us 2310.00 us 5023537.00 us 7 > STATFS > 0.16 68757.98 us 10.00 us 872148.00 us 189 > FSTAT > 0.31 143533.93 us 42.00 us 7002195.00 us 174 > OPEN > 0.40 160262.95 us 661.00 us 2825083.00 us 202 > READDIRP > 1.55 624450.87 us 31.00 us 7397432.00 us 203 > FXATTROP > 22.43 212161.62 us 12.00 us 7397413.00 us 8639 > INODELK > 74.60 541421.09 us 63.00 us 14463033.00 us 11261 > LOOKUP > Duration: 644071 seconds > Data Read: 140706386722 bytes > Data Written: 7549422894 bytes > Interval 0 Stats: > Block Size: 1b+ 2b+ 4b+ > No. of Reads: 7 18 89 > No. of Writes: 4 8 37 > Block Size: 8b+ 16b+ 32b+ > No. of Reads: 727 2325 9459 > No. of Writes: 108 54 188 > Block Size: 64b+ 128b+ 256b+ > No. of Reads: 12419 9313 27616 > No. of Writes: 360 85 772 > Block Size: 512b+ 1024b+ 2048b+ > No. of Reads: 23708 28691 18594 > No. of Writes: 847 313 138 > Block Size: 4096b+ 8192b+ 16384b+ > No. of Reads: 19484 12596 8458 > No. of Writes: 29185 79632 1431 > Block Size: 32768b+ 65536b+ 131072b+ > No. of Reads: 5695 5755 1062899 > No. of Writes: 6168 19435 32017 > Block Size: 262144b+ 1048576b+ > No. of Reads: 0 0 > No. of Writes: 1 105 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 13806534 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 17813657 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 223324 > RELEASEDIR > 0.00 560.00 us 560.00 us 560.00 us 1 > ENTRYLK > 0.00 3901.00 us 3901.00 us 3901.00 us 1 > SETXATTR > 0.00 4010.00 us 4010.00 us 4010.00 us 1 > REMOVEXATTR > 0.01 62446.08 us 8.00 us 365433.00 us 13 > FLUSH > 0.01 93887.77 us 52.00 us 588566.00 us 13 > SETATTR > 0.03 10772.83 us 28.00 us 1121761.00 us 253 > GETXATTR > 0.04 3190096.00 us 3190096.00 us 3190096.00 us 1 > READDIR > 0.09 558307.69 us 179931.00 us 3188951.00 us 13 > READ > 0.11 616756.00 us 74.00 us 7307745.00 us 14 > XATTROP > 0.12 4754785.50 us 48.00 us 9509523.00 us 2 > OPENDIR > 0.15 1799185.00 us 2310.00 us 5023537.00 us 7 > STATFS > 0.16 68757.98 us 10.00 us 872148.00 us 189 > FSTAT > 0.31 143533.93 us 42.00 us 7002195.00 us 174 > OPEN > 0.40 160262.95 us 661.00 us 2825083.00 us 202 > READDIRP > 1.55 624450.87 us 31.00 us 7397432.00 us 203 > FXATTROP > 22.43 212161.62 us 12.00 us 7397413.00 us 8639 > INODELK > 74.60 541421.09 us 63.00 us 14463033.00 us 11261 > LOOKUP > Duration: 644071 seconds > Data Read: 140706386722 bytes > Data Written: 7549422894 bytes > > On Fri, Aug 7, 2015 at 12:17 AM, Ravishankar N <ravishankar at redhat.com > <mailto:ravishankar at redhat.com>> wrote: > > > > On 08/07/2015 12:11 PM, Prasun Gera wrote: >> No, no noticeable difference. Still very high, possibly higher >> than before. > > I was guessing that the cpu usage could be because of the diff > algorithm which computes checksums (which is a cpu intensive > task). That doesn't seem to be the case. Could you do a volume > profile and see the FOPS that are happening on the bricks and > share the result? > 1.gluster volume profile <volname> start > 2. gluster volume profile <volname> info > 3. wait 10-15 seconds > 4.gluster volume profile <volname> info > > > >> The system has come down to a crawl. It's difficult to even ssh >> or run any commands on the terminal. Do you make anything of the >> logs ? The brick log is just a giant alternating stream of those >> two lines I mentioned earlier. > > >> >> On Thu, Aug 6, 2015 at 10:10 PM, Ravishankar N >> <ravishankar at redhat.com <mailto:ravishankar at redhat.com>> wrote: >> >> >> >> On 08/07/2015 01:33 AM, Prasun Gera wrote: >> >> I replaced the brick in a node in my 3x2 dist+repl volume >> (RHS 3). I'm seeing that the heal process, which should >> essentially be a dump from the working replica to the >> newly added one is taking exceptionally long. It has >> moved ~100 G over a day on a 1Gigabit network. The CPU >> usage on both the nodes of the replica has been pretty high. >> >> >> Does setting `cluster.data-self-heal-algorithm` to full make >> a difference in the cpu usage? >> >> >> I also think that nagios is making it worse. The heal is >> slow enough as it is, and nagios keeps triggering heal >> info, which I think never completes. I also see my logs >> filling up These are some of the log contents which I got >> by running tail on them: >> >> >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150807/93118fdf/attachment.html>