[Gluster-users] Self heal issues

Prasun Gera
2015-08-07T07:57:35Z

All the volume commands are taking several minutes to complete. Here's the
profiler's output:
Node3's brick is the one that was replaced. It's replica is node6


Brick: node1:/bricks/brickname
---------------------------------------------------
Cumulative Stats:
   Block Size:                  8b+                  16b+
 32b+
 No. of Reads:                    0                    13
  2
No. of Writes:                   60                     0
141

   Block Size:                 64b+                 128b+
256b+
 No. of Reads:                    3                    26
 17
No. of Writes:                  325                    87
738

   Block Size:                512b+                1024b+
 2048b+
 No. of Reads:                   99                   114
222
No. of Writes:                  877                   343
128

   Block Size:               4096b+                8192b+
16384b+
 No. of Reads:                  110                     5
  2
No. of Writes:                29401                 78829
 1448

   Block Size:              32768b+               65536b+
 131072b+
 No. of Reads:                   19                    33
34679
No. of Writes:                 6233                 22903
41202

   Block Size:             262144b+              524288b+
1048576b+
 No. of Reads:                    0                     1
513
No. of Writes:                    1                     0
105

 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us         126138
 FORGET
      0.00       0.00 us       0.00 us       0.00 us         141671
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117718
 RELEASEDIR
      0.02      43.00 us      21.00 us      80.00 us             17
 STAT
      0.02      51.62 us      27.00 us     131.00 us             21
 STATFS
      0.09      42.78 us      11.00 us    1640.00 us             95
FLUSH
      0.21      50.01 us      23.00 us     567.00 us            189
ENTRYLK
      0.21      50.01 us      19.00 us     291.00 us            190
 FINODELK
      0.56    8578.33 us      53.00 us   25625.00 us              3
 GETXATTR
      0.62     148.89 us      71.00 us    2761.00 us            190
XATTROP
      0.70     168.11 us      83.00 us    1019.00 us            190
 FXATTROP
      0.91     219.84 us      47.00 us   13732.00 us            190
SETATTR
      1.21      71.74 us      16.00 us   11516.00 us            775
INODELK
      1.47     354.88 us      56.00 us   22669.00 us            190
REMOVEXATTR
      2.60    1254.69 us     122.00 us   12514.00 us             95
WRITE
      3.25     194.10 us      51.00 us   48823.00 us            770
 LOOKUP
     88.15   43068.81 us     265.00 us  418819.00 us             94
 CREATE

    Duration: 644070 seconds
   Data Read: 5089537361 bytes
Data Written: 9083513756 bytes

Interval 0 Stats:
   Block Size:                  8b+                  16b+
 32b+
 No. of Reads:                    0                    13
  2
No. of Writes:                   60                     0
141

   Block Size:                 64b+                 128b+
256b+
 No. of Reads:                    3                    26
 17
No. of Writes:                  325                    87
738

   Block Size:                512b+                1024b+
 2048b+
 No. of Reads:                   99                   114
222
No. of Writes:                  877                   343
128

   Block Size:               4096b+                8192b+
16384b+
 No. of Reads:                  110                     5
  2
No. of Writes:                29401                 78829
 1448

   Block Size:              32768b+               65536b+
 131072b+
 No. of Reads:                   19                    33
34679
No. of Writes:                 6233                 22903
41202

   Block Size:             262144b+              524288b+
1048576b+
 No. of Reads:                    0                     1
513
No. of Writes:                    1                     0
105

 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us         126138
 FORGET
      0.00       0.00 us       0.00 us       0.00 us         141671
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117718
 RELEASEDIR
      0.02      43.00 us      21.00 us      80.00 us             17
 STAT
      0.02      51.62 us      27.00 us     131.00 us             21
 STATFS
      0.09      42.78 us      11.00 us    1640.00 us             95
FLUSH
      0.21      50.01 us      23.00 us     567.00 us            189
ENTRYLK
      0.21      50.01 us      19.00 us     291.00 us            190
 FINODELK
      0.56    8578.33 us      53.00 us   25625.00 us              3
 GETXATTR
      0.62     148.89 us      71.00 us    2761.00 us            190
XATTROP
      0.70     168.11 us      83.00 us    1019.00 us            190
 FXATTROP
      0.91     219.84 us      47.00 us   13732.00 us            190
SETATTR
      1.21      71.74 us      16.00 us   11516.00 us            775
INODELK
      1.47     354.88 us      56.00 us   22669.00 us            190
REMOVEXATTR
      2.60    1254.69 us     122.00 us   12514.00 us             95
WRITE
      3.25     194.10 us      51.00 us   48823.00 us            770
 LOOKUP
     88.15   43068.81 us     265.00 us  418819.00 us             94
 CREATE

    Duration: 644070 seconds
   Data Read: 5089537361 bytes
Data Written: 9083513756 bytes

Brick: node2:/bricks/brickname
-----------------------------------------------
Cumulative Stats:
   Block Size:                  8b+                  16b+
 32b+
 No. of Reads:                    0                     1
 23
No. of Writes:                   60                     0
141

   Block Size:                 64b+                 128b+
256b+
 No. of Reads:                   45                    47
363
No. of Writes:                  325                    87
738

   Block Size:                512b+                1024b+
 2048b+
 No. of Reads:                  515                    37
 42
No. of Writes:                  877                   343
128

   Block Size:               4096b+                8192b+
16384b+
 No. of Reads:                   17                     0
  1
No. of Writes:                29401                 78829
 1448

   Block Size:              32768b+               65536b+
 131072b+
 No. of Reads:                   15                    39
67031
No. of Writes:                 6233                 22903
41202

   Block Size:             262144b+             1048576b+
 No. of Reads:                    1                   105
No. of Writes:                    1                   105
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us         126136
 FORGET
      0.00       0.00 us       0.00 us       0.00 us         141671
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117718
 RELEASEDIR
      0.02      74.00 us      74.00 us      74.00 us              1
 STAT
      0.15     196.00 us     141.00 us     288.00 us              3
 GETXATTR
      0.59     105.23 us      44.00 us     146.00 us             22
 STATFS
      2.05      83.88 us      11.00 us     137.00 us             96
FLUSH
      4.79      98.61 us      20.00 us     146.00 us            191
ENTRYLK
      5.03     102.93 us      22.00 us     158.00 us            192
 FINODELK
      5.52     226.07 us     136.00 us     295.00 us             96
WRITE
      6.31     261.08 us     150.00 us     345.00 us             95
 CREATE
      6.72     137.53 us      50.00 us     214.00 us            192
SETATTR
      7.68     157.14 us      75.00 us     237.00 us            192
REMOVEXATTR
      8.13     166.49 us      81.00 us     282.00 us            192
XATTROP
      8.26     169.01 us      76.00 us     275.00 us            192
 FXATTROP
     17.22      86.46 us      16.00 us     216.00 us            783
INODELK
     27.54     138.09 us      43.00 us     266.00 us            784
 LOOKUP

    Duration: 644071 seconds
   Data Read: 8902589511 bytes
Data Written: 9083513756 bytes

Interval 0 Stats:
   Block Size:                  8b+                  16b+
 32b+
 No. of Reads:                    0                     1
 23
No. of Writes:                   60                     0
141

   Block Size:                 64b+                 128b+
256b+
 No. of Reads:                   45                    47
363
No. of Writes:                  325                    87
738

   Block Size:                512b+                1024b+
 2048b+
 No. of Reads:                  515                    37
 42
No. of Writes:                  877                   343
128

   Block Size:               4096b+                8192b+
16384b+
 No. of Reads:                   17                     0
  1
No. of Writes:                29401                 78829
 1448

   Block Size:              32768b+               65536b+
 131072b+
 No. of Reads:                   15                    39
67031
No. of Writes:                 6233                 22903
41202

   Block Size:             262144b+             1048576b+
 No. of Reads:                    1                   105
No. of Writes:                    1                   105
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us         126136
 FORGET
      0.00       0.00 us       0.00 us       0.00 us         141671
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117718
 RELEASEDIR
      0.02      74.00 us      74.00 us      74.00 us              1
 STAT
      0.15     196.00 us     141.00 us     288.00 us              3
 GETXATTR
      0.59     105.23 us      44.00 us     146.00 us             22
 STATFS
      2.05      83.88 us      11.00 us     137.00 us             96
FLUSH
      4.79      98.61 us      20.00 us     146.00 us            191
ENTRYLK
      5.03     102.93 us      22.00 us     158.00 us            192
 FINODELK
      5.52     226.07 us     136.00 us     295.00 us             96
WRITE
      6.31     261.08 us     150.00 us     345.00 us             95
 CREATE
      6.72     137.53 us      50.00 us     214.00 us            192
SETATTR
      7.68     157.14 us      75.00 us     237.00 us            192
REMOVEXATTR
      8.13     166.49 us      81.00 us     282.00 us            192
XATTROP
      8.26     169.01 us      76.00 us     275.00 us            192
 FXATTROP
     17.22      86.46 us      16.00 us     216.00 us            783
INODELK
     27.54     138.09 us      43.00 us     266.00 us            784
 LOOKUP

    Duration: 644071 seconds
   Data Read: 8902589511 bytes
Data Written: 9083513756 bytes

Brick: node3(sink):/bricks/brickname
------------------------------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+
4b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   11                    26
125

   Block Size:                  8b+                  16b+
 32b+
 No. of Reads:                    0                     0
  0
No. of Writes:                  829                  2341
 9599

   Block Size:                 64b+                 128b+
256b+
 No. of Reads:                    0                     0
  0
No. of Writes:                12674                  9229
27346

   Block Size:                512b+                1024b+
 2048b+
 No. of Reads:                    2                    10
  0
No. of Writes:                23414                 28727
18372

   Block Size:               4096b+                8192b+
16384b+
 No. of Reads:                    1                     0
  0
No. of Writes:                48347                 92134
 9675

   Block Size:              32768b+               65536b+
 131072b+
 No. of Reads:                    2                    11
 50
No. of Writes:                11717                 24948
1022216

 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us       13805186
 FORGET
      0.00       0.00 us       0.00 us       0.00 us       17674891
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         218068
 RELEASEDIR
      0.00      24.00 us      24.00 us      24.00 us              1
OPENDIR
      0.00      13.06 us       8.00 us      36.00 us             16
 STAT
      0.00      19.22 us       9.00 us      46.00 us             18
 STATFS
      0.00      45.54 us      22.00 us      82.00 us             13
 SETXATTR
      0.00     120.93 us      77.00 us     156.00 us             14
XATTROP
      0.00      11.01 us       7.00 us      68.00 us            156
ENTRYLK
      0.01     283.15 us     246.00 us     504.00 us             59
READDIR
      0.02     899.19 us      39.00 us   17518.00 us             26
SETATTR
      0.02    2004.85 us      38.00 us   10406.00 us             13
WRITE
      0.02    2022.77 us      24.00 us   21677.00 us             13
REMOVEXATTR
      0.03    2965.85 us      34.00 us   37695.00 us             13
FTRUNCATE
      0.04    3691.62 us      31.00 us   18386.00 us             13
FLUSH
      0.31    2105.65 us      23.00 us   57417.00 us            177
 OPEN
      0.43    2603.12 us      57.00 us   73929.00 us            202
 FXATTROP
      0.46    3030.94 us       7.00 us   87892.00 us            186
FSTAT
      1.07      33.17 us      18.00 us   17545.00 us          39491
 GETXATTR
      1.31  123033.46 us   75610.00 us  269227.00 us             13
FSYNC
     46.14     704.24 us       6.00 us  268597.00 us          79866
INODELK
     50.13     699.93 us      20.00 us  267607.00 us          87307
 LOOKUP

    Duration: 112674 seconds
   Data Read: 7441454 bytes
Data Written: 138577629032 bytes

Interval 0 Stats:
   Block Size:                  1b+                   2b+
4b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   11                    26
125

   Block Size:                  8b+                  16b+
 32b+
 No. of Reads:                    0                     0
  0
No. of Writes:                  829                  2341
 9599

   Block Size:                 64b+                 128b+
256b+
 No. of Reads:                    0                     0
  0
No. of Writes:                12674                  9229
27346

   Block Size:                512b+                1024b+
 2048b+
 No. of Reads:                    2                    10
  0
No. of Writes:                23414                 28727
18372

   Block Size:               4096b+                8192b+
16384b+
 No. of Reads:                    1                     0
  0
No. of Writes:                48347                 92134
 9675

   Block Size:              32768b+               65536b+
 131072b+
 No. of Reads:                    2                    11
 50
No. of Writes:                11717                 24948
1022216

 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us       13805186
 FORGET
      0.00       0.00 us       0.00 us       0.00 us       17674862
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         218068
 RELEASEDIR
      0.00      24.00 us      24.00 us      24.00 us              1
OPENDIR
      0.00      13.06 us       8.00 us      36.00 us             16
 STAT
      0.00      19.22 us       9.00 us      46.00 us             18
 STATFS
      0.00      45.54 us      22.00 us      82.00 us             13
 SETXATTR
      0.00     120.93 us      77.00 us     156.00 us             14
XATTROP
      0.00      11.01 us       7.00 us      68.00 us            156
ENTRYLK
      0.01     283.15 us     246.00 us     504.00 us             59
READDIR
      0.02     899.19 us      39.00 us   17518.00 us             26
SETATTR
      0.02    2004.85 us      38.00 us   10406.00 us             13
WRITE
      0.02    2022.77 us      24.00 us   21677.00 us             13
REMOVEXATTR
      0.03    2965.85 us      34.00 us   37695.00 us             13
FTRUNCATE
      0.04    3691.62 us      31.00 us   18386.00 us             13
FLUSH
      0.31    2105.65 us      23.00 us   57417.00 us            177
 OPEN
      0.43    2603.12 us      57.00 us   73929.00 us            202
 FXATTROP
      0.46    3030.94 us       7.00 us   87892.00 us            186
FSTAT
      1.07      33.17 us      18.00 us   17545.00 us          39491
 GETXATTR
      1.31  123033.46 us   75610.00 us  269227.00 us             13
FSYNC
     46.14     704.24 us       6.00 us  268597.00 us          79866
INODELK
     50.13     699.93 us      20.00 us  267607.00 us          87307
 LOOKUP

    Duration: 112674 seconds
   Data Read: 7441454 bytes
Data Written: 138577629032 bytes

Brick: node4:/bricks/brickname
-----------------------------------------------
Cumulative Stats:
   Block Size:                  8b+                  32b+
 64b+
 No. of Reads:                    0                     9
 24
No. of Writes:                   62                   128
335

   Block Size:                128b+                 256b+
512b+
 No. of Reads:                   21                   177
257
No. of Writes:                  186                   779
885

   Block Size:               1024b+                2048b+
 4096b+
 No. of Reads:                   30                    14
  7
No. of Writes:                  286                   101
29410

   Block Size:               8192b+               16384b+
32768b+
 No. of Reads:                    0                     9
  0
No. of Writes:                79662                  1379
 6187

   Block Size:              65536b+              131072b+
 262144b+
 No. of Reads:                   29                  3924
  0
No. of Writes:                22467                 32424
  1

   Block Size:            1048576b+
 No. of Reads:                    0
No. of Writes:                  105
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us         126295
 FORGET
      0.00       0.00 us       0.00 us       0.00 us         141875
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117220
 RELEASEDIR
      0.12     119.50 us     102.00 us     147.00 us              4
 GETXATTR
      0.19      68.18 us      42.00 us     109.00 us             11
 STAT
      0.44      92.16 us      19.00 us     141.00 us             19
 STATFS
      1.20      68.03 us      11.00 us     120.00 us             71
FLUSH
      2.93      83.02 us      18.00 us     136.00 us            142
ENTRYLK
      3.15      89.18 us      16.00 us     160.00 us            142
 FINODELK
      3.40     192.82 us      76.00 us     271.00 us             71
WRITE
      4.04     114.43 us      35.00 us     204.00 us            142
SETATTR
      4.87     138.05 us      49.00 us     222.00 us            142
REMOVEXATTR
      5.63     159.52 us      56.00 us     262.00 us            142
 FXATTROP
     10.68      73.81 us      11.00 us     202.00 us            582
INODELK
     19.90    1127.35 us     116.00 us   27717.00 us             71
 CREATE
     21.67     613.82 us      46.00 us   65260.00 us            142
XATTROP
     21.80     130.68 us      35.00 us     241.00 us            671
 LOOKUP

    Duration: 458509 seconds
   Data Read: 517180943 bytes
Data Written: 7895152670 bytes

Interval 0 Stats:
   Block Size:                  8b+                  32b+
 64b+
 No. of Reads:                    0                     9
 24
No. of Writes:                   62                   128
335

   Block Size:                128b+                 256b+
512b+
 No. of Reads:                   21                   177
257
No. of Writes:                  186                   779
885

   Block Size:               1024b+                2048b+
 4096b+
 No. of Reads:                   30                    14
  7
No. of Writes:                  286                   101
29410

   Block Size:               8192b+               16384b+
32768b+
 No. of Reads:                    0                     9
  0
No. of Writes:                79662                  1379
 6187

   Block Size:              65536b+              131072b+
 262144b+
 No. of Reads:                   29                  3924
  0
No. of Writes:                22467                 32424
  1

   Block Size:            1048576b+
 No. of Reads:                    0
No. of Writes:                  105
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us         126295
 FORGET
      0.00       0.00 us       0.00 us       0.00 us         141875
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117220
 RELEASEDIR
      0.12     119.50 us     102.00 us     147.00 us              4
 GETXATTR
      0.19      68.18 us      42.00 us     109.00 us             11
 STAT
      0.44      92.16 us      19.00 us     141.00 us             19
 STATFS
      1.20      68.03 us      11.00 us     120.00 us             71
FLUSH
      2.93      83.02 us      18.00 us     136.00 us            142
ENTRYLK
      3.15      89.18 us      16.00 us     160.00 us            142
 FINODELK
      3.40     192.82 us      76.00 us     271.00 us             71
WRITE
      4.04     114.43 us      35.00 us     204.00 us            142
SETATTR
      4.87     138.05 us      49.00 us     222.00 us            142
REMOVEXATTR
      5.63     159.52 us      56.00 us     262.00 us            142
 FXATTROP
     10.68      73.81 us      11.00 us     202.00 us            582
INODELK
     19.90    1127.35 us     116.00 us   27717.00 us             71
 CREATE
     21.67     613.82 us      46.00 us   65260.00 us            142
XATTROP
     21.80     130.68 us      35.00 us     241.00 us            671
 LOOKUP

    Duration: 458509 seconds
   Data Read: 517180943 bytes
Data Written: 7895152670 bytes

Brick: node5:/bricks/brickname
------------------------------------------------
Cumulative Stats:
   Block Size:                  8b+                  16b+
 32b+
 No. of Reads:                    0                    14
  9
No. of Writes:                   62                     0
128

   Block Size:                 64b+                 128b+
256b+
 No. of Reads:                   23                    56
225
No. of Writes:                  335                   186
779

   Block Size:                512b+                1024b+
 2048b+
 No. of Reads:                  357                   106
233
No. of Writes:                  885                   286
102

   Block Size:               4096b+                8192b+
16384b+
 No. of Reads:                  128                    11
 15
No. of Writes:                29410                 79662
 1379

   Block Size:              32768b+               65536b+
 131072b+
 No. of Reads:                   16                    34
28965
No. of Writes:                 6191                 22467
32424

   Block Size:             262144b+              524288b+
1048576b+
 No. of Reads:                    5                     3
984
No. of Writes:                    1                     0
105

 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us         126301
 FORGET
      0.00       0.00 us       0.00 us       0.00 us         141880
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117718
 RELEASEDIR
      0.01      51.75 us      41.00 us      69.00 us              4
 GETXATTR
      0.02      59.50 us      35.00 us     108.00 us              4
 STAT
      0.05      44.06 us      28.00 us     118.00 us             16
 STATFS
      0.10      24.78 us      15.00 us      99.00 us             59
FLUSH
      0.34      41.01 us      24.00 us     107.00 us            118
ENTRYLK
      0.37      44.43 us      25.00 us     156.00 us            118
 FINODELK
      0.76      90.88 us      70.00 us     183.00 us            118
SETATTR
      0.81     193.88 us     162.00 us     283.00 us             59
WRITE
      0.98     116.89 us      85.00 us     212.00 us            118
REMOVEXATTR
      1.10     131.18 us      86.00 us     219.00 us            118
 FXATTROP
      1.52      44.02 us      24.00 us    1004.00 us            487
INODELK
      3.35      86.29 us      52.00 us     183.00 us            549
 LOOKUP
      4.21     504.03 us      83.00 us   43099.00 us            118
XATTROP
     86.39   20696.02 us     207.00 us   69802.00 us             59
 CREATE

    Duration: 644071 seconds
   Data Read: 4837930222 bytes
Data Written: 7895351133 bytes

Interval 0 Stats:
   Block Size:                  8b+                  16b+
 32b+
 No. of Reads:                    0                    14
  9
No. of Writes:                   62                     0
128

   Block Size:                 64b+                 128b+
256b+
 No. of Reads:                   23                    56
225
No. of Writes:                  335                   186
779

   Block Size:                512b+                1024b+
 2048b+
 No. of Reads:                  357                   106
233
No. of Writes:                  885                   286
102

   Block Size:               4096b+                8192b+
16384b+
 No. of Reads:                  128                    11
 15
No. of Writes:                29410                 79662
 1379

   Block Size:              32768b+               65536b+
 131072b+
 No. of Reads:                   16                    34
28965
No. of Writes:                 6191                 22467
32424

   Block Size:             262144b+              524288b+
1048576b+
 No. of Reads:                    5                     3
984
No. of Writes:                    1                     0
105

 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us         126301
 FORGET
      0.00       0.00 us       0.00 us       0.00 us         141880
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117718
 RELEASEDIR
      0.01      51.75 us      41.00 us      69.00 us              4
 GETXATTR
      0.02      59.50 us      35.00 us     108.00 us              4
 STAT
      0.05      44.06 us      28.00 us     118.00 us             16
 STATFS
      0.10      24.78 us      15.00 us      99.00 us             59
FLUSH
      0.34      41.01 us      24.00 us     107.00 us            118
ENTRYLK
      0.37      44.43 us      25.00 us     156.00 us            118
 FINODELK
      0.76      90.88 us      70.00 us     183.00 us            118
SETATTR
      0.81     193.88 us     162.00 us     283.00 us             59
WRITE
      0.98     116.89 us      85.00 us     212.00 us            118
REMOVEXATTR
      1.10     131.18 us      86.00 us     219.00 us            118
 FXATTROP
      1.52      44.02 us      24.00 us    1004.00 us            487
INODELK
      3.35      86.29 us      52.00 us     183.00 us            549
 LOOKUP
      4.21     504.03 us      83.00 us   43099.00 us            118
XATTROP
     86.39   20696.02 us     207.00 us   69802.00 us             59
 CREATE

    Duration: 644071 seconds
   Data Read: 4837930222 bytes
Data Written: 7895351133 bytes

Brick: node6(source):/bricks/brickname
--------------------------------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+
4b+
 No. of Reads:                    7                    18
 89
No. of Writes:                    4                     8
 37

   Block Size:                  8b+                  16b+
 32b+
 No. of Reads:                  727                  2325
 9459
No. of Writes:                  108                    54
188

   Block Size:                 64b+                 128b+
256b+
 No. of Reads:                12419                  9313
27616
No. of Writes:                  360                    85
772

   Block Size:                512b+                1024b+
 2048b+
 No. of Reads:                23708                 28691
18594
No. of Writes:                  847                   313
138

   Block Size:               4096b+                8192b+
16384b+
 No. of Reads:                19484                 12596
 8458
No. of Writes:                29185                 79632
 1431

   Block Size:              32768b+               65536b+
 131072b+
 No. of Reads:                 5695                  5755
1062899
No. of Writes:                 6168                 19435
32017

   Block Size:             262144b+             1048576b+
 No. of Reads:                    0                     0
No. of Writes:                    1                   105
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us       13806534
 FORGET
      0.00       0.00 us       0.00 us       0.00 us       17813646
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         223324
 RELEASEDIR
      0.00     560.00 us     560.00 us     560.00 us              1
ENTRYLK
      0.00    3901.00 us    3901.00 us    3901.00 us              1
 SETXATTR
      0.00    4010.00 us    4010.00 us    4010.00 us              1
REMOVEXATTR
      0.01   62446.08 us       8.00 us  365433.00 us             13
FLUSH
      0.01   93887.77 us      52.00 us  588566.00 us             13
SETATTR
      0.03   10772.83 us      28.00 us 1121761.00 us            253
 GETXATTR
      0.04 3190096.00 us 3190096.00 us 3190096.00 us              1
READDIR
      0.09  558307.69 us  179931.00 us 3188951.00 us             13
 READ
      0.11  616756.00 us      74.00 us 7307745.00 us             14
XATTROP
      0.12 4754785.50 us      48.00 us 9509523.00 us              2
OPENDIR
      0.15 1799185.00 us    2310.00 us 5023537.00 us              7
 STATFS
      0.16   68757.98 us      10.00 us  872148.00 us            189
FSTAT
      0.31  143533.93 us      42.00 us 7002195.00 us            174
 OPEN
      0.40  160262.95 us     661.00 us 2825083.00 us            202
 READDIRP
      1.55  624450.87 us      31.00 us 7397432.00 us            203
 FXATTROP
     22.43  212161.62 us      12.00 us 7397413.00 us           8639
INODELK
     74.60  541421.09 us      63.00 us 14463033.00 us          11261
 LOOKUP

    Duration: 644071 seconds
   Data Read: 140706386722 bytes
Data Written: 7549422894 bytes

Interval 0 Stats:
   Block Size:                  1b+                   2b+
4b+
 No. of Reads:                    7                    18
 89
No. of Writes:                    4                     8
 37

   Block Size:                  8b+                  16b+
 32b+
 No. of Reads:                  727                  2325
 9459
No. of Writes:                  108                    54
188

   Block Size:                 64b+                 128b+
256b+
 No. of Reads:                12419                  9313
27616
No. of Writes:                  360                    85
772

   Block Size:                512b+                1024b+
 2048b+
 No. of Reads:                23708                 28691
18594
No. of Writes:                  847                   313
138

   Block Size:               4096b+                8192b+
16384b+
 No. of Reads:                19484                 12596
 8458
No. of Writes:                29185                 79632
 1431

   Block Size:              32768b+               65536b+
 131072b+
 No. of Reads:                 5695                  5755
1062899
No. of Writes:                 6168                 19435
32017

   Block Size:             262144b+             1048576b+
 No. of Reads:                    0                     0
No. of Writes:                    1                   105
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us       13806534
 FORGET
      0.00       0.00 us       0.00 us       0.00 us       17813657
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         223324
 RELEASEDIR
      0.00     560.00 us     560.00 us     560.00 us              1
ENTRYLK
      0.00    3901.00 us    3901.00 us    3901.00 us              1
 SETXATTR
      0.00    4010.00 us    4010.00 us    4010.00 us              1
REMOVEXATTR
      0.01   62446.08 us       8.00 us  365433.00 us             13
FLUSH
      0.01   93887.77 us      52.00 us  588566.00 us             13
SETATTR
      0.03   10772.83 us      28.00 us 1121761.00 us            253
 GETXATTR
      0.04 3190096.00 us 3190096.00 us 3190096.00 us              1
READDIR
      0.09  558307.69 us  179931.00 us 3188951.00 us             13
 READ
      0.11  616756.00 us      74.00 us 7307745.00 us             14
XATTROP
      0.12 4754785.50 us      48.00 us 9509523.00 us              2
OPENDIR
      0.15 1799185.00 us    2310.00 us 5023537.00 us              7
 STATFS
      0.16   68757.98 us      10.00 us  872148.00 us            189
FSTAT
      0.31  143533.93 us      42.00 us 7002195.00 us            174
 OPEN
      0.40  160262.95 us     661.00 us 2825083.00 us            202
 READDIRP
      1.55  624450.87 us      31.00 us 7397432.00 us            203
 FXATTROP
     22.43  212161.62 us      12.00 us 7397413.00 us           8639
INODELK
     74.60  541421.09 us      63.00 us 14463033.00 us          11261
 LOOKUP

    Duration: 644071 seconds
   Data Read: 140706386722 bytes
Data Written: 7549422894 bytes


On Fri, Aug 7, 2015 at 12:17 AM, Ravishankar N <ravishankar at redhat.com>
wrote:

>
>
> On 08/07/2015 12:11 PM, Prasun Gera wrote:
>
> No, no noticeable difference. Still very high, possibly higher than
> before.
>
>
> I was guessing that the cpu usage could be because of the diff algorithm
> which computes  checksums (which is a cpu intensive task). That doesn't
> seem to be the case. Could you do a volume profile and see the FOPS that
> are happening on the bricks and share the result?
> 1.gluster volume profile <volname> start
> 2. gluster volume profile <volname> info
> 3. wait 10-15 seconds
> 4.gluster volume profile <volname> info
>
>
>
> The system has come down to a crawl. It's difficult to even ssh or run
any
> commands on the terminal. Do you make anything of the logs ? The brick log
> is just a giant alternating stream of those two lines I mentioned earlier.
>
>
>
>
> On Thu, Aug 6, 2015 at 10:10 PM, Ravishankar N <ravishankar at
redhat.com>
> wrote:
>
>>
>>
>> On 08/07/2015 01:33 AM, Prasun Gera wrote:
>>
>>> I replaced the brick in a node in my 3x2 dist+repl volume (RHS 3).
I'm
>>> seeing that the heal process, which should essentially be a dump
from the
>>> working replica to the newly added one is taking exceptionally
long. It has
>>> moved ~100 G over a day on a 1Gigabit network. The CPU usage on
both the
>>> nodes of the replica has been pretty high.
>>>
>>
>> Does setting `cluster.data-self-heal-algorithm` to full make a
difference
>> in the cpu usage?
>>
>>
>> I also think that nagios is making it worse. The heal is slow enough as
>>> it is, and nagios keeps triggering heal info, which I think never
>>> completes. I also see my logs filling up These are some of the log
contents
>>> which I got by running tail on them:
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150807/e1290e72/attachment.html>