Hello:
I have a hyperconverged ovirt cluster with 3 replica 2+arb and one replica
3 volume. My replica 3 volume has been having lots of problems for a while
now.
First, after a week or two, I notice a lot of RAM usage (half of my
physical ram, plus lots of swap) for the replica 3 gluster process. The
amount of ram in use varies by machine, but they all have had the problem
show up (one at a time). Often the OOM killer kills it off before I catch
it.
This last time, I caught it. I migrated all the machines off the problem
server, and rebooted it. That was yesterday (thurs at 9am). Shortly after
coming back up, ALL 3 of my servers are showing very high load averages
(cpu % as reported by top is fairly low, althoguh iowait will spike to
25%). The one process that is using the most CPU is the glusterfsd process
for the replica 3 volume. It is consistant on all 3 machines, and it loads
down the machine and IO enough to be causing issues with the VMs running on
it (even IO to a seprate disk on the same machine) .
I watched network I/O and I can see occasional spikes to 50MB/s (gig link),
but its not consistant, and often spends considerable time near zero or in
kB/s range. the repica 3 has its own dedicated gig network.
As of last night I still had 4 unhealed items for the server I rebooted.
This AM, I'm down to 1, but its still there, and its still causing
considerable performance issues. On a server, if I stop that glusterfsd
process, then the loadavg returns to normal quickly.
HELP...what do I do? Lately it seems that all my outages and performace
issues have been glusterfsd taking down my servers that they are running
on. Is gluster ready for production, or did I misunderstand? My
reliability has been abismal on my vm cluster, and I still have 3 VMs that
won't boot anymore due to disk corruption caused by my last gluster outage
(which was the high RAM until it crashed on the server due to kernel OOM
killer).
Here's some info on the affected gluster volume:
Volume Name: data-hdd
Type: Replicate
Volume ID: d342a3ab-16f3-49f0-bbcf-f788be8ac5f1
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 172.172.1.11:/gluster/brick3/data-hdd
Brick2: 172.172.1.12:/gluster/brick3/data-hdd
Brick3: 172.172.1.13:/gluster/brick3/data-hdd
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
transport.address-family: inet
performance.readdir-ahead: on
geo-replication.indexing: on
geo-replication.ignore-pid-check: on
changelog.changelog: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
performance.low-prio-threads: 32
network.remote-dio: enable
cluster.eager-lock: off
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
features.shard: on
user.cifs: off
storage.owner-uid: 36
storage.owner-gid: 36
server.allow-insecure: on
network.ping-timeout: 30
---------------
[root at ovirt3 ~]# gluster volume profile data-hdd info
Brick: 172.172.1.13:/gluster/brick3/data-hdd
--------------------------------------------
Cumulative Stats:
Block Size: 256b+ 512b+
1024b+
No. of Reads: 188488 6126
17234
No. of Writes: 80 331655
3899
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 17228 1371069
265013
No. of Writes: 6365 26617689
15776028
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 350572 636120
189538
No. of Writes: 4475250 1557330
604455
Block Size: 131072b+ 262144b+
No. of Reads: 6442414 0
No. of Writes: 5194964 13
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 63
FORGET
0.00 0.00 us 0.00 us 0.00 us 9356
RELEASE
0.00 0.00 us 0.00 us 0.00 us 113830
RELEASEDIR
0.00 765.33 us 102.00 us 2091.00 us 3
FGETXATTR
0.00 352.90 us 263.00 us 1397.00 us 31
TRUNCATE
0.00 1094.05 us 360.00 us 5643.00 us 21
MKDIR
0.00 647.01 us 260.00 us 4059.00 us 83
MKNOD
0.00 899.64 us 68.00 us 27531.00 us 105
SETXATTR
0.00 139.19 us 26.00 us 4619.00 us 740
ENTRYLK
0.00 2121.54 us 1250.00 us 9726.00 us 63
RENAME
0.00 254.93 us 18.00 us 169316.00 us 932
FLUSH
0.00 79.37 us 14.00 us 3120.00 us 3210
LK
0.00 9377.76 us 374.00 us 197032.00 us 63
CREATE
0.00 4777.26 us 91.00 us 757927.00 us 197
FTRUNCATE
0.00 1787.09 us 117.00 us 505178.00 us 767
XATTROP
0.00 180.43 us 13.00 us 796920.00 us 33471
GETXATTR
0.01 398.80 us 72.00 us 2581011.00 us 18792
REMOVEXATTR
0.01 378.54 us 21.00 us 264034.00 us 32293
READDIR
0.01 152.76 us 1.00 us 952682.00 us 113830
OPENDIR
0.02 1460.88 us 64.00 us 4693687.00 us 18918
SETATTR
0.02 524.47 us 30.00 us 359603.00 us 56913
READDIRP
0.02 3564.23 us 59.00 us 4818020.00 us 9307
OPEN
0.04 125.61 us 21.00 us 355789.00 us 400177
STAT
0.11 132.13 us 24.00 us 460615.00 us 1143725
STATFS
0.26 111.57 us 18.00 us 3533816.00 us 3153747
FSTAT
0.81 383.95 us 22.00 us 3896810.00 us 2827323
LOOKUP
5.51 74.04 us 13.00 us 1883274.00 us 100195900
FINODELK
5.59 258.57 us 13.00 us 11221929.00 us 29091404
INODELK
6.50 922.80 us 25.00 us 7548801.00 us 9483802
READ
10.20 358.55 us 56.00 us 14993728.00 us 38295759
FXATTROP
12.42 306.31 us 47.00 us 30915095.00 us 54567728
WRITE
58.45 2495.18 us 43.00 us 21663381.00 us 31520936
FSYNC
Duration: 1946380 seconds
Data Read: 894661069137 bytes
Data Written: 1142499928350 bytes
Interval 2 Stats:
Block Size: 256b+ 512b+
1024b+
No. of Reads: 0 3
0
No. of Writes: 15 3702
126
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 0 9
58
No. of Writes: 198 62716
20361
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 13 1
1
No. of Writes: 13640 9197
3948
Block Size: 131072b+ 262144b+
No. of Reads: 1826585 0
No. of Writes: 7387 5
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 15
FORGET
0.00 0.00 us 0.00 us 0.00 us 3648
RELEASE
0.00 0.00 us 0.00 us 0.00 us 2421
RELEASEDIR
0.00 293.86 us 263.00 us 327.00 us 7
TRUNCATE
0.00 72.55 us 30.00 us 152.00 us 90
ENTRYLK
0.00 546.00 us 374.00 us 788.00 us 15
CREATE
0.00 475.86 us 178.00 us 2572.00 us 51
XATTROP
0.00 75.50 us 24.00 us 301.00 us 424
FLUSH
0.00 2889.13 us 1557.00 us 9171.00 us 15
RENAME
0.00 195.00 us 109.00 us 866.00 us 313
REMOVEXATTR
0.00 225.66 us 117.00 us 858.00 us 341
SETATTR
0.00 80.26 us 25.00 us 534.00 us 1041
LK
0.00 284.35 us 24.00 us 3965.00 us 408
READDIR
0.01 134.66 us 2.00 us 17518.00 us 2421
OPENDIR
0.01 202.33 us 34.00 us 15815.00 us 2413
FSTAT
0.01 624.48 us 23.00 us 279600.00 us 820
GETXATTR
0.04 107.65 us 29.00 us 1004.00 us 14106
STATFS
0.45 73.59 us 15.00 us 60009.00 us 250150
FINODELK
0.45 5080.05 us 62.00 us 1940405.00 us 3631
OPEN
0.97 869.22 us 35.00 us 1942304.00 us 45625
LOOKUP
1.50 378.77 us 58.00 us 1833454.00 us 163094
FXATTROP
1.77 599.60 us 55.00 us 1933975.00 us 121295
WRITE
15.58 5494.71 us 50.00 us 1958942.00 us 116418
FSYNC
29.74 668.39 us 52.00 us 1945979.00 us 1826670
READ
49.46 243.86 us 13.00 us 1945794.00 us 8325886
INODELK
Duration: 42487 seconds
Data Read: 239414982144 bytes
Data Written: 2475540892 bytes
Brick: 172.172.1.11:/gluster/brick3/data-hdd
--------------------------------------------
Cumulative Stats:
Block Size: 256b+ 512b+
1024b+
No. of Reads: 7915 524
274
No. of Writes: 31 9934
169
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 351 36927
5673
No. of Writes: 278 621338
316601
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 6003 2366
3082
No. of Writes: 110118 45664
16295
Block Size: 131072b+ 262144b+
No. of Reads: 3710107 0
No. of Writes: 556659 6
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 29
FORGET
0.00 0.00 us 0.00 us 0.00 us 6295
RELEASE
0.00 0.00 us 0.00 us 0.00 us 5574
RELEASEDIR
0.00 164.00 us 164.00 us 164.00 us 1
SETXATTR
0.00 92.33 us 66.00 us 130.00 us 3
FGETXATTR
0.00 308.93 us 215.00 us 408.00 us 15
TRUNCATE
0.00 70.94 us 22.00 us 271.00 us 186
ENTRYLK
0.00 70.06 us 21.00 us 265.00 us 669
FLUSH
0.00 1974.90 us 1490.00 us 6096.00 us 31
RENAME
0.00 76.58 us 24.00 us 3869.00 us 2582
LK
0.00 3252.80 us 175.00 us 244398.00 us 106
XATTROP
0.00 377.53 us 22.00 us 33026.00 us 1292
READDIR
0.00 940.44 us 85.00 us 563301.00 us 755
REMOVEXATTR
0.00 134434.50 us 119.00 us 563308.00 us 6
FTRUNCATE
0.00 677.32 us 19.00 us 641166.00 us 2303
GETXATTR
0.00 296.94 us 2.00 us 605194.00 us 5575
OPENDIR
0.00 3015.49 us 112.00 us 2198121.00 us 793
SETATTR
0.00 144086.32 us 461.00 us 1973731.00 us 31
CREATE
0.00 122.50 us 21.00 us 470389.00 us 37921
STATFS
0.00 322.28 us 25.00 us 2826178.00 us 17238
STAT
0.01 151.48 us 22.00 us 460622.00 us 77757
FSTAT
0.01 4641.92 us 35.00 us 7009636.00 us 3665
READDIRP
0.01 3514.85 us 61.00 us 9109607.00 us 6278
OPEN
0.04 800.50 us 23.00 us 8211795.00 us 109977
LOOKUP
0.60 1279.23 us 59.00 us 237262387.00 us 1023399
FXATTROP
1.50 872.91 us 35.00 us 6929426.00 us 3773222
READ
1.61 2096.57 us 50.00 us 237261777.00 us 1677093
WRITE
6.17 16780.27 us 51.00 us 22990372.00 us 804858
FSYNC
11.35 1890.83 us 14.00 us 1066342526.00 us 13144090
INODELK
78.70 73624.16 us 15.00 us 759064386.00 us 2340006
FINODELK
Duration: 84992 seconds
Data Read: 486964889723 bytes
Data Written: 84167404788 bytes
Interval 2 Stats:
Block Size: 256b+ 512b+
1024b+
No. of Reads: 3861 337
173
No. of Writes: 15 3702
126
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 159 5308
857
No. of Writes: 198 62716
20361
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 1008 1004
621
No. of Writes: 13640 9197
3948
Block Size: 131072b+ 262144b+
No. of Reads: 2341344 0
No. of Writes: 7387 5
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 15
FORGET
0.00 0.00 us 0.00 us 0.00 us 3647
RELEASE
0.00 0.00 us 0.00 us 0.00 us 2541
RELEASEDIR
0.00 267.86 us 215.00 us 322.00 us 7
TRUNCATE
0.00 66.18 us 22.00 us 144.00 us 90
ENTRYLK
0.00 450.02 us 197.00 us 2730.00 us 51
XATTROP
0.00 66.45 us 21.00 us 265.00 us 423
FLUSH
0.00 2049.80 us 1504.00 us 6096.00 us 15
RENAME
0.00 193.90 us 85.00 us 1329.00 us 313
REMOVEXATTR
0.00 74.23 us 24.00 us 3869.00 us 1041
LK
0.00 258.23 us 112.00 us 12048.00 us 341
SETATTR
0.00 384.60 us 22.00 us 2284.00 us 647
READDIR
0.00 415.01 us 19.00 us 122660.00 us 1306
GETXATTR
0.00 495.05 us 2.00 us 605194.00 us 2542
OPENDIR
0.00 98.96 us 30.00 us 2284.00 us 14106
STATFS
0.00 194.92 us 25.00 us 628797.00 us 8675
STAT
0.00 849.86 us 35.00 us 369113.00 us 2005
READDIRP
0.00 585.27 us 61.00 us 279889.00 us 3630
OPEN
0.00 141763.87 us 461.00 us 1973731.00 us 15
CREATE
0.00 155.27 us 23.00 us 169047.00 us 21344
FSTAT
0.02 442.07 us 26.00 us 544143.00 us 45744
LOOKUP
0.11 641.69 us 59.00 us 2227486.00 us 163094
FXATTROP
0.16 1215.83 us 58.00 us 2628020.00 us 121295
WRITE
1.25 9910.79 us 51.00 us 5116269.00 us 116418
FSYNC
1.30 511.11 us 35.00 us 2493646.00 us 2354672
READ
13.86 1535.67 us 14.00 us 688678811.00 us 8321169
INODELK
83.28 309339.72 us 16.00 us 688564320.00 us 248271
FINODELK
Duration: 42487 seconds
Data Read: 307016229315 bytes
Data Written: 2475540892 bytes
Brick: 172.172.1.12:/gluster/brick3/data-hdd
--------------------------------------------
Cumulative Stats:
Block Size: 256b+ 512b+
1024b+
No. of Reads: 3706 247
142
No. of Writes: 15 3600
124
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 94 5848
956
No. of Writes: 194 60615
19772
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 1287 12947
1248
No. of Writes: 13126 8886
3810
Block Size: 131072b+ 262144b+
No. of Reads: 959349 0
No. of Writes: 3178716 5
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 13
FORGET
0.00 0.00 us 0.00 us 0.00 us 3509
RELEASE
0.00 0.00 us 0.00 us 0.00 us 2425
RELEASEDIR
0.00 166.00 us 97.00 us 301.00 us 3
SETXATTR
0.00 42.62 us 19.00 us 123.00 us 90
ENTRYLK
0.00 431.00 us 80.00 us 8036.00 us 37
FTRUNCATE
0.00 160.27 us 17.00 us 63213.00 us 1041
LK
0.00 85684.71 us 166.00 us 598415.00 us 7
TRUNCATE
0.00 74683.31 us 143.00 us 2544472.00 us 35
XATTROP
0.00 22002.58 us 19.00 us 11281413.00 us 520
READDIR
0.01 6248116.73 us 1217.00 us 52589867.00 us 15
RENAME
0.01 6913389.40 us 332.00 us 34408990.00 us 15
CREATE
0.06 294044.50 us 25.00 us 100581145.00 us 1878
READDIRP
0.08 2357246.40 us 71.00 us 159200810.00 us 306
REMOVEXATTR
0.13 83641.44 us 20.00 us 69913926.00 us 13691
STATFS
0.14 1284416.00 us 15.00 us 173040222.00 us 964
GETXATTR
0.18 3713522.62 us 14.00 us 372969001.00 us 427
FLUSH
0.18 456678.40 us 53.00 us 130997224.00 us 3508
OPEN
0.34 1255418.93 us 1.00 us 267727475.00 us 2425
OPENDIR
0.64 16601795.78 us 73.00 us 392869613.00 us 347
SETATTR
0.74 235518.84 us 19.00 us 384250227.00 us 28134
FSTAT
1.10 1203934.76 us 16.00 us 407840076.00 us 8208
STAT
2.41 22009.35 us 29.00 us 421083048.00 us 985824
READ
2.53 93906.00 us 13.00 us 118521408.00 us 241629
FINODELK
3.78 4111.77 us 12.00 us 419936158.00 us 8251369
INODELK
3.84 218447.60 us 83.00 us 498122410.00 us 158007
FXATTROP
4.02 817162.39 us 19.00 us 409893532.00 us 44246
LOOKUP
25.08 68523.67 us 49.00 us 516930780.00 us 3288855
WRITE
54.73 4364231.40 us 51.00 us 688577405.00 us 112686
FSYNC
Duration: 41424 seconds
Data Read: 126333064115 bytes
Data Written: 418096774044 bytes
Interval 0 Stats:
Block Size: 256b+ 512b+
1024b+
No. of Reads: 3706 247
142
No. of Writes: 15 3600
124
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 94 5848
956
No. of Writes: 194 60615
19772
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 1287 12947
1248
No. of Writes: 13126 8886
3810
Block Size: 131072b+ 262144b+
No. of Reads: 959349 0
No. of Writes: 3178716 5
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 13
FORGET
0.00 0.00 us 0.00 us 0.00 us 3509
RELEASE
0.00 0.00 us 0.00 us 0.00 us 2425
RELEASEDIR
0.00 166.00 us 97.00 us 301.00 us 3
SETXATTR
0.00 42.62 us 19.00 us 123.00 us 90
ENTRYLK
0.00 431.00 us 80.00 us 8036.00 us 37
FTRUNCATE
0.00 160.27 us 17.00 us 63213.00 us 1041
LK
0.00 85684.71 us 166.00 us 598415.00 us 7
TRUNCATE
0.00 74683.31 us 143.00 us 2544472.00 us 35
XATTROP
0.00 22002.58 us 19.00 us 11281413.00 us 520
READDIR
0.01 6248116.73 us 1217.00 us 52589867.00 us 15
RENAME
0.01 6913389.40 us 332.00 us 34408990.00 us 15
CREATE
0.06 294044.50 us 25.00 us 100581145.00 us 1878
READDIRP
0.08 2357246.40 us 71.00 us 159200810.00 us 306
REMOVEXATTR
0.13 83641.44 us 20.00 us 69913926.00 us 13691
STATFS
0.14 1284416.00 us 15.00 us 173040222.00 us 964
GETXATTR
0.18 3720010.69 us 14.00 us 372969001.00 us 427
FLUSH
0.18 456678.40 us 53.00 us 130997224.00 us 3508
OPEN
0.34 1255418.93 us 1.00 us 267727475.00 us 2425
OPENDIR
0.64 16601795.78 us 73.00 us 392869613.00 us 347
SETATTR
0.74 235509.69 us 19.00 us 384250227.00 us 28134
FSTAT
1.10 1204610.00 us 16.00 us 407840076.00 us 8208
STAT
2.41 22009.63 us 29.00 us 421083048.00 us 985824
READ
2.52 93906.03 us 13.00 us 118521408.00 us 241629
FINODELK
3.77 4111.77 us 12.00 us 419936158.00 us 8251369
INODELK
3.84 218446.23 us 83.00 us 498122410.00 us 158007
FXATTROP
4.02 817497.87 us 19.00 us 409893532.00 us 44246
LOOKUP
25.07 68548.46 us 49.00 us 516930780.00 us 3288855
WRITE
54.76 4369255.44 us 51.00 us 688577405.00 us 112686
FSYNC
Duration: 41424 seconds
Data Read: 126333064115 bytes
Data Written: 418096774044 bytes
-----------
[this command took a few MINUTES to return]
[root at ovirt1 ~]# gluster volume heal data-hdd info
Brick 172.172.1.11:/gluster/brick3/data-hdd
/cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/647be733-f153-4cdc-85bd-ba72544c2631/b453a300-0602-4be1-8310-8bd5abe00971
Status: Connected
Number of entries: 1
Brick 172.172.1.12:/gluster/brick3/data-hdd
Status: Connected
Number of entries: 0
Brick 172.172.1.13:/gluster/brick3/data-hdd
/cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/647be733-f153-4cdc-85bd-ba72544c2631/b453a300-0602-4be1-8310-8bd5abe00971
Status: Connected
Number of entries: 1
-----------
09:43:02 up 16:09, 1 user, load average: 30.75, 39.10, 34.75
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180706/1654089b/attachment.html>
Darrell Budic
2018-Jul-06 21:32 UTC
[Gluster-users] Gluster-induced load averages of 50-200
I?ve had issues like that while playing with different healing config, too many shd threads will drive load higher than it needs to be and keep the boxes busy. I?ve had better luck with cluster.shd-max-threads: 1 cluster.shd-wait-qlength: 1024 than 8 & 10000. It does heal a little faster with 8/10k than 1/1k, but it keeps the servers too busy to do anything else. In a hyper converged install, I could see it causing problems for the VMs and keeping healing from effectively happening (it would cause disk read issues for me even with a separate server cluster). Also, what version are you running? If it?s 3.12, I?m finding that .9-.11 have a client side memory leak, just havn?t gotten everything together enough to post about it. .8 seems ok, and .6 is definitely ok though, flat memory use from both, steadily increasing from .9 and .11 clients. Good luck! -Darrell> From: Jim Kusznir <jim at palousetech.com> > Subject: [Gluster-users] Gluster-induced load averages of 50-200 > Date: July 6, 2018 at 11:43:21 AM CDT > To: gluster-user > > Hello: > > I have a hyperconverged ovirt cluster with 3 replica 2+arb and one replica 3 volume. My replica 3 volume has been having lots of problems for a while now. > > First, after a week or two, I notice a lot of RAM usage (half of my physical ram, plus lots of swap) for the replica 3 gluster process. The amount of ram in use varies by machine, but they all have had the problem show up (one at a time). Often the OOM killer kills it off before I catch it. > > This last time, I caught it. I migrated all the machines off the problem server, and rebooted it. That was yesterday (thurs at 9am). Shortly after coming back up, ALL 3 of my servers are showing very high load averages (cpu % as reported by top is fairly low, althoguh iowait will spike to 25%). The one process that is using the most CPU is the glusterfsd process for the replica 3 volume. It is consistant on all 3 machines, and it loads down the machine and IO enough to be causing issues with the VMs running on it (even IO to a seprate disk on the same machine) . > > I watched network I/O and I can see occasional spikes to 50MB/s (gig link), but its not consistant, and often spends considerable time near zero or in kB/s range. the repica 3 has its own dedicated gig network. > > As of last night I still had 4 unhealed items for the server I rebooted. This AM, I'm down to 1, but its still there, and its still causing considerable performance issues. On a server, if I stop that glusterfsd process, then the loadavg returns to normal quickly. > > HELP...what do I do? Lately it seems that all my outages and performace issues have been glusterfsd taking down my servers that they are running on. Is gluster ready for production, or did I misunderstand? My reliability has been abismal on my vm cluster, and I still have 3 VMs that won't boot anymore due to disk corruption caused by my last gluster outage (which was the high RAM until it crashed on the server due to kernel OOM killer). > > Here's some info on the affected gluster volume: > Volume Name: data-hdd > Type: Replicate > Volume ID: d342a3ab-16f3-49f0-bbcf-f788be8ac5f1 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 3 = 3 > Transport-type: tcp > Bricks: > Brick1: 172.172.1.11:/gluster/brick3/data-hdd > Brick2: 172.172.1.12:/gluster/brick3/data-hdd > Brick3: 172.172.1.13:/gluster/brick3/data-hdd > Options Reconfigured: > diagnostics.count-fop-hits: on > diagnostics.latency-measurement: on > transport.address-family: inet > performance.readdir-ahead: on > geo-replication.indexing: on > geo-replication.ignore-pid-check: on > changelog.changelog: on > performance.quick-read: off > performance.read-ahead: off > performance.io-cache: off > performance.stat-prefetch: off > performance.low-prio-threads: 32 > network.remote-dio: enable > cluster.eager-lock: off > cluster.quorum-type: auto > cluster.server-quorum-type: server > cluster.data-self-heal-algorithm: full > cluster.locking-scheme: granular > cluster.shd-max-threads: 8 > cluster.shd-wait-qlength: 10000 > features.shard: on > user.cifs: off > storage.owner-uid: 36 > storage.owner-gid: 36 > server.allow-insecure: on > network.ping-timeout: 30 > > --------------- > > [root at ovirt3 ~]# gluster volume profile data-hdd info > Brick: 172.172.1.13:/gluster/brick3/data-hdd > -------------------------------------------- > Cumulative Stats: > Block Size: 256b+ 512b+ 1024b+ > No. of Reads: 188488 6126 17234 > No. of Writes: 80 331655 3899 > > Block Size: 2048b+ 4096b+ 8192b+ > No. of Reads: 17228 1371069 265013 > No. of Writes: 6365 26617689 15776028 > > Block Size: 16384b+ 32768b+ 65536b+ > No. of Reads: 350572 636120 189538 > No. of Writes: 4475250 1557330 604455 > > Block Size: 131072b+ 262144b+ > No. of Reads: 6442414 0 > No. of Writes: 5194964 13 > %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop > --------- ----------- ----------- ----------- ------------ ---- > 0.00 0.00 us 0.00 us 0.00 us 63 FORGET > 0.00 0.00 us 0.00 us 0.00 us 9356 RELEASE > 0.00 0.00 us 0.00 us 0.00 us 113830 RELEASEDIR > 0.00 765.33 us 102.00 us 2091.00 us 3 FGETXATTR > 0.00 352.90 us 263.00 us 1397.00 us 31 TRUNCATE > 0.00 1094.05 us 360.00 us 5643.00 us 21 MKDIR > 0.00 647.01 us 260.00 us 4059.00 us 83 MKNOD > 0.00 899.64 us 68.00 us 27531.00 us 105 SETXATTR > 0.00 139.19 us 26.00 us 4619.00 us 740 ENTRYLK > 0.00 2121.54 us 1250.00 us 9726.00 us 63 RENAME > 0.00 254.93 us 18.00 us 169316.00 us 932 FLUSH > 0.00 79.37 us 14.00 us 3120.00 us 3210 LK > 0.00 9377.76 us 374.00 us 197032.00 us 63 CREATE > 0.00 4777.26 us 91.00 us 757927.00 us 197 FTRUNCATE > 0.00 1787.09 us 117.00 us 505178.00 us 767 XATTROP > 0.00 180.43 us 13.00 us 796920.00 us 33471 GETXATTR > 0.01 398.80 us 72.00 us 2581011.00 us 18792 REMOVEXATTR > 0.01 378.54 us 21.00 us 264034.00 us 32293 READDIR > 0.01 152.76 us 1.00 us 952682.00 us 113830 OPENDIR > 0.02 1460.88 us 64.00 us 4693687.00 us 18918 SETATTR > 0.02 524.47 us 30.00 us 359603.00 us 56913 READDIRP > 0.02 3564.23 us 59.00 us 4818020.00 us 9307 OPEN > 0.04 125.61 us 21.00 us 355789.00 us 400177 STAT > 0.11 132.13 us 24.00 us 460615.00 us 1143725 STATFS > 0.26 111.57 us 18.00 us 3533816.00 us 3153747 FSTAT > 0.81 383.95 us 22.00 us 3896810.00 us 2827323 LOOKUP > 5.51 74.04 us 13.00 us 1883274.00 us 100195900 FINODELK > 5.59 258.57 us 13.00 us 11221929.00 us 29091404 INODELK > 6.50 922.80 us 25.00 us 7548801.00 us 9483802 READ > 10.20 358.55 us 56.00 us 14993728.00 us 38295759 FXATTROP > 12.42 306.31 us 47.00 us 30915095.00 us 54567728 WRITE > 58.45 2495.18 us 43.00 us 21663381.00 us 31520936 FSYNC > > Duration: 1946380 seconds > Data Read: 894661069137 bytes > Data Written: 1142499928350 bytes > > Interval 2 Stats: > Block Size: 256b+ 512b+ 1024b+ > No. of Reads: 0 3 0 > No. of Writes: 15 3702 126 > > Block Size: 2048b+ 4096b+ 8192b+ > No. of Reads: 0 9 58 > No. of Writes: 198 62716 20361 > > Block Size: 16384b+ 32768b+ 65536b+ > No. of Reads: 13 1 1 > No. of Writes: 13640 9197 3948 > > Block Size: 131072b+ 262144b+ > No. of Reads: 1826585 0 > No. of Writes: 7387 5 > %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop > --------- ----------- ----------- ----------- ------------ ---- > 0.00 0.00 us 0.00 us 0.00 us 15 FORGET > 0.00 0.00 us 0.00 us 0.00 us 3648 RELEASE > 0.00 0.00 us 0.00 us 0.00 us 2421 RELEASEDIR > 0.00 293.86 us 263.00 us 327.00 us 7 TRUNCATE > 0.00 72.55 us 30.00 us 152.00 us 90 ENTRYLK > 0.00 546.00 us 374.00 us 788.00 us 15 CREATE > 0.00 475.86 us 178.00 us 2572.00 us 51 XATTROP > 0.00 75.50 us 24.00 us 301.00 us 424 FLUSH > 0.00 2889.13 us 1557.00 us 9171.00 us 15 RENAME > 0.00 195.00 us 109.00 us 866.00 us 313 REMOVEXATTR > 0.00 225.66 us 117.00 us 858.00 us 341 SETATTR > 0.00 80.26 us 25.00 us 534.00 us 1041 LK > 0.00 284.35 us 24.00 us 3965.00 us 408 READDIR > 0.01 134.66 us 2.00 us 17518.00 us 2421 OPENDIR > 0.01 202.33 us 34.00 us 15815.00 us 2413 FSTAT > 0.01 624.48 us 23.00 us 279600.00 us 820 GETXATTR > 0.04 107.65 us 29.00 us 1004.00 us 14106 STATFS > 0.45 73.59 us 15.00 us 60009.00 us 250150 FINODELK > 0.45 5080.05 us 62.00 us 1940405.00 us 3631 OPEN > 0.97 869.22 us 35.00 us 1942304.00 us 45625 LOOKUP > 1.50 378.77 us 58.00 us 1833454.00 us 163094 FXATTROP > 1.77 599.60 us 55.00 us 1933975.00 us 121295 WRITE > 15.58 5494.71 us 50.00 us 1958942.00 us 116418 FSYNC > 29.74 668.39 us 52.00 us 1945979.00 us 1826670 READ > 49.46 243.86 us 13.00 us 1945794.00 us 8325886 INODELK > > Duration: 42487 seconds > Data Read: 239414982144 bytes > Data Written: 2475540892 bytes > > Brick: 172.172.1.11:/gluster/brick3/data-hdd > -------------------------------------------- > Cumulative Stats: > Block Size: 256b+ 512b+ 1024b+ > No. of Reads: 7915 524 274 > No. of Writes: 31 9934 169 > > Block Size: 2048b+ 4096b+ 8192b+ > No. of Reads: 351 36927 5673 > No. of Writes: 278 621338 316601 > > Block Size: 16384b+ 32768b+ 65536b+ > No. of Reads: 6003 2366 3082 > No. of Writes: 110118 45664 16295 > > Block Size: 131072b+ 262144b+ > No. of Reads: 3710107 0 > No. of Writes: 556659 6 > %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop > --------- ----------- ----------- ----------- ------------ ---- > 0.00 0.00 us 0.00 us 0.00 us 29 FORGET > 0.00 0.00 us 0.00 us 0.00 us 6295 RELEASE > 0.00 0.00 us 0.00 us 0.00 us 5574 RELEASEDIR > 0.00 164.00 us 164.00 us 164.00 us 1 SETXATTR > 0.00 92.33 us 66.00 us 130.00 us 3 FGETXATTR > 0.00 308.93 us 215.00 us 408.00 us 15 TRUNCATE > 0.00 70.94 us 22.00 us 271.00 us 186 ENTRYLK > 0.00 70.06 us 21.00 us 265.00 us 669 FLUSH > 0.00 1974.90 us 1490.00 us 6096.00 us 31 RENAME > 0.00 76.58 us 24.00 us 3869.00 us 2582 LK > 0.00 3252.80 us 175.00 us 244398.00 us 106 XATTROP > 0.00 377.53 us 22.00 us 33026.00 us 1292 READDIR > 0.00 940.44 us 85.00 us 563301.00 us 755 REMOVEXATTR > 0.00 134434.50 us 119.00 us 563308.00 us 6 FTRUNCATE > 0.00 677.32 us 19.00 us 641166.00 us 2303 GETXATTR > 0.00 296.94 us 2.00 us 605194.00 us 5575 OPENDIR > 0.00 3015.49 us 112.00 us 2198121.00 us 793 SETATTR > 0.00 144086.32 us 461.00 us 1973731.00 us 31 CREATE > 0.00 122.50 us 21.00 us 470389.00 us 37921 STATFS > 0.00 322.28 us 25.00 us 2826178.00 us 17238 STAT > 0.01 151.48 us 22.00 us 460622.00 us 77757 FSTAT > 0.01 4641.92 us 35.00 us 7009636.00 us 3665 READDIRP > 0.01 3514.85 us 61.00 us 9109607.00 us 6278 OPEN > 0.04 800.50 us 23.00 us 8211795.00 us 109977 LOOKUP > 0.60 1279.23 us 59.00 us 237262387.00 us 1023399 FXATTROP > 1.50 872.91 us 35.00 us 6929426.00 us 3773222 READ > 1.61 2096.57 us 50.00 us 237261777.00 us 1677093 WRITE > 6.17 16780.27 us 51.00 us 22990372.00 us 804858 FSYNC > 11.35 1890.83 us 14.00 us 1066342526.00 us 13144090 INODELK > 78.70 73624.16 us 15.00 us 759064386.00 us 2340006 FINODELK > > Duration: 84992 seconds > Data Read: 486964889723 bytes > Data Written: 84167404788 bytes > > Interval 2 Stats: > Block Size: 256b+ 512b+ 1024b+ > No. of Reads: 3861 337 173 > No. of Writes: 15 3702 126 > > Block Size: 2048b+ 4096b+ 8192b+ > No. of Reads: 159 5308 857 > No. of Writes: 198 62716 20361 > > Block Size: 16384b+ 32768b+ 65536b+ > No. of Reads: 1008 1004 621 > No. of Writes: 13640 9197 3948 > > Block Size: 131072b+ 262144b+ > No. of Reads: 2341344 0 > No. of Writes: 7387 5 > %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop > --------- ----------- ----------- ----------- ------------ ---- > 0.00 0.00 us 0.00 us 0.00 us 15 FORGET > 0.00 0.00 us 0.00 us 0.00 us 3647 RELEASE > 0.00 0.00 us 0.00 us 0.00 us 2541 RELEASEDIR > 0.00 267.86 us 215.00 us 322.00 us 7 TRUNCATE > 0.00 66.18 us 22.00 us 144.00 us 90 ENTRYLK > 0.00 450.02 us 197.00 us 2730.00 us 51 XATTROP > 0.00 66.45 us 21.00 us 265.00 us 423 FLUSH > 0.00 2049.80 us 1504.00 us 6096.00 us 15 RENAME > 0.00 193.90 us 85.00 us 1329.00 us 313 REMOVEXATTR > 0.00 74.23 us 24.00 us 3869.00 us 1041 LK > 0.00 258.23 us 112.00 us 12048.00 us 341 SETATTR > 0.00 384.60 us 22.00 us 2284.00 us 647 READDIR > 0.00 415.01 us 19.00 us 122660.00 us 1306 GETXATTR > 0.00 495.05 us 2.00 us 605194.00 us 2542 OPENDIR > 0.00 98.96 us 30.00 us 2284.00 us 14106 STATFS > 0.00 194.92 us 25.00 us 628797.00 us 8675 STAT > 0.00 849.86 us 35.00 us 369113.00 us 2005 READDIRP > 0.00 585.27 us 61.00 us 279889.00 us 3630 OPEN > 0.00 141763.87 us 461.00 us 1973731.00 us 15 CREATE > 0.00 155.27 us 23.00 us 169047.00 us 21344 FSTAT > 0.02 442.07 us 26.00 us 544143.00 us 45744 LOOKUP > 0.11 641.69 us 59.00 us 2227486.00 us 163094 FXATTROP > 0.16 1215.83 us 58.00 us 2628020.00 us 121295 WRITE > 1.25 9910.79 us 51.00 us 5116269.00 us 116418 FSYNC > 1.30 511.11 us 35.00 us 2493646.00 us 2354672 READ > 13.86 1535.67 us 14.00 us 688678811.00 us 8321169 INODELK > 83.28 309339.72 us 16.00 us 688564320.00 us 248271 FINODELK > > Duration: 42487 seconds > Data Read: 307016229315 bytes > Data Written: 2475540892 bytes > > Brick: 172.172.1.12:/gluster/brick3/data-hdd > -------------------------------------------- > Cumulative Stats: > Block Size: 256b+ 512b+ 1024b+ > No. of Reads: 3706 247 142 > No. of Writes: 15 3600 124 > > Block Size: 2048b+ 4096b+ 8192b+ > No. of Reads: 94 5848 956 > No. of Writes: 194 60615 19772 > > Block Size: 16384b+ 32768b+ 65536b+ > No. of Reads: 1287 12947 1248 > No. of Writes: 13126 8886 3810 > > Block Size: 131072b+ 262144b+ > No. of Reads: 959349 0 > No. of Writes: 3178716 5 > %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop > --------- ----------- ----------- ----------- ------------ ---- > 0.00 0.00 us 0.00 us 0.00 us 13 FORGET > 0.00 0.00 us 0.00 us 0.00 us 3509 RELEASE > 0.00 0.00 us 0.00 us 0.00 us 2425 RELEASEDIR > 0.00 166.00 us 97.00 us 301.00 us 3 SETXATTR > 0.00 42.62 us 19.00 us 123.00 us 90 ENTRYLK > 0.00 431.00 us 80.00 us 8036.00 us 37 FTRUNCATE > 0.00 160.27 us 17.00 us 63213.00 us 1041 LK > 0.00 85684.71 us 166.00 us 598415.00 us 7 TRUNCATE > 0.00 74683.31 us 143.00 us 2544472.00 us 35 XATTROP > 0.00 22002.58 us 19.00 us 11281413.00 us 520 READDIR > 0.01 6248116.73 us 1217.00 us 52589867.00 us 15 RENAME > 0.01 6913389.40 us 332.00 us 34408990.00 us 15 CREATE > 0.06 294044.50 us 25.00 us 100581145.00 us 1878 READDIRP > 0.08 2357246.40 us 71.00 us 159200810.00 us 306 REMOVEXATTR > 0.13 83641.44 us 20.00 us 69913926.00 us 13691 STATFS > 0.14 1284416.00 us 15.00 us 173040222.00 us 964 GETXATTR > 0.18 3713522.62 us 14.00 us 372969001.00 us 427 FLUSH > 0.18 456678.40 us 53.00 us 130997224.00 us 3508 OPEN > 0.34 1255418.93 us 1.00 us 267727475.00 us 2425 OPENDIR > 0.64 16601795.78 us 73.00 us 392869613.00 us 347 SETATTR > 0.74 235518.84 us 19.00 us 384250227.00 us 28134 FSTAT > 1.10 1203934.76 us 16.00 us 407840076.00 us 8208 STAT > 2.41 22009.35 us 29.00 us 421083048.00 us 985824 READ > 2.53 93906.00 us 13.00 us 118521408.00 us 241629 FINODELK > 3.78 4111.77 us 12.00 us 419936158.00 us 8251369 INODELK > 3.84 218447.60 us 83.00 us 498122410.00 us 158007 FXATTROP > 4.02 817162.39 us 19.00 us 409893532.00 us 44246 LOOKUP > 25.08 68523.67 us 49.00 us 516930780.00 us 3288855 WRITE > 54.73 4364231.40 us 51.00 us 688577405.00 us 112686 FSYNC > > Duration: 41424 seconds > Data Read: 126333064115 bytes > Data Written: 418096774044 bytes > > Interval 0 Stats: > Block Size: 256b+ 512b+ 1024b+ > No. of Reads: 3706 247 142 > No. of Writes: 15 3600 124 > > Block Size: 2048b+ 4096b+ 8192b+ > No. of Reads: 94 5848 956 > No. of Writes: 194 60615 19772 > > Block Size: 16384b+ 32768b+ 65536b+ > No. of Reads: 1287 12947 1248 > No. of Writes: 13126 8886 3810 > > Block Size: 131072b+ 262144b+ > No. of Reads: 959349 0 > No. of Writes: 3178716 5 > %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop > --------- ----------- ----------- ----------- ------------ ---- > 0.00 0.00 us 0.00 us 0.00 us 13 FORGET > 0.00 0.00 us 0.00 us 0.00 us 3509 RELEASE > 0.00 0.00 us 0.00 us 0.00 us 2425 RELEASEDIR > 0.00 166.00 us 97.00 us 301.00 us 3 SETXATTR > 0.00 42.62 us 19.00 us 123.00 us 90 ENTRYLK > 0.00 431.00 us 80.00 us 8036.00 us 37 FTRUNCATE > 0.00 160.27 us 17.00 us 63213.00 us 1041 LK > 0.00 85684.71 us 166.00 us 598415.00 us 7 TRUNCATE > 0.00 74683.31 us 143.00 us 2544472.00 us 35 XATTROP > 0.00 22002.58 us 19.00 us 11281413.00 us 520 READDIR > 0.01 6248116.73 us 1217.00 us 52589867.00 us 15 RENAME > 0.01 6913389.40 us 332.00 us 34408990.00 us 15 CREATE > 0.06 294044.50 us 25.00 us 100581145.00 us 1878 READDIRP > 0.08 2357246.40 us 71.00 us 159200810.00 us 306 REMOVEXATTR > 0.13 83641.44 us 20.00 us 69913926.00 us 13691 STATFS > 0.14 1284416.00 us 15.00 us 173040222.00 us 964 GETXATTR > 0.18 3720010.69 us 14.00 us 372969001.00 us 427 FLUSH > 0.18 456678.40 us 53.00 us 130997224.00 us 3508 OPEN > 0.34 1255418.93 us 1.00 us 267727475.00 us 2425 OPENDIR > 0.64 16601795.78 us 73.00 us 392869613.00 us 347 SETATTR > 0.74 235509.69 us 19.00 us 384250227.00 us 28134 FSTAT > 1.10 1204610.00 us 16.00 us 407840076.00 us 8208 STAT > 2.41 22009.63 us 29.00 us 421083048.00 us 985824 READ > 2.52 93906.03 us 13.00 us 118521408.00 us 241629 FINODELK > 3.77 4111.77 us 12.00 us 419936158.00 us 8251369 INODELK > 3.84 218446.23 us 83.00 us 498122410.00 us 158007 FXATTROP > 4.02 817497.87 us 19.00 us 409893532.00 us 44246 LOOKUP > 25.07 68548.46 us 49.00 us 516930780.00 us 3288855 WRITE > 54.76 4369255.44 us 51.00 us 688577405.00 us 112686 FSYNC > > Duration: 41424 seconds > Data Read: 126333064115 bytes > Data Written: 418096774044 bytes > > ----------- > [this command took a few MINUTES to return] > > [root at ovirt1 ~]# gluster volume heal data-hdd info > Brick 172.172.1.11:/gluster/brick3/data-hdd > /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/647be733-f153-4cdc-85bd-ba72544c2631/b453a300-0602-4be1-8310-8bd5abe00971 > Status: Connected > Number of entries: 1 > > Brick 172.172.1.12:/gluster/brick3/data-hdd > Status: Connected > Number of entries: 0 > > Brick 172.172.1.13:/gluster/brick3/data-hdd > /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/647be733-f153-4cdc-85bd-ba72544c2631/b453a300-0602-4be1-8310-8bd5abe00971 > Status: Connected > Number of entries: 1 > > ----------- > 09:43:02 up 16:09, 1 user, load average: 30.75, 39.10, 34.75 > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180706/37a13d50/attachment.html>