Hi, All
I have glusterfs 3.10.0 running with two replicated nodes, but one of the node
has a bad CPU performance:
Node1/Node2: 4 core, 8GB, using thin LVM for volumes
This is node1:
[cid:image005.png at 01D2AA33.B41E14D0]
And this is node2:
[cid:image006.png at 01D2AA33.B41E14D0]
Currently I have only Prometheus using glusterfs volumes, so I started profile
on volume prometheus and got this:
Brick: 10.10.0.100:/glusterfsvolumes/devops/devops-prometheus/brick
-------------------------------------------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 1 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 1 0
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 4 6 32
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 154 281 484
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 993 15290
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245895 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.00 20.50 us 19.00 us 22.00 us 2 FSYNCDIR
0.01 99.00 us 99.00 us 99.00 us 1 GETXATTR
0.01 68.50 us 48.00 us 89.00 us 2 OPENDIR
0.03 84.33 us 40.00 us 164.00 us 6 FSTAT
0.10 64.35 us 34.00 us 177.00 us 23 STATFS
0.16 62.87 us 29.00 us 172.00 us 38 STAT
0.17 54.76 us 22.00 us 194.00 us 46 ENTRYLK
0.18 41.66 us 24.00 us 95.00 us 62 FLUSH
0.22 198.44 us 138.00 us 339.00 us 16 CREATE
0.32 101.65 us 65.00 us 184.00 us 46 OPEN
0.35 726.43 us 350.00 us 1165.00 us 7 UNLINK
0.52 56.22 us 25.00 us 228.00 us 136 FINODELK
0.77 129.15 us 49.00 us 1138.00 us 87 READ
1.57 154.70 us 59.00 us 360.00 us 148 FXATTROP
2.44 159.13 us 41.00 us 5836.00 us 224 WRITE
3.64 188.62 us 54.00 us 848.00 us 282 LOOKUP
89.50 20741.70 us 9457.00 us 84733.00 us 63 FSYNC
Duration: 362318 seconds
Data Read: 2132763575 bytes
Data Written: 83056358475 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 1 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 1 0
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 4 6 32
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 154 281 484
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 993 15290
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245895 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.00 20.50 us 19.00 us 22.00 us 2 FSYNCDIR
0.01 99.00 us 99.00 us 99.00 us 1 GETXATTR
0.01 68.50 us 48.00 us 89.00 us 2 OPENDIR
0.03 84.33 us 40.00 us 164.00 us 6 FSTAT
0.10 64.35 us 34.00 us 177.00 us 23 STATFS
0.16 62.87 us 29.00 us 172.00 us 38 STAT
0.17 54.76 us 22.00 us 194.00 us 46 ENTRYLK
0.18 41.66 us 24.00 us 95.00 us 62 FLUSH
0.22 198.44 us 138.00 us 339.00 us 16 CREATE
0.32 101.65 us 65.00 us 184.00 us 46 OPEN
0.35 726.43 us 350.00 us 1165.00 us 7 UNLINK
0.52 56.22 us 25.00 us 228.00 us 136 FINODELK
0.77 129.15 us 49.00 us 1138.00 us 87 READ
1.57 154.70 us 59.00 us 360.00 us 148 FXATTROP
2.44 159.13 us 41.00 us 5836.00 us 224 WRITE
3.64 188.62 us 54.00 us 848.00 us 282 LOOKUP
89.50 20741.70 us 9457.00 us 84733.00 us 63 FSYNC
Duration: 362318 seconds
Data Read: 2132763575 bytes
Data Written: 83056358475 bytes
Brick: 10.10.0.101:/glusterfsvolumes/devops/devops-prometheus/brick
-------------------------------------------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 1 3 2
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 2 14 22
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 166 253 502
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 893 14392
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245861 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.01 89.00 us 89.00 us 89.00 us 1 GETXATTR
0.02 62.00 us 53.00 us 71.00 us 2 OPENDIR
0.02 57.00 us 51.00 us 61.00 us 3 FSTAT
0.04 178.00 us 38.00 us 318.00 us 2 FSYNCDIR
0.12 474.00 us 451.00 us 497.00 us 2 READDIRP
0.18 64.00 us 28.00 us 113.00 us 23 STATFS
0.28 48.30 us 18.00 us 184.00 us 46 ENTRYLK
0.31 41.33 us 18.00 us 87.00 us 60 FLUSH
0.33 66.12 us 28.00 us 246.00 us 40 STAT
0.57 102.39 us 62.00 us 236.00 us 44 OPEN
0.58 664.86 us 311.00 us 855.00 us 7 UNLINK
0.86 51.37 us 22.00 us 101.00 us 134 FINODELK
1.07 118.28 us 46.00 us 437.00 us 72 READ
3.83 137.39 us 41.00 us 2431.00 us 222 WRITE
4.54 250.94 us 57.00 us 15444.00 us 144 FXATTROP
5.89 2932.56 us 134.00 us 17675.00 us 16 CREATE
7.83 224.34 us 61.00 us 12509.00 us 278 LOOKUP
73.50 9592.56 us 6897.00 us 36321.00 us 61 FSYNC
Duration: 362152 seconds
Data Read: 2005452176 bytes
Data Written: 83056358475 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 1 3 2
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 2 14 22
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 166 253 502
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 893 14392
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245861 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.01 89.00 us 89.00 us 89.00 us 1 GETXATTR
0.02 62.00 us 53.00 us 71.00 us 2 OPENDIR
0.02 57.00 us 51.00 us 61.00 us 3 FSTAT
0.04 178.00 us 38.00 us 318.00 us 2 FSYNCDIR
0.12 474.00 us 451.00 us 497.00 us 2 READDIRP
0.18 64.00 us 28.00 us 113.00 us 23 STATFS
0.28 48.30 us 18.00 us 184.00 us 46 ENTRYLK
0.31 41.33 us 18.00 us 87.00 us 60 FLUSH
0.33 66.12 us 28.00 us 246.00 us 40 STAT
0.57 102.39 us 62.00 us 236.00 us 44 OPEN
0.58 664.86 us 311.00 us 855.00 us 7 UNLINK
0.86 51.37 us 22.00 us 101.00 us 134 FINODELK
1.07 118.28 us 46.00 us 437.00 us 72 READ
3.83 137.39 us 41.00 us 2431.00 us 222 WRITE
4.54 250.94 us 57.00 us 15444.00 us 144 FXATTROP
5.89 2932.56 us 134.00 us 17675.00 us 16 CREATE
7.83 224.34 us 61.00 us 12509.00 us 278 LOOKUP
73.50 9592.56 us 6897.00 us 36321.00 us 61 FSYNC
Duration: 362152 seconds
Data Read: 2005452176 bytes
Data Written: 83056358475 bytes
Seems latency of FSYNC is very high. Sometimes cpu wait is also very high:
top - 15:27:39 up 4 days, 4:58, 2 users, load average: 0.75, 0.72, 0.70
Tasks: 210 total, 1 running, 209 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.3 us, 0.2 sy, 0.0 ni, 62.9 id, 36.5 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 8043984 total, 883420 free, 1788912 used, 5371652 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 5819396 avail Mem
Any ideas?
Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170331/455c9152/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 9491E54ADE8B47A088284EAA75AEEC31.png
Type: image/png
Size: 96159 bytes
Desc: 9491E54ADE8B47A088284EAA75AEEC31.png
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170331/455c9152/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1AF5C885C5A5497196A56960EC888E2E.png
Type: image/png
Size: 57197 bytes
Desc: 1AF5C885C5A5497196A56960EC888E2E.png
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170331/455c9152/attachment-0001.png>
Prometheus uses *LOTS* of small files and do small random-io. I do not recommend to use data volume of prometheus on glusterfs. Why you need that? On Fri, Mar 31, 2017 at 10:30 AM, Yong Zhang <hiscal at outlook.com> wrote:> Hi, All > > > > I have glusterfs 3.10.0 running with two replicated nodes, but one of the > node has a bad CPU performance: > > > > Node1/Node2: 4 core, 8GB, using thin LVM for volumes > > > > This is node1: > > > > And this is node2: > > > > Currently I have only Prometheus using glusterfs volumes, so I started > profile on volume prometheus and got this: > > > > Brick: 10.10.0.100:/glusterfsvolumes/devops/devops-prometheus/brick > > ------------------------------------------------------------------- > > Cumulative Stats: > > Block Size: 2b+ > 4b+ 8b+ > > No. of Reads: 0 > 0 0 > > No. of Writes: 1 > 1 31 > > > > Block Size: 16b+ 32b+ > 64b+ > > No. of Reads: 0 > 1 0 > > No. of Writes: 20 4283 > 3085 > > > > Block Size: 128b+ 256b+ > 512b+ > > No. of Reads: 0 > 1 0 > > No. of Writes: 395 > 465 555 > > > > Block Size: 1024b+ 2048b+ > 4096b+ > > No. of Reads: 4 > 6 32 > > No. of Writes: 72453 62462 > 52882 > > > > Block Size: 8192b+ 16384b+ > 32768b+ > > No. of Reads: 154 > 281 484 > > No. of Writes: 43122 21700 > 17817 > > > > Block Size: 65536b+ 131072b+ > > No. of Reads: 993 15290 > > No. of Writes: 56237 581071 > > %-latency Avg-latency Min-Latency Max-Latency No. of > calls Fop > > --------- ----------- ----------- ----------- ------------ > ---- > > 0.00 0.00 us 0.00 us 0.00 us 205094 > FORGET > > 0.00 0.00 us 0.00 us 0.00 us 245895 > RELEASE > > 0.00 0.00 us 0.00 us 0.00 us 40492 > RELEASEDIR > > 0.00 20.50 us 19.00 us 22.00 us 2 > FSYNCDIR > > 0.01 99.00 us 99.00 us 99.00 us 1 > GETXATTR > > 0.01 68.50 us 48.00 us 89.00 us 2 > OPENDIR > > 0.03 84.33 us 40.00 us 164.00 us 6 > FSTAT > > 0.10 64.35 us 34.00 us 177.00 us 23 > STATFS > > 0.16 62.87 us 29.00 us 172.00 us 38 > STAT > > 0.17 54.76 us 22.00 us 194.00 us 46 > ENTRYLK > > 0.18 41.66 us 24.00 us 95.00 us 62 > FLUSH > > 0.22 198.44 us 138.00 us 339.00 us 16 > CREATE > > 0.32 101.65 us 65.00 us 184.00 us 46 > OPEN > > 0.35 726.43 us 350.00 us 1165.00 us 7 > UNLINK > > 0.52 56.22 us 25.00 us 228.00 us 136 > FINODELK > > 0.77 129.15 us 49.00 us 1138.00 us 87 > READ > > 1.57 154.70 us 59.00 us 360.00 us 148 > FXATTROP > > 2.44 159.13 us 41.00 us 5836.00 us 224 > WRITE > > 3.64 188.62 us 54.00 us 848.00 us 282 > LOOKUP > > 89.50 20741.70 us 9457.00 us 84733.00 us 63 > FSYNC > > Duration: 362318 seconds > > Data Read: 2132763575 <(213)%20276-3575> bytes > > Data Written: 83056358475 bytes > > Interval 0 Stats: > > Block Size: 2b+ > 4b+ 8b+ > > No. of Reads: 0 > 0 0 > > No. of Writes: 1 > 1 31 > > > > Block Size: 16b+ 32b+ > 64b+ > > No. of Reads: 0 > 1 0 > > No. of Writes: 20 4283 > 3085 > > > > Block Size: 128b+ 256b+ > 512b+ > > No. of Reads: 0 > 1 0 > > No. of Writes: 395 > 465 555 > > > > Block Size: 1024b+ 2048b+ > 4096b+ > > No. of Reads: 4 > 6 32 > > No. of Writes: 72453 62462 > 52882 > > > > Block Size: 8192b+ 16384b+ > 32768b+ > > No. of Reads: 154 > 281 484 > > No. of Writes: 43122 21700 > 17817 > > > > Block Size: 65536b+ 131072b+ > > No. of Reads: 993 15290 > > No. of Writes: 56237 581071 > > %-latency Avg-latency Min-Latency Max-Latency No. of > calls Fop > > --------- ----------- ----------- ----------- ------------ > ---- > > 0.00 0.00 us 0.00 us 0.00 us 205094 > FORGET > > 0.00 0.00 us 0.00 us 0.00 us 245895 > RELEASE > > 0.00 0.00 us 0.00 us 0.00 us 40492 > RELEASEDIR > > 0.00 20.50 us 19.00 us 22.00 us 2 > FSYNCDIR > > 0.01 99.00 us 99.00 us 99.00 us 1 > GETXATTR > > 0.01 68.50 us 48.00 us 89.00 us 2 > OPENDIR > > 0.03 84.33 us 40.00 us 164.00 us 6 > FSTAT > > 0.10 64.35 us 34.00 us 177.00 us 23 > STATFS > > 0.16 62.87 us 29.00 us 172.00 us 38 > STAT > > 0.17 54.76 us 22.00 us 194.00 us 46 > ENTRYLK > > 0.18 41.66 us 24.00 us 95.00 us 62 > FLUSH > > 0.22 198.44 us 138.00 us 339.00 us 16 > CREATE > > 0.32 101.65 us 65.00 us 184.00 us 46 > OPEN > > 0.35 726.43 us 350.00 us 1165.00 us 7 > UNLINK > > 0.52 56.22 us 25.00 us 228.00 us 136 > FINODELK > > 0.77 129.15 us 49.00 us 1138.00 us 87 > READ > > 1.57 154.70 us 59.00 us 360.00 us 148 > FXATTROP > > 2.44 159.13 us 41.00 us 5836.00 us 224 > WRITE > > 3.64 188.62 us 54.00 us 848.00 us 282 > LOOKUP > > 89.50 20741.70 us 9457.00 us 84733.00 us 63 > FSYNC > > Duration: 362318 seconds > > Data Read: 2132763575 <(213)%20276-3575> bytes > > Data Written: 83056358475 bytes > > Brick: 10.10.0.101:/glusterfsvolumes/devops/devops-prometheus/brick > > ------------------------------------------------------------------- > > Cumulative Stats: > > Block Size: 2b+ > 4b+ 8b+ > > No. of Reads: 0 0 > 0 > > No. of Writes: 1 > 1 31 > > > > Block Size: 16b+ 32b+ > 64b+ > > No. of Reads: 0 > 0 0 > > No. of Writes: 20 4283 > 3085 > > > > Block Size: 128b+ 256b+ > 512b+ > > No. of Reads: 1 > 3 2 > > No. of Writes: 395 > 465 555 > > > > Block Size: 1024b+ 2048b+ > 4096b+ > > No. of Reads: 2 > 14 22 > > No. of Writes: 72453 62462 > 52882 > > > > Block Size: 8192b+ 16384b+ > 32768b+ > > No. of Reads: 166 > 253 502 > > No. of Writes: 43122 21700 > 17817 > > > > Block Size: 65536b+ 131072b+ > > No. of Reads: 893 14392 > > No. of Writes: 56237 581071 > > %-latency Avg-latency Min-Latency Max-Latency No. of > calls Fop > > --------- ----------- ----------- ----------- ------------ > ---- > > 0.00 0.00 us 0.00 us 0.00 us 205094 > FORGET > > 0.00 0.00 us 0.00 us 0.00 us 245861 > RELEASE > > 0.00 0.00 us 0.00 us 0.00 us 40492 > RELEASEDIR > > 0.01 89.00 us 89.00 us 89.00 us 1 > GETXATTR > > 0.02 62.00 us 53.00 us 71.00 us 2 > OPENDIR > > 0.02 57.00 us 51.00 us 61.00 us 3 > FSTAT > > 0.04 178.00 us 38.00 us 318.00 us 2 > FSYNCDIR > > 0.12 474.00 us 451.00 us 497.00 us 2 > READDIRP > > 0.18 64.00 us 28.00 us 113.00 us 23 > STATFS > > 0.28 48.30 us 18.00 us 184.00 us 46 > ENTRYLK > > 0.31 41.33 us 18.00 us 87.00 us 60 > FLUSH > > 0.33 66.12 us 28.00 us 246.00 us 40 > STAT > > 0.57 102.39 us 62.00 us 236.00 us 44 > OPEN > > 0.58 664.86 us 311.00 us 855.00 us 7 > UNLINK > > 0.86 51.37 us 22.00 us 101.00 us 134 > FINODELK > > 1.07 118.28 us 46.00 us 437.00 us 72 > READ > > 3.83 137.39 us 41.00 us 2431.00 us 222 > WRITE > > 4.54 250.94 us 57.00 us 15444.00 us 144 > FXATTROP > > 5.89 2932.56 us 134.00 us 17675.00 us 16 > CREATE > > 7.83 224.34 us 61.00 us 12509.00 us 278 > LOOKUP > > 73.50 9592.56 us 6897.00 us 36321.00 us 61 > FSYNC > > Duration: 362152 seconds > > Data Read: 2005452176 bytes > > Data Written: 83056358475 bytes > > Interval 0 Stats: > > Block Size: 2b+ > 4b+ 8b+ > > No. of Reads: 0 > 0 0 > > No. of Writes: 1 > 1 31 > > > > Block Size: 16b+ 32b+ > 64b+ > > No. of Reads: 0 > 0 0 > > No. of Writes: 20 4283 > 3085 > > > > Block Size: 128b+ 256b+ > 512b+ > > No. of Reads: 1 > 3 2 > > No. of Writes: 395 > 465 555 > > > > Block Size: 1024b+ 2048b+ > 4096b+ > > No. of Reads: 2 > 14 22 > > No. of Writes: 72453 62462 > 52882 > > > > Block Size: 8192b+ 16384b+ > 32768b+ > > No. of Reads: 166 > 253 502 > > No. of Writes: 43122 21700 > 17817 > > > > Block Size: 65536b+ 131072b+ > > No. of Reads: 893 14392 > > No. of Writes: 56237 581071 > > %-latency Avg-latency Min-Latency Max-Latency No. of > calls Fop > > --------- ----------- ----------- ----------- ------------ > ---- > > 0.00 0.00 us 0.00 us 0.00 us 205094 > FORGET > > 0.00 0.00 us 0.00 us 0.00 us 245861 > RELEASE > > 0.00 0.00 us 0.00 us 0.00 us 40492 > RELEASEDIR > > 0.01 89.00 us 89.00 us 89.00 us 1 > GETXATTR > > 0.02 62.00 us 53.00 us 71.00 us 2 > OPENDIR > > 0.02 57.00 us 51.00 us 61.00 us 3 > FSTAT > > 0.04 178.00 us 38.00 us 318.00 us 2 > FSYNCDIR > > 0.12 474.00 us 451.00 us 497.00 us 2 > READDIRP > > 0.18 64.00 us 28.00 us 113.00 us 23 > STATFS > > 0.28 48.30 us 18.00 us 184.00 us 46 > ENTRYLK > > 0.31 41.33 us 18.00 us 87.00 us 60 > FLUSH > > 0.33 66.12 us 28.00 us 246.00 us 40 > STAT > > 0.57 102.39 us 62.00 us 236.00 us 44 > OPEN > > 0.58 664.86 us 311.00 us 855.00 us 7 > UNLINK > > 0.86 51.37 us 22.00 us 101.00 us 134 > FINODELK > > 1.07 118.28 us 46.00 us 437.00 us 72 > READ > > 3.83 137.39 us 41.00 us 2431.00 us 222 > WRITE > > 4.54 250.94 us 57.00 us 15444.00 us 144 > FXATTROP > > 5.89 2932.56 us 134.00 us 17675.00 us 16 > CREATE > > 7.83 224.34 us 61.00 us 12509.00 us 278 > LOOKUP > > 73.50 9592.56 us 6897.00 us 36321.00 us 61 > FSYNC > > Duration: 362152 seconds > > Data Read: 2005452176 bytes > > Data Written: 83056358475 bytes > > > > > > Seems latency of FSYNC is very high. Sometimes cpu wait is also very high: > > > > top - 15:27:39 up 4 days, 4:58, 2 users, load average: 0.75, 0.72, 0.70 > > Tasks: 210 total, 1 running, 209 sleeping, 0 stopped, 0 zombie > > %Cpu(s): 0.3 us, 0.2 sy, 0.0 ni, 62.9 id, *36.5* wa, 0.0 hi, 0.1 > si, 0.0 st > > KiB Mem : 8043984 total, 883420 free, 1788912 used, 5371652 buff/cache > > KiB Swap: 0 total, 0 free, 0 used. 5819396 avail Mem > > Any ideas? > > > > Thanks. > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170331/255e1e1d/attachment-0001.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: 9491E54ADE8B47A088284EAA75AEEC31.png Type: image/png Size: 96159 bytes Desc: not available URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170331/255e1e1d/attachment-0002.png> -------------- next part -------------- A non-text attachment was scrubbed... Name: 1AF5C885C5A5497196A56960EC888E2E.png Type: image/png Size: 57197 bytes Desc: not available URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170331/255e1e1d/attachment-0003.png>
Hi, Serkan
I?m using Glusterfs as my Kubernetes storage backend, and Prometheus is running
inside Kubernetes cluster. If this is not recommended I?ll consider running
Prometheus outside of Kubernetes. But why performance is totally different on
two nodes? Will data be written to both nodes at the same time?
From: Serkan ?oban<mailto:cobanserkan at gmail.com>
Sent: Friday, March 31, 2017 4:08 PM
To: Yong Zhang<mailto:hiscal at outlook.com>
Cc: gluster-users at gluster.org<mailto:gluster-users at gluster.org>
Subject: Re: [Gluster-users] High CPU/load, High latency on FSYNC
Prometheus uses *LOTS* of small files and do small random-io.
I do not recommend to use data volume of prometheus on glusterfs.
Why you need that?
On Fri, Mar 31, 2017 at 10:30 AM, Yong Zhang <hiscal at
outlook.com<mailto:hiscal at outlook.com>> wrote:
Hi, All
I have glusterfs 3.10.0 running with two replicated nodes, but one of the node
has a bad CPU performance:
Node1/Node2: 4 core, 8GB, using thin LVM for volumes
This is node1:
[cid:image005.png at 01D2AA33.B41E14D0]
And this is node2:
[cid:image006.png at 01D2AA33.B41E14D0]
Currently I have only Prometheus using glusterfs volumes, so I started profile
on volume prometheus and got this:
Brick: 10.10.0.100:/glusterfsvolumes/devops/devops-prometheus/brick
-------------------------------------------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 1 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 1 0
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 4 6 32
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 154 281 484
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 993 15290
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245895 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.00 20.50 us 19.00 us 22.00 us 2 FSYNCDIR
0.01 99.00 us 99.00 us 99.00 us 1 GETXATTR
0.01 68.50 us 48.00 us 89.00 us 2 OPENDIR
0.03 84.33 us 40.00 us 164.00 us 6 FSTAT
0.10 64.35 us 34.00 us 177.00 us 23 STATFS
0.16 62.87 us 29.00 us 172.00 us 38 STAT
0.17 54.76 us 22.00 us 194.00 us 46 ENTRYLK
0.18 41.66 us 24.00 us 95.00 us 62 FLUSH
0.22 198.44 us 138.00 us 339.00 us 16 CREATE
0.32 101.65 us 65.00 us 184.00 us 46 OPEN
0.35 726.43 us 350.00 us 1165.00 us 7 UNLINK
0.52 56.22 us 25.00 us 228.00 us 136 FINODELK
0.77 129.15 us 49.00 us 1138.00 us 87 READ
1.57 154.70 us 59.00 us 360.00 us 148 FXATTROP
2.44 159.13 us 41.00 us 5836.00 us 224 WRITE
3.64 188.62 us 54.00 us 848.00 us 282 LOOKUP
89.50 20741.70 us 9457.00 us 84733.00 us 63 FSYNC
Duration: 362318 seconds
Data Read: 2132763575<tel:(213)%20276-3575> bytes
Data Written: 83056358475 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 1 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 1 0
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 4 6 32
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 154 281 484
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 993 15290
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245895 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.00 20.50 us 19.00 us 22.00 us 2 FSYNCDIR
0.01 99.00 us 99.00 us 99.00 us 1 GETXATTR
0.01 68.50 us 48.00 us 89.00 us 2 OPENDIR
0.03 84.33 us 40.00 us 164.00 us 6 FSTAT
0.10 64.35 us 34.00 us 177.00 us 23 STATFS
0.16 62.87 us 29.00 us 172.00 us 38 STAT
0.17 54.76 us 22.00 us 194.00 us 46 ENTRYLK
0.18 41.66 us 24.00 us 95.00 us 62 FLUSH
0.22 198.44 us 138.00 us 339.00 us 16 CREATE
0.32 101.65 us 65.00 us 184.00 us 46 OPEN
0.35 726.43 us 350.00 us 1165.00 us 7 UNLINK
0.52 56.22 us 25.00 us 228.00 us 136 FINODELK
0.77 129.15 us 49.00 us 1138.00 us 87 READ
1.57 154.70 us 59.00 us 360.00 us 148 FXATTROP
2.44 159.13 us 41.00 us 5836.00 us 224 WRITE
3.64 188.62 us 54.00 us 848.00 us 282 LOOKUP
89.50 20741.70 us 9457.00 us 84733.00 us 63 FSYNC
Duration: 362318 seconds
Data Read: 2132763575<tel:(213)%20276-3575> bytes
Data Written: 83056358475 bytes
Brick: 10.10.0.101:/glusterfsvolumes/devops/devops-prometheus/brick
-------------------------------------------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 1 3 2
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 2 14 22
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 166 253 502
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 893 14392
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245861 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.01 89.00 us 89.00 us 89.00 us 1 GETXATTR
0.02 62.00 us 53.00 us 71.00 us 2 OPENDIR
0.02 57.00 us 51.00 us 61.00 us 3 FSTAT
0.04 178.00 us 38.00 us 318.00 us 2 FSYNCDIR
0.12 474.00 us 451.00 us 497.00 us 2 READDIRP
0.18 64.00 us 28.00 us 113.00 us 23 STATFS
0.28 48.30 us 18.00 us 184.00 us 46 ENTRYLK
0.31 41.33 us 18.00 us 87.00 us 60 FLUSH
0.33 66.12 us 28.00 us 246.00 us 40 STAT
0.57 102.39 us 62.00 us 236.00 us 44 OPEN
0.58 664.86 us 311.00 us 855.00 us 7 UNLINK
0.86 51.37 us 22.00 us 101.00 us 134 FINODELK
1.07 118.28 us 46.00 us 437.00 us 72 READ
3.83 137.39 us 41.00 us 2431.00 us 222 WRITE
4.54 250.94 us 57.00 us 15444.00 us 144 FXATTROP
5.89 2932.56 us 134.00 us 17675.00 us 16 CREATE
7.83 224.34 us 61.00 us 12509.00 us 278 LOOKUP
73.50 9592.56 us 6897.00 us 36321.00 us 61 FSYNC
Duration: 362152 seconds
Data Read: 2005452176 bytes
Data Written: 83056358475 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 1 3 2
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 2 14 22
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 166 253 502
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 893 14392
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245861 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.01 89.00 us 89.00 us 89.00 us 1 GETXATTR
0.02 62.00 us 53.00 us 71.00 us 2 OPENDIR
0.02 57.00 us 51.00 us 61.00 us 3 FSTAT
0.04 178.00 us 38.00 us 318.00 us 2 FSYNCDIR
0.12 474.00 us 451.00 us 497.00 us 2 READDIRP
0.18 64.00 us 28.00 us 113.00 us 23 STATFS
0.28 48.30 us 18.00 us 184.00 us 46 ENTRYLK
0.31 41.33 us 18.00 us 87.00 us 60 FLUSH
0.33 66.12 us 28.00 us 246.00 us 40 STAT
0.57 102.39 us 62.00 us 236.00 us 44 OPEN
0.58 664.86 us 311.00 us 855.00 us 7 UNLINK
0.86 51.37 us 22.00 us 101.00 us 134 FINODELK
1.07 118.28 us 46.00 us 437.00 us 72 READ
3.83 137.39 us 41.00 us 2431.00 us 222 WRITE
4.54 250.94 us 57.00 us 15444.00 us 144 FXATTROP
5.89 2932.56 us 134.00 us 17675.00 us 16 CREATE
7.83 224.34 us 61.00 us 12509.00 us 278 LOOKUP
73.50 9592.56 us 6897.00 us 36321.00 us 61 FSYNC
Duration: 362152 seconds
Data Read: 2005452176 bytes
Data Written: 83056358475 bytes
Seems latency of FSYNC is very high. Sometimes cpu wait is also very high:
top - 15:27:39 up 4 days, 4:58, 2 users, load average: 0.75, 0.72, 0.70
Tasks: 210 total, 1 running, 209 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.3 us, 0.2 sy, 0.0 ni, 62.9 id, 36.5 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 8043984 total, 883420 free, 1788912 used, 5371652 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 5819396 avail Mem
Any ideas?
Thanks.
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170401/00224335/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 9491E54ADE8B47A088284EAA75AEEC31.png
Type: image/png
Size: 96159 bytes
Desc: 9491E54ADE8B47A088284EAA75AEEC31.png
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170401/00224335/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1AF5C885C5A5497196A56960EC888E2E.png
Type: image/png
Size: 57197 bytes
Desc: 1AF5C885C5A5497196A56960EC888E2E.png
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170401/00224335/attachment-0001.png>