Bryan Whitehead
2012-Jul-12 21:47 UTC
[Gluster-users] RDMA high cpu usage and poor performance
I see both glusterfsd and glusterfs eat a good 70-100% of CPU while dd
runs (see below)
[root at lab0 ~]# gluster volume info
Volume Name: testrdma
Type: Replicate
Volume ID: bf7b42aa-5680-4f5c-8027-d0a56cc5e65d
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: rdma
Bricks:
Brick1: 10.1.0.10:/glust
Brick2: 10.1.0.11:/glust
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
performance.io-thread-count: 64
[root at lab0 ~]#
output from dd bs=1M if=/dev/zero of=/test/test.file :
29992+0 records in
29992+0 records out
31448891392 bytes (31 GB) copied, 319.076 s, 98.6 MB/s
[root at lab0 ~]# gluster volume profile testrdma info
Brick: 10.1.0.10:/glust
-----------------------
Cumulative Stats:
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 245984
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 FORGET
0.00 0.00 us 0.00 us 0.00 us 5 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1 RELEASEDIR
0.00 10.50 us 9.00 us 12.00 us 2 FLUSH
0.00 13.00 us 11.00 us 15.00 us 2 ENTRYLK
0.00 28.00 us 28.00 us 28.00 us 1 GETXATTR
0.00 59.50 us 18.00 us 101.00 us 2 READDIR
0.00 201.00 us 201.00 us 201.00 us 1 CREATE
0.00 56.00 us 18.00 us 93.00 us 5 LOOKUP
17.89 123.50 us 47.00 us 29371.00 us 239936 WRITE
23.06 79.60 us 2.00 us 41325.00 us 479877 FINODELK
59.06 227.42 us 18.00 us 52835.00 us 430226 FXATTROP
Duration: 3790 seconds
Data Read: 0 bytes
Data Written: 32241614848 bytes
Interval 3 Stats:
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 82837
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 RELEASE
0.00 9.00 us 9.00 us 9.00 us 1 FLUSH
0.00 18.00 us 18.00 us 18.00 us 1 LOOKUP
0.00 28.00 us 28.00 us 28.00 us 1 GETXATTR
0.00 59.50 us 18.00 us 101.00 us 2 READDIR
16.61 129.30 us 47.00 us 23023.00 us 82837 WRITE
19.79 77.02 us 3.00 us 30051.00 us 165668 FINODELK
63.61 282.00 us 37.00 us 48149.00 us 145458 FXATTROP
Duration: 472 seconds
Data Read: 0 bytes
Data Written: 10857611264 bytes
Brick: 10.1.0.11:/glust
-----------------------
Cumulative Stats:
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 245984
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 FORGET
0.00 0.00 us 0.00 us 0.00 us 5 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1 RELEASEDIR
0.00 13.00 us 12.00 us 14.00 us 2 FLUSH
0.00 28.00 us 28.00 us 28.00 us 1 GETXATTR
0.00 32.00 us 25.00 us 39.00 us 2 ENTRYLK
0.00 63.50 us 32.00 us 95.00 us 2 READDIR
0.00 179.00 us 179.00 us 179.00 us 1 CREATE
0.00 57.40 us 27.00 us 87.00 us 5 LOOKUP
7.76 177.75 us 40.00 us 54211.00 us 239936 WRITE
44.67 512.94 us 17.00 us 75783.00 us 478774 FXATTROP
47.57 544.98 us 2.00 us 99430.00 us 479877 FINODELK
Duration: 3790 seconds
Data Read: 0 bytes
Data Written: 32241614848 bytes
Interval 3 Stats:
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 82837
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 RELEASE
0.00 14.00 us 14.00 us 14.00 us 1 FLUSH
0.00 27.00 us 27.00 us 27.00 us 1 LOOKUP
0.00 28.00 us 28.00 us 28.00 us 1 GETXATTR
0.00 63.50 us 32.00 us 95.00 us 2 READDIR
7.43 218.70 us 40.00 us 37871.00 us 82837 WRITE
45.45 670.49 us 37.00 us 75783.00 us 165308 FXATTROP
47.12 693.66 us 2.00 us 99430.00 us 165678 FINODELK
Duration: 472 seconds
Data Read: 0 bytes
Data Written: 10857611264 bytes
I'll open a bug once redhat gets out of maintenance mode.
-Bryan