[Gluster-users] High load on glusterfs client

Sebastian.Gumprich at t-systems.com
Mon, 14 Mar 2016 07:35:55 -0000

Today, after setting the self-heal options to off, the problem happened again.

Here?s what happens when I try to heal the problems:

[root at nfs02 gluster volume heal opt info
Brick nfs01:/opt/bkk
Number of entries: 0

Brick nfs02:/opt/bkk
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Standalone_template_source_948126c20bcb68e1839bd1cb11b2326776a5a89a.php
<gfid:869bab4e-f83f-4876-b619-a7c008361bd2>
<gfid:e5cd468a-c3cc-453f-bd10-67d4474bd57c>
<gfid:e792eef8-a58f-4808-985b-dc906a707593>
<gfid:7ab28ace-94ae-468e-9a11-0b5a3a42a9d7>
<gfid:ae890cd6-97b8-493a-a6ea-bedf45707f17>
<gfid:7bd13cdb-4c6c-4aa5-a7df-1f3cf78bfca3>
<gfid:d7f63c6b-0b69-4271-827e-16bb1bfa689e>
<gfid:a128f009-1edc-47fb-8e53-9490887f9937>
<gfid:d0a0d264-ebc2-4370-8114-4732fb894e66>
<gfid:317f6a2e-f243-4ead-a4f4-8508b70d76d2>
<gfid:082a3550-15ad-4719-a550-c60c7f8c8791>
<gfid:e5d7c0fb-42e5-4f81-aebd-9f55fb12d8cf>
<gfid:4ec4830b-c65e-451b-a754-d48bd4c385a6>
<gfid:f0453756-50b4-423f-84b5-82bf840ae465>
<gfid:264d0bba-7208-4866-9820-e561439afbb8>
<gfid:ebe43f9f-6b55-40bb-8a59-750ab3dc7b1f>
<gfid:6f235178-5981-42d8-83fe-ce222c22600a>
<gfid:c2ee0feb-8edc-4fe3-bda6-55845c6ab351>
/releases/1.0.1/typo3temp/locks/b3e7327e62654b234caf95033dd5f5db
<gfid:f450c66b-d6ae-4c65-a95a-f13ef5285900>
<gfid:98aabb13-dafe-474c-87e0-2741645a1e6b>
<gfid:1bebfba1-13c1-4c22-8b1b-3d34e5d7b968>
<gfid:97e5a8cc-7c66-47b0-8dbd-9f9098b419d9>
<gfid:438a2995-32d2-4a21-ae5d-45593a74addf>
<gfid:008f5f63-9dda-408b-abf7-2201b029462e>
<gfid:f89a5507-f9c7-45f1-8760-ea9a897c4659>
<gfid:3da89478-9c08-4c57-a44f-0df2dc59efb4>
<gfid:f0969ff6-f49a-4497-a24f-648d4e130775>
<gfid:6c7f9d1e-64e5-43f7-a454-9f823641f0ae>
<gfid:1d7c6cc0-fd07-43bb-8caf-30e80544e516>
/releases/1.0.1/typo3temp/locks/dabccbb64a6426cf9b1005b79f5765a9
Number of entries: 32

[root at nfs02 ~]# gluster volume heal opt full
Launching heal operation to perform full self heal on volume opt has been
unsuccessful

[root at nfs02 ~]# gluster volume heal opt
Launching heal operation to perform index self heal on volume opt has been
successful
Use heal info commands to check status

[root at nfs02 ~]# gluster volume heal opt info
Brick nfs01:/opt/bkk
Number of entries: 0

Brick nfs02:/opt/bkk
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Standalone_template_source_948126c20bcb68e1839bd1cb11b2326776a5a89a.php
<gfid:869bab4e-f83f-4876-b619-a7c008361bd2>
<gfid:e5cd468a-c3cc-453f-bd10-67d4474bd57c>
<gfid:e792eef8-a58f-4808-985b-dc906a707593>
<gfid:7ab28ace-94ae-468e-9a11-0b5a3a42a9d7>
<gfid:ae890cd6-97b8-493a-a6ea-bedf45707f17>
<gfid:7bd13cdb-4c6c-4aa5-a7df-1f3cf78bfca3>
<gfid:d7f63c6b-0b69-4271-827e-16bb1bfa689e>
<gfid:a128f009-1edc-47fb-8e53-9490887f9937>
<gfid:d0a0d264-ebc2-4370-8114-4732fb894e66>
<gfid:317f6a2e-f243-4ead-a4f4-8508b70d76d2>
<gfid:082a3550-15ad-4719-a550-c60c7f8c8791>
<gfid:e5d7c0fb-42e5-4f81-aebd-9f55fb12d8cf>
<gfid:4ec4830b-c65e-451b-a754-d48bd4c385a6>
<gfid:f0453756-50b4-423f-84b5-82bf840ae465>
<gfid:264d0bba-7208-4866-9820-e561439afbb8>
<gfid:ebe43f9f-6b55-40bb-8a59-750ab3dc7b1f>
<gfid:6f235178-5981-42d8-83fe-ce222c22600a>
<gfid:c2ee0feb-8edc-4fe3-bda6-55845c6ab351>
/releases/1.0.1/typo3temp/locks/b3e7327e62654b234caf95033dd5f5db
<gfid:f450c66b-d6ae-4c65-a95a-f13ef5285900>
<gfid:98aabb13-dafe-474c-87e0-2741645a1e6b>
<gfid:1bebfba1-13c1-4c22-8b1b-3d34e5d7b968>
<gfid:97e5a8cc-7c66-47b0-8dbd-9f9098b419d9>
<gfid:438a2995-32d2-4a21-ae5d-45593a74addf>
<gfid:008f5f63-9dda-408b-abf7-2201b029462e>
<gfid:f89a5507-f9c7-45f1-8760-ea9a897c4659>
<gfid:3da89478-9c08-4c57-a44f-0df2dc59efb4>
<gfid:f0969ff6-f49a-4497-a24f-648d4e130775>
<gfid:6c7f9d1e-64e5-43f7-a454-9f823641f0ae>
<gfid:1d7c6cc0-fd07-43bb-8caf-30e80544e516>
Number of entries: 31


Here?s the log to it:
[2016-03-14 07:25:34.593287] W [MSGID: 108008]
[afr-self-heal-name.c:359:afr_selfheal_name_gfid_mismatch_check]
0-opt-replicate-0: GFID mismatch for
<gfid:7bfa065f-50c7-43d6-b513-df6cd76fa144>/Standalone_layout_Mail_fa22189c871e1e7c1a36b59209ef0812d5e709e9.php
082a3550-15ad-4719-a550-c60c7f8c8791 on opt-client-1 and
c2f289ca-6058-46ae-af8a-18491975eb7d on opt-client-0


[root at -nfs02 ~]# gluster volume info

Volume Name: opt
Type: Replicate
Volume ID: 5b77070f-5378-45ec-9eda-5f7dd007ff8a
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: nfs01:/opt/bkk
Brick2: nfs02:/opt/bkk
Options Reconfigured:
cluster.metadata-self-heal: off
cluster.data-self-heal: off
cluster.entry-self-heal: off
performance.md-cache-timeout: 1
performance.cache-max-file-size: 2MB
performance.io-thread-count: 16
network.ping-timeout: 42
performance.write-behind-window-size: 4MB
performance.read-ahead: off
performance.cache-refresh-timeout: 10
performance.cache-size: 512MB
performance.quick-read: off
performance.readdir-ahead: on


Any other ideas?

Thanks!

Von: Krutika Dhananjay [mailto:kdhananj at redhat.com]
Gesendet: Dienstag, 8. M?rz 2016 04:04
An: Gumprich, Sebastian
Cc: gluster-users at gluster.org
Betreff: Re: [Gluster-users] High load on glusterfs client

Could you try disabling client-side heals and see if it works for you?
Here's what you'd need to do:
#gluster volume set <VOL> entry-self-heal off
#gluster volume set <VOL> data-self-heal off
#gluster volume set <VOL> metadata-self-heal off
-Krutika

On Wed, Mar 2, 2016 at 12:37 AM, <Sebastian.Gumprich at
t-systems.com<mailto:Sebastian.Gumprich at t-systems.com>> wrote:
Hello everyone,

I?m experiencing high load on our glusterfs clients.

Here?s the setup:

There are to glusterfs server:

Nfs01 and nfs02 with the following configuration:

[root nfs01 ~]# gluster volume info opt

Volume Name: opt
Type: Replicate
Volume ID: 5b77070f-5378-45ec-9eda-5f7dd007ff8a
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1:  nfs01:/opt/bkk
Brick2: nfs02:/opt/bkk
Options Reconfigured:
performance.readdir-ahead: on
performance.quick-read: off
performance.cache-size: 512MB
performance.cache-refresh-timeout: 10
performance.read-ahead: off
performance.write-behind-window-size: 4MB
network.ping-timeout: 2
performance.io-thread-count: 16
performance.cache-max-file-size: 2MB
performance.md-cache-timeout: 1

Then there are two clients (web01 and web02) that mount the brick via a virtual
ip-address (nfs-VIP):
nfs-VIP:/opt on /opt/bkk type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

operating system on all server is CentOS Linux release 7.2.1511 (Core).
Glusterfs version is glusterfs 3.7.6 built on Nov  9 2015 15:20:26

On the brick lies the PHP dynamic webcontent from a typo3 CMS.

On the client (web01) the following is logged in the gluster.log:

iner_08850598886fb5f39c9cf1d269d7e20677f97ede.php>,
e09948dd-1e9b-4430-8f55-3df64cda2385 on opt-client-1 and
ba80a475-7b83-4c83-bd0c-798a108bfb63 on opt-client-0. Skipping conservative
merge on the file.
[2016-03-01 18:40:50.570040] E [MSGID: 108008]
[afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch]
0-opt-replicate-0: Gfid mismatch detected for
<4c6dda77-6a2b-4996-bca4-9ace4cee45cc/News_News_layout_Detail_html_bd113d9c433c8f88376e47547db3b94e698a5ecd.php>,
739ee14c-2d5d-458b-bffd-83595bfcbe6a on opt-client-1 and
5a311733-731e-4478-ad3c-a70fbf66ba30 on opt-client-0. Skipping conservative
merge on the file.
[2016-03-01 18:40:50.572992] E [MSGID: 108008]
[afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch]
0-opt-replicate-0: Gfid mismatch detected for
<4c6dda77-6a2b-4996-bca4-9ace4cee45cc/News_News_partial_Detail_FalMediaContainer_9c1b3fd40fca9019726b3f6b8bc04618ffadab7b.php>,
bb6907a1-ce80-4e03-92df-6fbc69d24a4d on opt-client-1 and
6f88aa67-cb81-4c26-94b2-e3aaa8704e8d on opt-client-0. Skipping conservative
merge on the file.
[2016-03-01 18:40:50.791704] E [MSGID: 108008]
[afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch]
0-opt-replicate-0: Gfid mismatch detected for
<4c6dda77-6a2b-4996-bca4-9ace4cee45cc/Powermail_Form_action_create_f40464a6a7f73d86cda514065167d59a7ddece73.php>,
5e5b224b-ea20-4d38-8504-61b24f5d6a3b on opt-client-1 and
fab07af5-2aa5-4873-a6e2-6265ec78e304 on opt-client-0. Skipping conservative
merge on the file.
[2016-03-01 18:40:54.085964] E [MSGID: 108008]
[afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch]
0-opt-replicate-0: Gfid mismatch detected for
<4c6dda77-6a2b-4996-bca4-9ace4cee45cc/News_News_action_detail_8d30b654cd8343fe40616b8a2f8a5343b1ed776e.php>,
4d75a687-b9ab-4f97-b698-38668d1981ae on opt-client-1 and
110b315e-2e28-4859-a8b9-e0f1629faa3c on opt-client-0. Skipping conservative
merge on the file.
[2016-03-01 18:40:56.153651] E [MSGID: 108008]
[afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch]
0-opt-replicate-0: Gfid mismatch detected for
<4c6dda77-6a2b-4996-bca4-9ace4cee45cc/Powermail_Form_layout_Default_aae217b167ad82f4b1258bb01fa73f305844dbd8.php>,
6f7e2709-8c14-486a-85a2-a3cb48af4ca5 on opt-client-1 and
6ab62408-0406-4834-96b9-a51e18441d4c on opt-client-0. Skipping conservative
merge on the file.
[2016-03-01 18:41:05.476126] I [MSGID: 108026]
[afr-self-heal-entry.c:593:afr_selfheal_entry_do] 0-opt-replicate-0: performing
entry selfheal on 7a922c37-48d0-4dfb-8abb-18a435c948af
[2016-03-01 18:41:05.597093] I [MSGID: 108026]
[afr-self-heal-common.c:651:afr_log_selfheal] 0-opt-replicate-0: Completed entry
selfheal on 7a922c37-48d0-4dfb-8abb-18a435c948af. source=1 sinks=0
[2016-03-01 18:41:05.790944] E [MSGID: 108008]
[afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch]
0-opt-replicate-0: Gfid mismatch detected for
<4c6dda77-6a2b-4996-bca4-9ace4cee45cc/Powermail_Form_action_form_f0755f8526150f023fd98252b510a40c49586dbd.php>,
118668d9-608a-477a-b655-bcc6c2298bf4 on opt-client-1 and
a87943a4-e18a-4642-adff-1ad765496533 on opt-client-0. Skipping conservative
merge on the file.
[2016-03-01 18:41:06.649695] W [MSGID: 108008]
[afr-self-heal-name.c:359:afr_selfheal_name_gfid_mismatch_check]
0-opt-replicate-0: GFID mismatch for
<gfid:4c6dda77-6a2b-4996-bca4-9ace4cee45cc>/Powermail_Form_action_form_f0755f8526150f023fd98252b510a40c49586dbd.php
118668d9-608a-477a-b655-bcc6c2298bf4 on opt-client-1 and
a87943a4-e18a-4642-adff-1ad765496533 on opt-client-0
[2016-03-01 18:41:06.661277] W [fuse-bridge.c:462:fuse_entry_cbk]
0-glusterfs-fuse: 184415191: LOOKUP()
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_action_form_f0755f8526150f023fd98252b510a40c49586dbd.php
=> -1 (Input/output error)
[2016-03-01 18:41:06.680968] W [fuse-bridge.c:462:fuse_entry_cbk]
0-glusterfs-fuse: 184422672: LOOKUP()
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_action_form_f0755f8526150f023fd98252b510a40c49586dbd.php
=> -1 (Input/output error)
[2016-03-01 18:41:06.680222] W [MSGID: 108008]
[afr-self-heal-name.c:359:afr_selfheal_name_gfid_mismatch_check]
0-opt-replicate-0: GFID mismatch for
<gfid:4c6dda77-6a2b-4996-bca4-9ace4cee45cc>/Powermail_Form_action_form_f0755f8526150f023fd98252b510a40c49586dbd.php
118668d9-608a-477a-b655-bcc6c2298bf4 on opt-client-1 and
a87943a4-e18a-4642-adff-1ad765496533 on opt-client-0

There are many more of these entries, this is just a really small excerpt. The
files that have a mismatch are tempary php-cache files.
When I delete these files, the load goes down and the files in the volume heal
info become less (see below).

Here?s the output of gluster volume heal opt info. Note that this output is
*after* deleting most of the cache files, before that there were many more
entries.

[root at nfs01 fluid_template]# gluster volume heal opt info
Brick nfs01:/opt/bkk
<gfid:23fc1027-0aec-4b84-9ffb-c164a9d43d20>
<gfid:92cb9dde-2721-4c11-93a6-2582ed9edd5d>
<gfid:a0dbcf8a-67f8-4870-ab57-3d5d1218601c>
<gfid:947cbcc4-1978-4b9e-b726-2acd0a4fda5a>
<gfid:440b9b36-bad5-4cb6-b935-8a004132340a>
Number of entries: 5

Brick nfs02:/opt/bkk
/releases/1.0.1/typo3temp/Cache/Code/fluid_template - Possibly undergoing heal

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Select_7bf809152d985037de761d8d375d286e44b4f13a.php
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Misc_GoogleAdwordsConversion_f7254aeb252ea43cd89f9051b5a43109d47938f1.php
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_File_4b3a3f667c475577847aa77118f3af5666ecb2c6.php
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_action_form_f0755f8526150f023fd98252b510a40c49586dbd.php
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Input_b3e08744b23680f0a14e60e716f9994d9580e3f4.php
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/News_News_action_detail_8d30b654cd8343fe40616b8a2f8a5343b1ed776e.php
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/News_News_layout_Detail_html_bd113d9c433c8f88376e47547db3b94e698a5ecd.php
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/News_News_partial_Detail_Opengraph_b98680f3686dccf00e22181e66d11ca9de7a44bd.php
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/News_News_partial_Detail_FalMediaContainer_9c1b3fd40fca9019726b3f6b8bc04618ffadab7b.php
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/News_News_partial_Detail_MediaContainer_08850598886fb5f39c9cf1d269d7e20677f97ede.php
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Text_c10766db8d335d5cd9555878aef5d886dcb6926e.php
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_action_create_f40464a6a7f73d86cda514065167d59a7ddece73.php
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_layout_Default_aae217b167ad82f4b1258bb01fa73f305844dbd8.php
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Misc_HoneyPod_fc83c414f744612c3cb44c8827372a30f17791d0.php
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Textarea_39d24d8e3e2813636dfff2a89b7cefb8e9117c97.php
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Submit_86e69c50ccebf20584db2e3c74859373c53d320f.php
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Check_a2a11c64ac58dab16eab29e4cda88518c15a4d25.php
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Misc_FormError_7cade8e8fc1d23c761360c0efbe8cb145eed2e39.php
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Radio_d038a263b5ea81f0f7795e1e47516e5e2937cbd9.php
/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Hidden_a7651f5498e0d36b4e2eae5fca015ef0e9365067.php
Number of entries: 21

Here are some heal-infos during the high load:
Starting time of crawl: Tue Mar  1 19:09:45 2016

Ending time of crawl: Tue Mar  1 19:09:52 2016

Type of crawl: INDEX
No. of entries healed: 2
No. of entries in split-brain: 0
No. of heal failed entries: 168


And here?s the performance monitoring info during 60 seconds of high load:

Brick: nfs01:/opt/bkk
------------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+
No. of Reads:                   18                    33                   308
No. of Writes:                   64                    88                  2994

   Block Size:                  8b+                  16b+                  32b+
No. of Reads:                  117                   154                 15612
No. of Writes:                  370                   369                  1432

   Block Size:                 64b+                 128b+                 256b+
No. of Reads:                 3721                 12884                 19917
No. of Writes:                 7585                900221                135011

   Block Size:                512b+                1024b+                2048b+
No. of Reads:                23929                 12251                 19835
No. of Writes:                63067                 30950                 23540

   Block Size:               4096b+                8192b+               16384b+
No. of Reads:                 9096                  9449                  5566
No. of Writes:                40455                 36397                 13926

   Block Size:              32768b+               65536b+              131072b+
No. of Reads:                 5159                  6055                 34001
No. of Writes:                20722                  6600                 12762

%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         212065      FORGET
      0.00       0.00 us       0.00 us       0.00 us        4118713     RELEASE
      0.00       0.00 us       0.00 us       0.00 us        9931097  RELEASEDIR
      0.00      47.00 us      47.00 us      47.00 us              1    GETXATTR
      0.00     124.00 us     124.00 us     124.00 us              1     XATTROP
      0.00     144.00 us     144.00 us     144.00 us              1      UNLINK
      0.00      37.17 us      35.00 us      41.00 us              6      STATFS
      0.01      36.84 us      32.00 us      61.00 us             19       FSTAT
      0.01      48.80 us      44.00 us      62.00 us             20        STAT
      0.05      75.52 us      54.00 us     156.00 us             48 REMOVEXATTR
      0.05      81.69 us      69.00 us     146.00 us             48     SETATTR
      0.05      37.16 us      14.00 us     436.00 us            109       FLUSH
      0.05      38.31 us      18.00 us     115.00 us            108    FINODELK
      0.06      73.16 us      46.00 us     170.00 us             63        OPEN
      0.10      38.70 us      20.00 us     167.00 us            195     INODELK
      0.11     335.37 us      40.00 us     563.00 us             27     READDIR
      0.15      50.58 us      27.00 us     392.00 us            232     OPENDIR
      0.15      81.74 us      35.00 us     215.00 us            144    FXATTROP
      0.16     130.53 us      68.00 us     697.00 us            100       WRITE
      0.93    1532.04 us     171.00 us   15356.00 us             48      CREATE
      1.40     284.90 us      25.00 us    1146.00 us            390    READDIRP
     10.20      52.29 us      28.00 us    2323.00 us          15482    READLINK
     18.94      33.25 us      11.00 us   27242.00 us          45219     ENTRYLK
     67.58      93.74 us      32.00 us   27521.00 us          57223      LOOKUP

    Duration: 6492593 seconds
   Data Read: 5679496942 bytes
Data Written: 4510536316 bytes

Interval 1 Stats:
   Block Size:               4096b+                8192b+               16384b+
No. of Reads:                    0                     0                     0
No. of Writes:                    1                    50                     2

   Block Size:              32768b+
No. of Reads:                    0
No. of Writes:                   47
%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             48      FORGET
      0.00       0.00 us       0.00 us       0.00 us            104     RELEASE
      0.00       0.00 us       0.00 us       0.00 us            231  RELEASEDIR
      0.00     124.00 us     124.00 us     124.00 us              1     XATTROP
      0.00     144.00 us     144.00 us     144.00 us              1      UNLINK
      0.00      36.40 us      35.00 us      37.00 us              5      STATFS
      0.01      36.84 us      32.00 us      61.00 us             19       FSTAT
      0.01      48.80 us      44.00 us      62.00 us             20        STAT
      0.05      75.52 us      54.00 us     156.00 us             48 REMOVEXATTR
      0.05      37.69 us      14.00 us     436.00 us            101       FLUSH
      0.06      81.69 us      69.00 us     146.00 us             48     SETATTR
      0.06      74.26 us      46.00 us     170.00 us             53        OPEN
      0.06      38.31 us      18.00 us     115.00 us            108    FINODELK
      0.10     311.33 us      40.00 us     563.00 us             24     READDIR
      0.11      38.70 us      20.00 us     167.00 us            195     INODELK
      0.16      50.58 us      27.00 us     392.00 us            231     OPENDIR
      0.17      81.74 us      35.00 us     215.00 us            144    FXATTROP
      0.18     130.53 us      68.00 us     697.00 us            100       WRITE
      1.03    1532.04 us     171.00 us   15356.00 us             48      CREATE
      1.56     284.90 us      25.00 us    1146.00 us            390    READDIRP
     11.31      52.27 us      28.00 us    2323.00 us          15395    READLINK
     18.24      33.67 us      11.00 us   27242.00 us          38571     ENTRYLK
     66.83      94.49 us      32.00 us   27521.00 us          50338      LOOKUP

    Duration: 68 seconds
   Data Read: 0 bytes
Data Written: 3347998 bytes

Brick: nfs02:/opt/bkk
------------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+
No. of Reads:                   26                    49                   541
No. of Writes:                   64                    94                  3848

   Block Size:                  8b+                  16b+                  32b+
No. of Reads:                  218                   205                  1267
No. of Writes:                  452                   417                  1448

   Block Size:                 64b+                 128b+                 256b+
No. of Reads:                 6097                 39042                 11111
No. of Writes:                 8617                924503                136768

   Block Size:                512b+                1024b+                2048b+
No. of Reads:               120819                 37802                 16506
No. of Writes:                64399                 35996                 24999

   Block Size:               4096b+                8192b+               16384b+
No. of Reads:                76162                 20449                 10948
No. of Writes:                41302                 37488                 14034

   Block Size:              32768b+               65536b+              131072b+
No. of Reads:                 7733                  7306                 31648
No. of Writes:                20849                  6750                 12886

%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         231622      FORGET
      0.00       0.00 us       0.00 us       0.00 us        6123626     RELEASE
      0.00       0.00 us       0.00 us       0.00 us       10869781  RELEASEDIR
      0.00     116.00 us     116.00 us     116.00 us              1     XATTROP
      0.00      40.00 us      38.00 us      43.00 us              6      STATFS
      0.01      40.85 us      33.00 us      97.00 us             13       FSTAT
      0.01      46.26 us      29.00 us     100.00 us             23        STAT
      0.04      73.96 us      53.00 us     150.00 us             48 REMOVEXATTR
      0.04      76.85 us      61.00 us      99.00 us             48     SETATTR
      0.04      35.83 us      28.00 us     155.00 us            103      UNLINK
      0.04      36.28 us      15.00 us     142.00 us            112    FINODELK
      0.05      38.68 us      13.00 us     220.00 us            133       FLUSH
      0.09     324.11 us      28.00 us     589.00 us             28     READDIR
      0.12      85.88 us      36.00 us     215.00 us            144    FXATTROP
      0.12     124.55 us      76.00 us     192.00 us            100       WRITE
      0.13      78.76 us      19.00 us     529.00 us            161    GETXATTR
      0.20      78.65 us      43.00 us     384.00 us            261        OPEN
      0.23      54.09 us       2.00 us     260.00 us            426     OPENDIR
      0.60    1261.23 us     174.00 us   10655.00 us             48      CREATE
      0.66      81.15 us      17.00 us    9254.00 us            819     INODELK
      1.21     279.97 us      23.00 us    1587.00 us            434    READDIRP
      8.10      52.61 us      27.00 us    1283.00 us          15496    READLINK
     15.48      34.54 us      10.00 us   13810.00 us          45133     ENTRYLK
     72.84     104.30 us      14.00 us   14613.00 us          70322      LOOKUP

    Duration: 6492593 seconds
   Data Read: 6308054987 bytes
Data Written: 4579768980 bytes

Interval 1 Stats:
   Block Size:               4096b+                8192b+               16384b+
No. of Reads:                    0                     0                     0
No. of Writes:                    1                    50                     2

   Block Size:              32768b+
No. of Reads:                    0
No. of Writes:                   47
%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             48      FORGET
      0.00       0.00 us       0.00 us       0.00 us            286     RELEASE
      0.00       0.00 us       0.00 us       0.00 us            400  RELEASEDIR
      0.00     116.00 us     116.00 us     116.00 us              1     XATTROP
      0.00      40.40 us      38.00 us      43.00 us              5      STATFS
      0.01      40.85 us      33.00 us      97.00 us             13       FSTAT
      0.01      46.14 us      29.00 us     100.00 us             22        STAT
      0.04      73.96 us      53.00 us     150.00 us             48 REMOVEXATTR
      0.04      76.85 us      61.00 us      99.00 us             48     SETATTR
      0.04      35.83 us      28.00 us     155.00 us            103      UNLINK
      0.05      36.28 us      15.00 us     142.00 us            112    FINODELK
      0.05      39.64 us      13.00 us     220.00 us            117       FLUSH
      0.10     330.28 us      28.00 us     589.00 us             25     READDIR
      0.14      85.88 us      36.00 us     215.00 us            144    FXATTROP
      0.14     124.55 us      76.00 us     192.00 us            100       WRITE
      0.14      79.13 us      19.00 us     529.00 us            159    GETXATTR
      0.22      79.51 us      43.00 us     384.00 us            235        OPEN
      0.25      54.27 us       2.00 us     260.00 us            400     OPENDIR
      0.70    1261.23 us     174.00 us   10655.00 us             48      CREATE
      0.77      81.15 us      17.00 us    9254.00 us            819     INODELK
      1.26     283.31 us      23.00 us    1587.00 us            386    READDIRP
      7.67      52.88 us      27.00 us    1283.00 us          12595    READLINK
     15.47      34.83 us      10.00 us   13810.00 us          38576     ENTRYLK
     72.91     105.05 us      14.00 us   14613.00 us          60295      LOOKUP

    Duration: 68 seconds
   Data Read: 0 bytes
Data Written: 3347998 bytes



Can anybody tell me how to fix the problem with the high load and these cache
files?

Thanks in advance!

Regards
Sebastian

_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
http://www.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160314/3f5bf329/attachment.html>