I'm running a syncthing instance on a raspi 5 which is part of a gluster cluster ((*) details see below). Without the running syncthing service I'm absolutely happy with the glusterfs performance (100+MB/sec), but when I start the syncthing service, performance drops after some time to nearly standstill (1MB/sec and less). # ls -alh SOME_DIR takes 10-15 sec (or even longer) # cd INTO_SOME_DIR takes 10-15sec I see some glusterfs daemon activity (30-40 %), but nearly no network- or disk activity (2+GB free RAM), that means to me, syncthing is not consuming the resources (but possibly blocking some resources). I read https://docs.gluster.org/en/main/Administrator-Guide/Performance-Testing/ and tried: ??? gluster volume profile wdVolume start ??? setfattr -n trusted.io-stats-dump -v io-stats-pre.txt /mnt/wdVolume results (**) but I don't know what to do with the numbers. The cluster is healthy, see (***). I can't see any serious in glusterd.log, see (****) A few seconds after I stopped the syncthing service, the performance is back at 100+MB/sec. I have no idea, where to start my search for the performance, any ideas are welcome. Thank in advance Thomas ################################################################################### (*) root at tpi5wb3:/mnt/wdVolume# gluster volume info wdVolume Volume Name: wdVolume Type: Distributed-Disperse Volume ID: 5b47f69a-7731-4c7b-85bf-a5014e2a5209 Status: Started Snapshot Count: 0 Number of Bricks: 2 x (2 + 1) = 6 Transport-type: tcp Bricks: Brick1: tPi5Wb:/mnt/glusterBricks/extWd18a/data Brick2: 192.168.129.9:/mnt/glusterBricks/extWd18b/data Brick3: tPi5Wb3:/mnt/glusterBricks/extWd18c/data Brick4: tPi5Wb:/mnt/glusterBricks/extWd5a/data Brick5: 192.168.129.9:/mnt/glusterBricks/extWd5b/data Brick6: tPi5Wb3:/mnt/glusterBricks/extWd5x/data Options Reconfigured: storage.build-pgfid: on diagnostics.client-log-level: DEBUG cluster.disperse-self-heal-daemon: enable transport.address-family: inet storage.fips-mode-rchecksum: on features.bitrot: on features.scrub: Active performance.cache-size: 64MB disperse.shd-max-threads: 16 features.cache-invalidation: on features.cache-invalidation-timeout: 600 performance.stat-prefetch: on performance.cache-invalidation: on performance.md-cache-timeout: 600 network.inode-lru-limit: 200000 performance.readdir-ahead: on performance.parallel-readdir: on performance.nl-cache: on performance.nl-cache-timeout: 600 performance.nl-cache-positive-entry: on ########################################################################### (**) === Interval 69 stats == ????? Duration : 103 secs ???? BytesRead : 117440512 ? BytesWritten : 0 Block Size?? :????????? 131072B+ Read Count?? :?????????????? 896 Write Count? :???????????????? 0 Fop?????????? Call Count??? Avg-Latency??? Min-Latency Max-Latency ---?????????? ----------??? -----------??? ----------- ----------- RELEASEDIR??????????? 21?????????? 0 us?????????? 0 us?????????? 0 us ------ ----- ----- ----- ----- ----- ----- -----? ----- ----- ----- ----- Sa 3. Mai 12:21:40 CEST 2025 === Cumulative stats == ????? Duration : 1027061 secs ???? BytesRead : 3263114620730 ? BytesWritten : 4524375262 Block Size?? :?????????????? 1B+?????????????? 2B+ 4B+ Read Count?? :???????????????? 0???????????????? 0 0 Write Count? :?????????????? 317???????????????? 5 12 Block Size?? :?????????????? 8B+????????????? 16B+ 32B+ Read Count?? :???????????????? 0???????????????? 0 4 Write Count? :??????????????? 25?????????????? 354 384 Block Size?? :????????????? 64B+???????????? 128B+ 256B+ Read Count?? :???????????????? 4??????????????? 12 3 Write Count? :?????????????? 452????????????? 3174 177 Block Size?? :???????????? 512B+??????????? 1024B+ 2048B+ Read Count?? :???????????????? 5??????????????? 67 20 Write Count? :?????????????? 819?????????????? 644 517 Block Size?? :??????????? 4096B+??????????? 8192B+ 16384B+ Read Count?? :??????????????? 94?????????????? 159 238 Write Count? :?????????????? 657??????????????? 15 14 Block Size?? :?????????? 32768B+?????????? 65536B+ 131072B+ Read Count?? :?????????????? 487????????????? 1542 24894452 Write Count? :??????????????? 20??????????????? 62 34400 Fop?????????? Call Count??? Avg-Latency??? Min-Latency Max-Latency ---?????????? ----------??? -----------??? ----------- ----------- STAT???????????????? 855 1636322233.15 us???? 129481.00 us 7613060937.00 us READ?????????????? 22774? 867794690.83 us??? 7150668.00 us 5464263390.00 us FLUSH????????????????? 1 1019668889.00 us 1019668889.00 us 1019668889.00 us SETXATTR????????????? 10 4586558316.50 us 3273936240.00 us 6897317026.00 us OPENDIR????????????? 459 3863731511.82 us? 565874532.00 us 7436044801.00 us FSTAT????????????????? 1???? 130574.00 us???? 130574.00 us 130574.00 us LOOKUP???????????? 20464? 793541909.91 us????? 26574.00 us 4389861051.00 us SETATTR????????????? 124 4928152166.21 us 1786135647.00 us 8324248584.00 us READDIRP??????????? 1905? 315443817.07 us????? 44445.00 us 2690928799.00 us FORGET????????????? 1000???????????? 0 us???????????? 0 us???????????? 0 us RELEASE???????????? 1930???????????? 0 us???????????? 0 us???????????? 0 us RELEASEDIR??????? 745442???????????? 0 us???????????? 0 us???????????? 0 us ------ ----- ----- ----- ----- ----- ----- -----? ----- ----- ----- ----- Current open fd's: 4 Max open fd's: 21 time 2025-04-21 13:29:31.114171 +0000 ... ######################################################################################## (***) # gluster volume heal wdVolume info Brick tPi5Wb:/mnt/glusterBricks/extWd18a/data Status: Connected Number of entries: 0 Brick 192.168.129.9:/mnt/glusterBricks/extWd18b/data Status: Connected Number of entries: 0 Brick tPi5Wb3:/mnt/glusterBricks/extWd18c/data Status: Connected Number of entries: 0 Brick tPi5Wb:/mnt/glusterBricks/extWd5a/data Status: Connected Number of entries: 0 Brick 192.168.129.9:/mnt/glusterBricks/extWd5b/data Status: Connected Number of entries: 0 Brick tPi5Wb3:/mnt/glusterBricks/extWd5x/data Status: Connected Number of entries: 0 ######################################################################################## (****) root at tpi5wb3:/var/log/glusterfs# cat glusterd.log | grep ' E ' [2025-05-03 23:23:04.415227 +0000] E [MSGID: 106061] [glusterd-utils.c:9944:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd: failed to get index from rsp dict [2025-05-04 00:23:06.167938 +0000] E [MSGID: 106061] [glusterd-utils.c:9944:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd: failed to get index from rsp dict [2025-05-04 01:23:06.427137 +0000] E [MSGID: 106061] [glusterd-utils.c:9944:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd: failed to get index from rsp dict [2025-05-04 02:23:07.828266 +0000] E [MSGID: 106061] [glusterd-utils.c:9944:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd: failed to get index from rsp dict [2025-05-04 03:23:09.458889 +0000] E [MSGID: 106061] [glusterd-utils.c:9944:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd: failed to get index from rsp dict [2025-05-04 04:23:07.200384 +0000] E [MSGID: 106061] [glusterd-utils.c:9944:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd: failed to get index from rsp dict [2025-05-04 05:23:12.085744 +0000] E [MSGID: 106061] [glusterd-utils.c:9944:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd: failed to get index from rsp dict [2025-05-04 06:23:08.550034 +0000] E [MSGID: 106061] [glusterd-utils.c:9944:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd: failed to get index from rsp dict [2025-05-04 07:23:10.401254 +0000] E [MSGID: 106061] [glusterd-utils.c:9944:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd: failed to get index from rsp dict root at tpi5wb3:/var/log/glusterfs# cat glusterd.log | grep ' W ' [2025-05-04 02:29:24.540484 +0000] W [glusterd-locks.c:729:glusterd_mgmt_v3_unlock] (-->/lib/aarch64-linux-gnu/libgfrpc.so.0(+0xd9e0) [0xffffbb65d9e0] -->/usr/lib/aarch64-linux-gnu/glusterfs/11.1/xlator/mgmt/glusterd.so(+0x2f6f0) [0xffffb653f6f0] -->/usr/lib/aarch64-linux-gnu/glusterfs/11.1/xlator/mgmt/glusterd.so(+0xc0dc8) [0xffffb65d0dc8] ) 0-management: Lock for vol wdVolume not held [2025-05-04 02:29:24.540534 +0000] W [MSGID: 106117] [glusterd-handler.c:6680:__glusterd_peer_rpc_notify] 0-management: Lock not released for wdVolume root at tpi5wb3:/var/log/glusterfs# ##########################################################################################