Benjamin Kingston
2017-May-13 06:46 UTC
[Gluster-users] Fwd: Reliability issues with Gluster 3.10 and shard
Hello all, I'm trying to take advantage of the shard xlator, however I've found it causes a lot of issues that I hope is easily resolvable 1) large file operations work well (copy file from folder a to folder b 2) seek operations and list operations frequently fail (ls directory, read bytes xyz at offset 235567) 3) Another issue is samba shares through samba-vfs show all files as 4MB, I've also seen this when mounting with fuse, however nfs-ganesha reflects correct file sizes always- Turning off the shard feature resolves this issue for new files created in the volume. mounted using the gluster fuse mount here's my volume settings, please let me know if there's some changes I can make. Volume Name: storage2 Type: Distributed-Replicate Volume ID: adaabca5-25ed-4e7f-ae86-2f20fc0143a8 Status: Started Snapshot Count: 0 Number of Bricks: 3 x (2 + 1) = 9 Transport-type: tcp Bricks: Brick1: fd00:0:0:3::6:/mnt/gluster/storage/brick0/glusterfs2 Brick2: fd00:0:0:3::8:/mnt/gluster/storage/brick0/glusterfs2 Brick3: fd00:0:0:3::10:/mnt/gluster/storage/brick0/glusterfs (arbiter) Brick4: fd00:0:0:3::6:/mnt/gluster/storage/brick1/glusterfs2 Brick5: fd00:0:0:3::8:/mnt/gluster/storage/brick1/glusterfs2 Brick6: fd00:0:0:3::10:/mnt/gluster/storage/brick1/glusterfs (arbiter) Brick7: fd00:0:0:3::6:/mnt/gluster/storage/brick2/glusterfs2 Brick8: fd00:0:0:3::8:/mnt/gluster/storage/brick2/glusterfs2 Brick9: fd00:0:0:3::10:/mnt/gluster/storage/brick2/glusterfs (arbiter) Options Reconfigured: features.ctr-enabled: on features.shard-block-size: 4MB network.inode-lru-limit: 90000 features.cache-invalidation: on performance.readdir-ahead: on client.event-threads: 3 performance.cache-ima-xattrs: on cluster.data-self-heal-algorithm: diff network.remote-dio: disable cluster.use-compound-fops: on cluster.read-freq-threshold: 2 cluster.write-freq-threshold: 2 features.record-counters: on disperse.shd-max-threads: 4 performance.parallel-readdir: on performance.client-io-threads: on server.event-threads: 3 cluster.lookup-optimize: on performance.open-behind: on performance.stat-prefetch: on performance.quick-read: off performance.io-cache: on performance.read-ahead: off performance.write-behind: on features.scrub: Active features.bitrot: on features.leases: on features.shard: off transport.address-family: inet6 nfs.disable: on server.allow-insecure: on cluster.shd-max-threads: 8 performance.low-prio-threads: 32 cluster.locking-scheme: granular cluster.shd-wait-qlength: 10000 user.cifs: off cluster.eager-lock: enable cluster.quorum-type: auto cluster.server-quorum-type: server cluster.tier-compact: on storage.linux-aio: on transport.keepalive: on performance.write-behind-window-size: 2GB performance.flush-behind: on performance.cache-size: 1GB cluster.choose-local: on performance.io-thread-count: 64 cluster.brick-multiplex: off cluster.enable-shared-storage: enable nfs-ganesha: enable -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170512/167a150b/attachment.html>
Benjamin Kingston
2017-May-13 19:20 UTC
[Gluster-users] Reliability issues with Gluster 3.10 and shard
Hers's some log entries from nfs-ganesha gfapi [2017-05-13 19:02:54.105936] E [MSGID: 133010] [shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup on shard 11 failed. Base file gfid = 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error] [2017-05-13 19:02:54.106176] E [MSGID: 133010] [shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup on shard 2 failed. Base file gfid = 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error] [2017-05-13 19:02:54.106288] E [MSGID: 133010] [shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup on shard 1 failed. Base file gfid = 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error] [2017-05-13 19:02:54.384922] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-storage2-replicate-2: performing metadata selfheal on fe651475-226e-42a3-be2d-751d4f58e383 [2017-05-13 19:02:54.385894] W [MSGID: 114031] [client-rpc-fops.c:2258:client3_3_setattr_cbk] 0-storage2-client-8: remote operation failed [Operation not permitted] [2017-05-13 19:02:54.401187] I [MSGID: 108026] [afr-self-heal-common.c:1255:afr_log_selfheal] 0-storage2-replicate-2: Completed metadata selfheal on fe651475-226e-42a3-be2d-751d4f58e383. sources=[0] 1 sinks[2017-05-13 19:02:57.830019] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2 (hash=storage2-readdir-ahead-0/cache=<nul>) [2017-05-13 19:08:22.014899] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr.tmp (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr (hash=storage2-readdir-ahead-1/cache=<nul>) [2017-05-13 19:08:22.463840] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04 (hash=storage2-readdir-ahead-0/cache=<nul>) [2017-05-13 19:08:22.769542] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r01.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r01 (hash=storage2-readdir-ahead-0/cache=<nul>) [2017-05-13 19:08:23.141069] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.nfo.tmp (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.nfo (hash=storage2-readdir-ahead-0/cache=<nul>) [2017-05-13 19:08:23.468554] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00.tmp (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 (hash=storage2-readdir-ahead-2/cache=<nul>) [2017-05-13 19:08:23.671753] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.sfv.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.sfv (hash=storage2-readdir-ahead-2/cache=<nul>) [2017-05-13 19:08:23.812152] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11.tmp (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 (hash=storage2-readdir-ahead-1/cache=<nul>) [2017-05-13 19:08:24.244584] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12.tmp (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 (hash=storage2-readdir-ahead-1/cache=<nul>) [2017-05-13 19:08:24.711781] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r09.tmp (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r09 (hash=storage2-readdir-ahead-1/cache=<nul>) [2017-05-13 19:08:25.029102] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r06.tmp (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r06 (hash=storage2-readdir-ahead-0/cache=<nul>) [2017-05-13 19:08:25.206011] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r05.tmp (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r05 (hash=storage2-readdir-ahead-0/cache=<nul>) [2017-05-13 19:08:25.467312] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r03.tmp (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r03 (hash=storage2-readdir-ahead-1/cache=<nul>) [2017-05-13 19:08:25.726208] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r07.tmp (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r07 (hash=storage2-readdir-ahead-0/cache=<nul>) [2017-05-13 19:08:25.860386] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.rar.tmp (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.rar (hash=storage2-readdir-ahead-1/cache=<nul>) [2017-05-13 19:09:00.803859] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r08.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r08 (hash=storage2-readdir-ahead-0/cache=<nul>) [2017-05-13 19:09:01.826525] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r02.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r02 (hash=storage2-readdir-ahead-2/cache=<nul>) [2017-05-13 19:09:02.531354] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Videos/Cinema/Bambi (1942)/.Bambi (1942).mkv.hp9oW8 (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Videos/Cinema/Bambi (1942)/.nfsa223c532d51b3c7700000006 (hash=storage2-readdir-ahead-2/cache=<nul>) [2017-05-13 19:09:39.695354] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r10.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r10 (hash=storage2-readdir-ahead-2/cache=<nul>) [2017-05-13 19:10:59.938286] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 (7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7): trying duplicate remote fd set. [2017-05-13 19:10:59.938510] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 (7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7): trying duplicate remote fd set. [2017-05-13 19:10:59.939004] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 (7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7): trying duplicate remote fd set. [2017-05-13 19:10:59.939246] E [shard.c:426:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/3.10.1/xlator/cluster/distribute.so(+0x65df0) [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xba6e) [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xb29b) [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get trusted.glusterfs.shard.file-size for 7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7 [2017-05-13 19:11:01.884482] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 (2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e): trying duplicate remote fd set. [2017-05-13 19:11:01.884482] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 (2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e): trying duplicate remote fd set. [2017-05-13 19:11:01.884557] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 (2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e): trying duplicate remote fd set. [2017-05-13 19:11:01.884918] E [shard.c:426:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/3.10.1/xlator/cluster/distribute.so(+0x65df0) [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xba6e) [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xb29b) [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get trusted.glusterfs.shard.file-size for 2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e [2017-05-13 19:11:05.136517] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 (b2745d17-1972-4738-afa9-22e9597fa787): trying duplicate remote fd set. [2017-05-13 19:11:05.136518] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 (b2745d17-1972-4738-afa9-22e9597fa787): trying duplicate remote fd set. [2017-05-13 19:11:05.136648] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 (b2745d17-1972-4738-afa9-22e9597fa787): trying duplicate remote fd set. [2017-05-13 19:11:05.136871] E [shard.c:426:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/3.10.1/xlator/cluster/distribute.so(+0x65df0) [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xba6e) [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xb29b) [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get trusted.glusterfs.shard.file-size for b2745d17-1972-4738-afa9-22e9597fa787 -ben On Fri, May 12, 2017 at 11:46 PM, Benjamin Kingston <ben at nexusnebula.net> wrote:> > Hello all, > > I'm trying to take advantage of the shard xlator, however I've found it > causes a lot of issues that I hope is easily resolvable > > 1) large file operations work well (copy file from folder a to folder b > 2) seek operations and list operations frequently fail (ls directory, read > bytes xyz at offset 235567) > 3) Another issue is samba shares through samba-vfs show all files as 4MB, > I've also seen this when mounting with fuse, however nfs-ganesha reflects > correct file sizes always- > > > Turning off the shard feature resolves this issue for new files created in > the volume. mounted using the gluster fuse mount > > here's my volume settings, please let me know if there's some changes I > can make. > > Volume Name: storage2 > Type: Distributed-Replicate > Volume ID: adaabca5-25ed-4e7f-ae86-2f20fc0143a8 > Status: Started > Snapshot Count: 0 > Number of Bricks: 3 x (2 + 1) = 9 > Transport-type: tcp > Bricks: > Brick1: fd00:0:0:3::6:/mnt/gluster/storage/brick0/glusterfs2 > Brick2: fd00:0:0:3::8:/mnt/gluster/storage/brick0/glusterfs2 > Brick3: fd00:0:0:3::10:/mnt/gluster/storage/brick0/glusterfs (arbiter) > Brick4: fd00:0:0:3::6:/mnt/gluster/storage/brick1/glusterfs2 > Brick5: fd00:0:0:3::8:/mnt/gluster/storage/brick1/glusterfs2 > Brick6: fd00:0:0:3::10:/mnt/gluster/storage/brick1/glusterfs (arbiter) > Brick7: fd00:0:0:3::6:/mnt/gluster/storage/brick2/glusterfs2 > Brick8: fd00:0:0:3::8:/mnt/gluster/storage/brick2/glusterfs2 > Brick9: fd00:0:0:3::10:/mnt/gluster/storage/brick2/glusterfs (arbiter) > Options Reconfigured: > features.ctr-enabled: on > features.shard-block-size: 4MB > network.inode-lru-limit: 90000 > features.cache-invalidation: on > performance.readdir-ahead: on > client.event-threads: 3 > performance.cache-ima-xattrs: on > cluster.data-self-heal-algorithm: diff > network.remote-dio: disable > cluster.use-compound-fops: on > cluster.read-freq-threshold: 2 > cluster.write-freq-threshold: 2 > features.record-counters: on > disperse.shd-max-threads: 4 > performance.parallel-readdir: on > performance.client-io-threads: on > server.event-threads: 3 > cluster.lookup-optimize: on > performance.open-behind: on > performance.stat-prefetch: on > performance.quick-read: off > performance.io-cache: on > performance.read-ahead: off > performance.write-behind: on > features.scrub: Active > features.bitrot: on > features.leases: on > features.shard: off > transport.address-family: inet6 > nfs.disable: on > server.allow-insecure: on > cluster.shd-max-threads: 8 > performance.low-prio-threads: 32 > cluster.locking-scheme: granular > cluster.shd-wait-qlength: 10000 > user.cifs: off > cluster.eager-lock: enable > cluster.quorum-type: auto > cluster.server-quorum-type: server > cluster.tier-compact: on > storage.linux-aio: on > transport.keepalive: on > performance.write-behind-window-size: 2GB > performance.flush-behind: on > performance.cache-size: 1GB > cluster.choose-local: on > performance.io-thread-count: 64 > cluster.brick-multiplex: off > cluster.enable-shared-storage: enable > nfs-ganesha: enable >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170513/a5d5e4b4/attachment.html>