Benjamin Kingston
2017-May-13 19:20 UTC
[Gluster-users] Reliability issues with Gluster 3.10 and shard
Hers's some log entries from nfs-ganesha gfapi [2017-05-13 19:02:54.105936] E [MSGID: 133010] [shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup on shard 11 failed. Base file gfid = 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error] [2017-05-13 19:02:54.106176] E [MSGID: 133010] [shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup on shard 2 failed. Base file gfid = 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error] [2017-05-13 19:02:54.106288] E [MSGID: 133010] [shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup on shard 1 failed. Base file gfid = 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error] [2017-05-13 19:02:54.384922] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-storage2-replicate-2: performing metadata selfheal on fe651475-226e-42a3-be2d-751d4f58e383 [2017-05-13 19:02:54.385894] W [MSGID: 114031] [client-rpc-fops.c:2258:client3_3_setattr_cbk] 0-storage2-client-8: remote operation failed [Operation not permitted] [2017-05-13 19:02:54.401187] I [MSGID: 108026] [afr-self-heal-common.c:1255:afr_log_selfheal] 0-storage2-replicate-2: Completed metadata selfheal on fe651475-226e-42a3-be2d-751d4f58e383. sources=[0] 1 sinks[2017-05-13 19:02:57.830019] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2 (hash=storage2-readdir-ahead-0/cache=<nul>) [2017-05-13 19:08:22.014899] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr.tmp (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr (hash=storage2-readdir-ahead-1/cache=<nul>) [2017-05-13 19:08:22.463840] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04 (hash=storage2-readdir-ahead-0/cache=<nul>) [2017-05-13 19:08:22.769542] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r01.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r01 (hash=storage2-readdir-ahead-0/cache=<nul>) [2017-05-13 19:08:23.141069] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.nfo.tmp (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.nfo (hash=storage2-readdir-ahead-0/cache=<nul>) [2017-05-13 19:08:23.468554] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00.tmp (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 (hash=storage2-readdir-ahead-2/cache=<nul>) [2017-05-13 19:08:23.671753] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.sfv.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.sfv (hash=storage2-readdir-ahead-2/cache=<nul>) [2017-05-13 19:08:23.812152] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11.tmp (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 (hash=storage2-readdir-ahead-1/cache=<nul>) [2017-05-13 19:08:24.244584] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12.tmp (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 (hash=storage2-readdir-ahead-1/cache=<nul>) [2017-05-13 19:08:24.711781] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r09.tmp (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r09 (hash=storage2-readdir-ahead-1/cache=<nul>) [2017-05-13 19:08:25.029102] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r06.tmp (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r06 (hash=storage2-readdir-ahead-0/cache=<nul>) [2017-05-13 19:08:25.206011] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r05.tmp (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r05 (hash=storage2-readdir-ahead-0/cache=<nul>) [2017-05-13 19:08:25.467312] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r03.tmp (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r03 (hash=storage2-readdir-ahead-1/cache=<nul>) [2017-05-13 19:08:25.726208] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r07.tmp (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r07 (hash=storage2-readdir-ahead-0/cache=<nul>) [2017-05-13 19:08:25.860386] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.rar.tmp (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.rar (hash=storage2-readdir-ahead-1/cache=<nul>) [2017-05-13 19:09:00.803859] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r08.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r08 (hash=storage2-readdir-ahead-0/cache=<nul>) [2017-05-13 19:09:01.826525] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r02.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r02 (hash=storage2-readdir-ahead-2/cache=<nul>) [2017-05-13 19:09:02.531354] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Videos/Cinema/Bambi (1942)/.Bambi (1942).mkv.hp9oW8 (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Videos/Cinema/Bambi (1942)/.nfsa223c532d51b3c7700000006 (hash=storage2-readdir-ahead-2/cache=<nul>) [2017-05-13 19:09:39.695354] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r10.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r10 (hash=storage2-readdir-ahead-2/cache=<nul>) [2017-05-13 19:10:59.938286] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 (7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7): trying duplicate remote fd set. [2017-05-13 19:10:59.938510] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 (7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7): trying duplicate remote fd set. [2017-05-13 19:10:59.939004] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 (7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7): trying duplicate remote fd set. [2017-05-13 19:10:59.939246] E [shard.c:426:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/3.10.1/xlator/cluster/distribute.so(+0x65df0) [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xba6e) [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xb29b) [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get trusted.glusterfs.shard.file-size for 7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7 [2017-05-13 19:11:01.884482] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 (2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e): trying duplicate remote fd set. [2017-05-13 19:11:01.884482] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 (2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e): trying duplicate remote fd set. [2017-05-13 19:11:01.884557] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 (2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e): trying duplicate remote fd set. [2017-05-13 19:11:01.884918] E [shard.c:426:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/3.10.1/xlator/cluster/distribute.so(+0x65df0) [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xba6e) [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xb29b) [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get trusted.glusterfs.shard.file-size for 2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e [2017-05-13 19:11:05.136517] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 (b2745d17-1972-4738-afa9-22e9597fa787): trying duplicate remote fd set. [2017-05-13 19:11:05.136518] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 (b2745d17-1972-4738-afa9-22e9597fa787): trying duplicate remote fd set. [2017-05-13 19:11:05.136648] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 (b2745d17-1972-4738-afa9-22e9597fa787): trying duplicate remote fd set. [2017-05-13 19:11:05.136871] E [shard.c:426:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/3.10.1/xlator/cluster/distribute.so(+0x65df0) [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xba6e) [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xb29b) [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get trusted.glusterfs.shard.file-size for b2745d17-1972-4738-afa9-22e9597fa787 -ben On Fri, May 12, 2017 at 11:46 PM, Benjamin Kingston <ben at nexusnebula.net> wrote:> > Hello all, > > I'm trying to take advantage of the shard xlator, however I've found it > causes a lot of issues that I hope is easily resolvable > > 1) large file operations work well (copy file from folder a to folder b > 2) seek operations and list operations frequently fail (ls directory, read > bytes xyz at offset 235567) > 3) Another issue is samba shares through samba-vfs show all files as 4MB, > I've also seen this when mounting with fuse, however nfs-ganesha reflects > correct file sizes always- > > > Turning off the shard feature resolves this issue for new files created in > the volume. mounted using the gluster fuse mount > > here's my volume settings, please let me know if there's some changes I > can make. > > Volume Name: storage2 > Type: Distributed-Replicate > Volume ID: adaabca5-25ed-4e7f-ae86-2f20fc0143a8 > Status: Started > Snapshot Count: 0 > Number of Bricks: 3 x (2 + 1) = 9 > Transport-type: tcp > Bricks: > Brick1: fd00:0:0:3::6:/mnt/gluster/storage/brick0/glusterfs2 > Brick2: fd00:0:0:3::8:/mnt/gluster/storage/brick0/glusterfs2 > Brick3: fd00:0:0:3::10:/mnt/gluster/storage/brick0/glusterfs (arbiter) > Brick4: fd00:0:0:3::6:/mnt/gluster/storage/brick1/glusterfs2 > Brick5: fd00:0:0:3::8:/mnt/gluster/storage/brick1/glusterfs2 > Brick6: fd00:0:0:3::10:/mnt/gluster/storage/brick1/glusterfs (arbiter) > Brick7: fd00:0:0:3::6:/mnt/gluster/storage/brick2/glusterfs2 > Brick8: fd00:0:0:3::8:/mnt/gluster/storage/brick2/glusterfs2 > Brick9: fd00:0:0:3::10:/mnt/gluster/storage/brick2/glusterfs (arbiter) > Options Reconfigured: > features.ctr-enabled: on > features.shard-block-size: 4MB > network.inode-lru-limit: 90000 > features.cache-invalidation: on > performance.readdir-ahead: on > client.event-threads: 3 > performance.cache-ima-xattrs: on > cluster.data-self-heal-algorithm: diff > network.remote-dio: disable > cluster.use-compound-fops: on > cluster.read-freq-threshold: 2 > cluster.write-freq-threshold: 2 > features.record-counters: on > disperse.shd-max-threads: 4 > performance.parallel-readdir: on > performance.client-io-threads: on > server.event-threads: 3 > cluster.lookup-optimize: on > performance.open-behind: on > performance.stat-prefetch: on > performance.quick-read: off > performance.io-cache: on > performance.read-ahead: off > performance.write-behind: on > features.scrub: Active > features.bitrot: on > features.leases: on > features.shard: off > transport.address-family: inet6 > nfs.disable: on > server.allow-insecure: on > cluster.shd-max-threads: 8 > performance.low-prio-threads: 32 > cluster.locking-scheme: granular > cluster.shd-wait-qlength: 10000 > user.cifs: off > cluster.eager-lock: enable > cluster.quorum-type: auto > cluster.server-quorum-type: server > cluster.tier-compact: on > storage.linux-aio: on > transport.keepalive: on > performance.write-behind-window-size: 2GB > performance.flush-behind: on > performance.cache-size: 1GB > cluster.choose-local: on > performance.io-thread-count: 64 > cluster.brick-multiplex: off > cluster.enable-shared-storage: enable > nfs-ganesha: enable >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170513/a5d5e4b4/attachment.html>
Benjamin Kingston
2017-May-15 05:31 UTC
[Gluster-users] Reliability issues with Gluster 3.10 and shard
I resolved this with the following settings, particularly disabling features.ctr-enabled olume Name: storage2 Type: Distributed-Replicate Volume ID: adaabca5-25ed-4e7f-ae86-2f20fc0143a8 Status: Started Snapshot Count: 0 Number of Bricks: 3 x (2 + 1) = 9 Transport-type: tcp Bricks: Brick1: fd00:0:0:3::6:/mnt/gluster/storage/brick0/glusterfs2 Brick2: fd00:0:0:3::8:/mnt/gluster/storage/brick0/glusterfs2 Brick3: fd00:0:0:3::10:/mnt/gluster/storage/brick0/glusterfs (arbiter) Brick4: fd00:0:0:3::6:/mnt/gluster/storage/brick1/glusterfs2 Brick5: fd00:0:0:3::8:/mnt/gluster/storage/brick1/glusterfs2 Brick6: fd00:0:0:3::10:/mnt/gluster/storage/brick1/glusterfs (arbiter) Brick7: fd00:0:0:3::6:/mnt/gluster/storage/brick2/glusterfs2 Brick8: fd00:0:0:3::8:/mnt/gluster/storage/brick2/glusterfs2 Brick9: fd00:0:0:3::10:/mnt/gluster/storage/brick2/glusterfs (arbiter) Options Reconfigured: performance.write-behind-window-size: 4MB performance.cache-invalidation: on transport.keepalive: on performance.write-behind: on performance.read-ahead: on performance.io-cache: on performance.stat-prefetch: on performance.open-behind: on cluster.use-compound-fops: on performance.cache-ima-xattrs: on features.cache-invalidation: on client.event-threads: 4 cluster.data-self-heal-algorithm: full performance.client-io-threads: on server.event-threads: 4 performance.quick-read: on features.scrub: Active features.bitrot: on features.shard: on transport.address-family: inet6 nfs.disable: on server.allow-insecure: on user.cifs: off cluster.quorum-type: auto cluster.server-quorum-type: server cluster.tier-compact: on diagnostics.brick-log-level: WARNING diagnostics.client-log-level: WARNING cluster.self-heal-daemon: enable performance.cache-samba-metadata: on cluster.brick-multiplex: off cluster.enable-shared-storage: enable nfs-ganesha: enable -ben On Sat, May 13, 2017 at 12:20 PM, Benjamin Kingston <ben at nexusnebula.net> wrote:> Hers's some log entries from nfs-ganesha gfapi > > [2017-05-13 19:02:54.105936] E [MSGID: 133010] [shard.c:1706:shard_common_lookup_shards_cbk] > 0-storage2-shard: Lookup on shard 11 failed. Base file gfid > 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error] > [2017-05-13 19:02:54.106176] E [MSGID: 133010] [shard.c:1706:shard_common_lookup_shards_cbk] > 0-storage2-shard: Lookup on shard 2 failed. Base file gfid > 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error] > [2017-05-13 19:02:54.106288] E [MSGID: 133010] [shard.c:1706:shard_common_lookup_shards_cbk] > 0-storage2-shard: Lookup on shard 1 failed. Base file gfid > 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error] > [2017-05-13 19:02:54.384922] I [MSGID: 108026] > [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] > 0-storage2-replicate-2: performing metadata selfheal on > fe651475-226e-42a3-be2d-751d4f58e383 > [2017-05-13 19:02:54.385894] W [MSGID: 114031] [client-rpc-fops.c:2258:client3_3_setattr_cbk] > 0-storage2-client-8: remote operation failed [Operation not permitted] > [2017-05-13 19:02:54.401187] I [MSGID: 108026] > [afr-self-heal-common.c:1255:afr_log_selfheal] 0-storage2-replicate-2: > Completed metadata selfheal on fe651475-226e-42a3-be2d-751d4f58e383. > sources=[0] 1 sinks> [2017-05-13 19:02:57.830019] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2.tmp > (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2 > (hash=storage2-readdir-ahead-0/cache=<nul>) > > [2017-05-13 19:08:22.014899] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr.tmp > (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr > (hash=storage2-readdir-ahead-1/cache=<nul>) > [2017-05-13 19:08:22.463840] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04.tmp > (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04 > (hash=storage2-readdir-ahead-0/cache=<nul>) > [2017-05-13 19:08:22.769542] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r01.tmp > (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r01 > (hash=storage2-readdir-ahead-0/cache=<nul>) > [2017-05-13 19:08:23.141069] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.nfo.tmp > (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.nfo > (hash=storage2-readdir-ahead-0/cache=<nul>) > [2017-05-13 19:08:23.468554] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00.tmp > (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 > (hash=storage2-readdir-ahead-2/cache=<nul>) > [2017-05-13 19:08:23.671753] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.sfv.tmp > (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.sfv > (hash=storage2-readdir-ahead-2/cache=<nul>) > [2017-05-13 19:08:23.812152] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11.tmp > (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 > (hash=storage2-readdir-ahead-1/cache=<nul>) > [2017-05-13 19:08:24.244584] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12.tmp > (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 > (hash=storage2-readdir-ahead-1/cache=<nul>) > [2017-05-13 19:08:24.711781] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r09.tmp > (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r09 > (hash=storage2-readdir-ahead-1/cache=<nul>) > [2017-05-13 19:08:25.029102] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r06.tmp > (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r06 > (hash=storage2-readdir-ahead-0/cache=<nul>) > [2017-05-13 19:08:25.206011] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r05.tmp > (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r05 > (hash=storage2-readdir-ahead-0/cache=<nul>) > [2017-05-13 19:08:25.467312] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r03.tmp > (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r03 > (hash=storage2-readdir-ahead-1/cache=<nul>) > [2017-05-13 19:08:25.726208] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r07.tmp > (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r07 > (hash=storage2-readdir-ahead-0/cache=<nul>) > [2017-05-13 19:08:25.860386] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.rar.tmp > (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.rar > (hash=storage2-readdir-ahead-1/cache=<nul>) > [2017-05-13 19:09:00.803859] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r08.tmp > (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r08 > (hash=storage2-readdir-ahead-0/cache=<nul>) > [2017-05-13 19:09:01.826525] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r02.tmp > (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r02 > (hash=storage2-readdir-ahead-2/cache=<nul>) > [2017-05-13 19:09:02.531354] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Videos/Cinema/Bambi (1942)/.Bambi (1942).mkv.hp9oW8 > (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => > /content/Videos/Cinema/Bambi (1942)/.nfsa223c532d51b3c7700000006 > (hash=storage2-readdir-ahead-2/cache=<nul>) > [2017-05-13 19:09:39.695354] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r10.tmp > (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r10 > (hash=storage2-readdir-ahead-2/cache=<nul>) > [2017-05-13 19:10:59.938286] I [MSGID: 114024] > [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0: > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 > (7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7): trying duplicate remote fd set. > [2017-05-13 19:10:59.938510] I [MSGID: 114024] > [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6: > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 > (7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7): trying duplicate remote fd set. > [2017-05-13 19:10:59.939004] I [MSGID: 114024] > [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1: > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 > (7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7): trying duplicate remote fd set. > [2017-05-13 19:10:59.939246] E [shard.c:426:shard_modify_size_and_block_count] > (-->/usr/lib64/glusterfs/3.10.1/xlator/cluster/distribute.so(+0x65df0) > [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xba6e) > [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xb29b) > [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get > trusted.glusterfs.shard.file-size for 7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7 > [2017-05-13 19:11:01.884482] I [MSGID: 114024] > [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1: > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 > (2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e): trying duplicate remote fd set. > [2017-05-13 19:11:01.884482] I [MSGID: 114024] > [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0: > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 > (2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e): trying duplicate remote fd set. > [2017-05-13 19:11:01.884557] I [MSGID: 114024] > [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6: > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 > (2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e): trying duplicate remote fd set. > [2017-05-13 19:11:01.884918] E [shard.c:426:shard_modify_size_and_block_count] > (-->/usr/lib64/glusterfs/3.10.1/xlator/cluster/distribute.so(+0x65df0) > [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xba6e) > [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xb29b) > [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get > trusted.glusterfs.shard.file-size for 2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e > [2017-05-13 19:11:05.136517] I [MSGID: 114024] > [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0: > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 > (b2745d17-1972-4738-afa9-22e9597fa787): trying duplicate remote fd set. > [2017-05-13 19:11:05.136518] I [MSGID: 114024] > [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1: > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 > (b2745d17-1972-4738-afa9-22e9597fa787): trying duplicate remote fd set. > [2017-05-13 19:11:05.136648] I [MSGID: 114024] > [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6: > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 > (b2745d17-1972-4738-afa9-22e9597fa787): trying duplicate remote fd set. > [2017-05-13 19:11:05.136871] E [shard.c:426:shard_modify_size_and_block_count] > (-->/usr/lib64/glusterfs/3.10.1/xlator/cluster/distribute.so(+0x65df0) > [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xba6e) > [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xb29b) > [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get > trusted.glusterfs.shard.file-size for b2745d17-1972-4738-afa9-22e9597fa787 > > > -ben > > On Fri, May 12, 2017 at 11:46 PM, Benjamin Kingston <ben at nexusnebula.net> > wrote: > >> >> Hello all, >> >> I'm trying to take advantage of the shard xlator, however I've found it >> causes a lot of issues that I hope is easily resolvable >> >> 1) large file operations work well (copy file from folder a to folder b >> 2) seek operations and list operations frequently fail (ls directory, >> read bytes xyz at offset 235567) >> 3) Another issue is samba shares through samba-vfs show all files as 4MB, >> I've also seen this when mounting with fuse, however nfs-ganesha reflects >> correct file sizes always- >> >> >> Turning off the shard feature resolves this issue for new files created >> in the volume. mounted using the gluster fuse mount >> >> here's my volume settings, please let me know if there's some changes I >> can make. >> >> Volume Name: storage2 >> Type: Distributed-Replicate >> Volume ID: adaabca5-25ed-4e7f-ae86-2f20fc0143a8 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 3 x (2 + 1) = 9 >> Transport-type: tcp >> Bricks: >> Brick1: fd00:0:0:3::6:/mnt/gluster/storage/brick0/glusterfs2 >> Brick2: fd00:0:0:3::8:/mnt/gluster/storage/brick0/glusterfs2 >> Brick3: fd00:0:0:3::10:/mnt/gluster/storage/brick0/glusterfs (arbiter) >> Brick4: fd00:0:0:3::6:/mnt/gluster/storage/brick1/glusterfs2 >> Brick5: fd00:0:0:3::8:/mnt/gluster/storage/brick1/glusterfs2 >> Brick6: fd00:0:0:3::10:/mnt/gluster/storage/brick1/glusterfs (arbiter) >> Brick7: fd00:0:0:3::6:/mnt/gluster/storage/brick2/glusterfs2 >> Brick8: fd00:0:0:3::8:/mnt/gluster/storage/brick2/glusterfs2 >> Brick9: fd00:0:0:3::10:/mnt/gluster/storage/brick2/glusterfs (arbiter) >> Options Reconfigured: >> features.ctr-enabled: on >> features.shard-block-size: 4MB >> network.inode-lru-limit: 90000 >> features.cache-invalidation: on >> performance.readdir-ahead: on >> client.event-threads: 3 >> performance.cache-ima-xattrs: on >> cluster.data-self-heal-algorithm: diff >> network.remote-dio: disable >> cluster.use-compound-fops: on >> cluster.read-freq-threshold: 2 >> cluster.write-freq-threshold: 2 >> features.record-counters: on >> disperse.shd-max-threads: 4 >> performance.parallel-readdir: on >> performance.client-io-threads: on >> server.event-threads: 3 >> cluster.lookup-optimize: on >> performance.open-behind: on >> performance.stat-prefetch: on >> performance.quick-read: off >> performance.io-cache: on >> performance.read-ahead: off >> performance.write-behind: on >> features.scrub: Active >> features.bitrot: on >> features.leases: on >> features.shard: off >> transport.address-family: inet6 >> nfs.disable: on >> server.allow-insecure: on >> cluster.shd-max-threads: 8 >> performance.low-prio-threads: 32 >> cluster.locking-scheme: granular >> cluster.shd-wait-qlength: 10000 >> user.cifs: off >> cluster.eager-lock: enable >> cluster.quorum-type: auto >> cluster.server-quorum-type: server >> cluster.tier-compact: on >> storage.linux-aio: on >> transport.keepalive: on >> performance.write-behind-window-size: 2GB >> performance.flush-behind: on >> performance.cache-size: 1GB >> cluster.choose-local: on >> performance.io-thread-count: 64 >> cluster.brick-multiplex: off >> cluster.enable-shared-storage: enable >> nfs-ganesha: enable >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170514/97e08aa4/attachment.html>
Krutika Dhananjay
2017-May-15 06:29 UTC
[Gluster-users] Reliability issues with Gluster 3.10 and shard
Shard translator is currently supported only for VM image store workload. -Krutika On Sun, May 14, 2017 at 12:50 AM, Benjamin Kingston <ben at nexusnebula.net> wrote:> Hers's some log entries from nfs-ganesha gfapi > > [2017-05-13 19:02:54.105936] E [MSGID: 133010] [shard.c:1706:shard_common_lookup_shards_cbk] > 0-storage2-shard: Lookup on shard 11 failed. Base file gfid > 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error] > [2017-05-13 19:02:54.106176] E [MSGID: 133010] [shard.c:1706:shard_common_lookup_shards_cbk] > 0-storage2-shard: Lookup on shard 2 failed. Base file gfid > 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error] > [2017-05-13 19:02:54.106288] E [MSGID: 133010] [shard.c:1706:shard_common_lookup_shards_cbk] > 0-storage2-shard: Lookup on shard 1 failed. Base file gfid > 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error] > [2017-05-13 19:02:54.384922] I [MSGID: 108026] > [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] > 0-storage2-replicate-2: performing metadata selfheal on > fe651475-226e-42a3-be2d-751d4f58e383 > [2017-05-13 19:02:54.385894] W [MSGID: 114031] [client-rpc-fops.c:2258:client3_3_setattr_cbk] > 0-storage2-client-8: remote operation failed [Operation not permitted] > [2017-05-13 19:02:54.401187] I [MSGID: 108026] > [afr-self-heal-common.c:1255:afr_log_selfheal] 0-storage2-replicate-2: > Completed metadata selfheal on fe651475-226e-42a3-be2d-751d4f58e383. > sources=[0] 1 sinks> [2017-05-13 19:02:57.830019] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2.tmp > (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2 > (hash=storage2-readdir-ahead-0/cache=<nul>) > > [2017-05-13 19:08:22.014899] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr.tmp > (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr > (hash=storage2-readdir-ahead-1/cache=<nul>) > [2017-05-13 19:08:22.463840] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04.tmp > (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04 > (hash=storage2-readdir-ahead-0/cache=<nul>) > [2017-05-13 19:08:22.769542] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r01.tmp > (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r01 > (hash=storage2-readdir-ahead-0/cache=<nul>) > [2017-05-13 19:08:23.141069] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.nfo.tmp > (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.nfo > (hash=storage2-readdir-ahead-0/cache=<nul>) > [2017-05-13 19:08:23.468554] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00.tmp > (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 > (hash=storage2-readdir-ahead-2/cache=<nul>) > [2017-05-13 19:08:23.671753] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.sfv.tmp > (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.sfv > (hash=storage2-readdir-ahead-2/cache=<nul>) > [2017-05-13 19:08:23.812152] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11.tmp > (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 > (hash=storage2-readdir-ahead-1/cache=<nul>) > [2017-05-13 19:08:24.244584] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12.tmp > (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 > (hash=storage2-readdir-ahead-1/cache=<nul>) > [2017-05-13 19:08:24.711781] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r09.tmp > (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r09 > (hash=storage2-readdir-ahead-1/cache=<nul>) > [2017-05-13 19:08:25.029102] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r06.tmp > (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r06 > (hash=storage2-readdir-ahead-0/cache=<nul>) > [2017-05-13 19:08:25.206011] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r05.tmp > (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r05 > (hash=storage2-readdir-ahead-0/cache=<nul>) > [2017-05-13 19:08:25.467312] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r03.tmp > (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r03 > (hash=storage2-readdir-ahead-1/cache=<nul>) > [2017-05-13 19:08:25.726208] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r07.tmp > (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r07 > (hash=storage2-readdir-ahead-0/cache=<nul>) > [2017-05-13 19:08:25.860386] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.rar.tmp > (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.rar > (hash=storage2-readdir-ahead-1/cache=<nul>) > [2017-05-13 19:09:00.803859] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r08.tmp > (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r08 > (hash=storage2-readdir-ahead-0/cache=<nul>) > [2017-05-13 19:09:01.826525] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r02.tmp > (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r02 > (hash=storage2-readdir-ahead-2/cache=<nul>) > [2017-05-13 19:09:02.531354] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Videos/Cinema/Bambi (1942)/.Bambi (1942).mkv.hp9oW8 > (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => > /content/Videos/Cinema/Bambi (1942)/.nfsa223c532d51b3c7700000006 > (hash=storage2-readdir-ahead-2/cache=<nul>) > [2017-05-13 19:09:39.695354] I [MSGID: 109066] > [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r10.tmp > (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r10 > (hash=storage2-readdir-ahead-2/cache=<nul>) > [2017-05-13 19:10:59.938286] I [MSGID: 114024] > [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0: > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 > (7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7): trying duplicate remote fd set. > [2017-05-13 19:10:59.938510] I [MSGID: 114024] > [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6: > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 > (7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7): trying duplicate remote fd set. > [2017-05-13 19:10:59.939004] I [MSGID: 114024] > [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1: > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 > (7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7): trying duplicate remote fd set. > [2017-05-13 19:10:59.939246] E [shard.c:426:shard_modify_size_and_block_count] > (-->/usr/lib64/glusterfs/3.10.1/xlator/cluster/distribute.so(+0x65df0) > [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xba6e) > [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xb29b) > [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get > trusted.glusterfs.shard.file-size for 7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7 > [2017-05-13 19:11:01.884482] I [MSGID: 114024] > [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1: > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 > (2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e): trying duplicate remote fd set. > [2017-05-13 19:11:01.884482] I [MSGID: 114024] > [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0: > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 > (2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e): trying duplicate remote fd set. > [2017-05-13 19:11:01.884557] I [MSGID: 114024] > [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6: > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 > (2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e): trying duplicate remote fd set. > [2017-05-13 19:11:01.884918] E [shard.c:426:shard_modify_size_and_block_count] > (-->/usr/lib64/glusterfs/3.10.1/xlator/cluster/distribute.so(+0x65df0) > [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xba6e) > [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xb29b) > [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get > trusted.glusterfs.shard.file-size for 2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e > [2017-05-13 19:11:05.136517] I [MSGID: 114024] > [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0: > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 > (b2745d17-1972-4738-afa9-22e9597fa787): trying duplicate remote fd set. > [2017-05-13 19:11:05.136518] I [MSGID: 114024] > [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1: > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 > (b2745d17-1972-4738-afa9-22e9597fa787): trying duplicate remote fd set. > [2017-05-13 19:11:05.136648] I [MSGID: 114024] > [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6: > /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. > 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 > (b2745d17-1972-4738-afa9-22e9597fa787): trying duplicate remote fd set. > [2017-05-13 19:11:05.136871] E [shard.c:426:shard_modify_size_and_block_count] > (-->/usr/lib64/glusterfs/3.10.1/xlator/cluster/distribute.so(+0x65df0) > [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xba6e) > [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xb29b) > [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get > trusted.glusterfs.shard.file-size for b2745d17-1972-4738-afa9-22e9597fa787 > > > -ben > > On Fri, May 12, 2017 at 11:46 PM, Benjamin Kingston <ben at nexusnebula.net> > wrote: > >> >> Hello all, >> >> I'm trying to take advantage of the shard xlator, however I've found it >> causes a lot of issues that I hope is easily resolvable >> >> 1) large file operations work well (copy file from folder a to folder b >> 2) seek operations and list operations frequently fail (ls directory, >> read bytes xyz at offset 235567) >> 3) Another issue is samba shares through samba-vfs show all files as 4MB, >> I've also seen this when mounting with fuse, however nfs-ganesha reflects >> correct file sizes always- >> >> >> Turning off the shard feature resolves this issue for new files created >> in the volume. mounted using the gluster fuse mount >> >> here's my volume settings, please let me know if there's some changes I >> can make. >> >> Volume Name: storage2 >> Type: Distributed-Replicate >> Volume ID: adaabca5-25ed-4e7f-ae86-2f20fc0143a8 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 3 x (2 + 1) = 9 >> Transport-type: tcp >> Bricks: >> Brick1: fd00:0:0:3::6:/mnt/gluster/storage/brick0/glusterfs2 >> Brick2: fd00:0:0:3::8:/mnt/gluster/storage/brick0/glusterfs2 >> Brick3: fd00:0:0:3::10:/mnt/gluster/storage/brick0/glusterfs (arbiter) >> Brick4: fd00:0:0:3::6:/mnt/gluster/storage/brick1/glusterfs2 >> Brick5: fd00:0:0:3::8:/mnt/gluster/storage/brick1/glusterfs2 >> Brick6: fd00:0:0:3::10:/mnt/gluster/storage/brick1/glusterfs (arbiter) >> Brick7: fd00:0:0:3::6:/mnt/gluster/storage/brick2/glusterfs2 >> Brick8: fd00:0:0:3::8:/mnt/gluster/storage/brick2/glusterfs2 >> Brick9: fd00:0:0:3::10:/mnt/gluster/storage/brick2/glusterfs (arbiter) >> Options Reconfigured: >> features.ctr-enabled: on >> features.shard-block-size: 4MB >> network.inode-lru-limit: 90000 >> features.cache-invalidation: on >> performance.readdir-ahead: on >> client.event-threads: 3 >> performance.cache-ima-xattrs: on >> cluster.data-self-heal-algorithm: diff >> network.remote-dio: disable >> cluster.use-compound-fops: on >> cluster.read-freq-threshold: 2 >> cluster.write-freq-threshold: 2 >> features.record-counters: on >> disperse.shd-max-threads: 4 >> performance.parallel-readdir: on >> performance.client-io-threads: on >> server.event-threads: 3 >> cluster.lookup-optimize: on >> performance.open-behind: on >> performance.stat-prefetch: on >> performance.quick-read: off >> performance.io-cache: on >> performance.read-ahead: off >> performance.write-behind: on >> features.scrub: Active >> features.bitrot: on >> features.leases: on >> features.shard: off >> transport.address-family: inet6 >> nfs.disable: on >> server.allow-insecure: on >> cluster.shd-max-threads: 8 >> performance.low-prio-threads: 32 >> cluster.locking-scheme: granular >> cluster.shd-wait-qlength: 10000 >> user.cifs: off >> cluster.eager-lock: enable >> cluster.quorum-type: auto >> cluster.server-quorum-type: server >> cluster.tier-compact: on >> storage.linux-aio: on >> transport.keepalive: on >> performance.write-behind-window-size: 2GB >> performance.flush-behind: on >> performance.cache-size: 1GB >> cluster.choose-local: on >> performance.io-thread-count: 64 >> cluster.brick-multiplex: off >> cluster.enable-shared-storage: enable >> nfs-ganesha: enable >> > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170515/33548a45/attachment.html>