Hi Guys, I've got a distributed replica 2+1 (rep 3 arbiter 1) cluster, and it appears a shard has been assigned different GFIDs on each replica set. ==============[2018-11-29 10:05:12.035422] W [MSGID: 109009] [dht-common.c:2148:dht_lookup_linkfile_cbk] 0-data-novo-dht: /.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846: gfid different on data file on data-novo-replicate-1, gfid local 00000000-0000-0000-0000-000000000000, gfid node 492d52d6-e3d1-4ed4-918d-9cdab7a135e0 [2018-11-29 10:05:12.036120] W [MSGID: 109009] [dht-common.c:1887:dht_lookup_everywhere_cbk] 0-data-novo-dht: /.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846: gfid differs on subvolume data-novo-replicate-1, gfid local c7f6cc63-ae40-4d1a-aa6f-fe97f7912036, gfid node 492d52d6-e3d1-4ed4-918d-9cdab7a135e0 [2018-11-29 10:05:12.036159] E [MSGID: 133010] [shard.c:1582:shard_common_lookup_shards_cbk] 0-data-novo-shard: Lookup on shard 1846 failed. Base file gfid = a46fd27c-5aa6-4fc8-b8e1-c097065e7096 [Stale file handle] [2018-11-29 10:05:12.036184] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 6916126: READ => -1 gfid=a46fd27c-5aa6-4fc8-b8e1-c097065e7096 fd=0x7f12fa57f06c (Stale file handle) ============== ==== FIRST REPLICA SET (v0-v2) === v0:~$ ls -l /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 ---------T. 2 root root 0 Oct 17 11:28 /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 v0:~$ getfattr -d -m . -e hex /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 # file: gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.gfid=0xc7f6cc63ae404d1aaa6ffe97f7912036 trusted.glusterfs.dht.linkto=0x646174612d6e6f766f2d7265706c69636174652d3100 v1:~$ ls -l /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 ---------T. 2 root root 0 Oct 17 11:28 /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 v1:~$ getfattr -d -m . -e hex /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 # file: gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.gfid=0xc7f6cc63ae404d1aaa6ffe97f7912036 trusted.glusterfs.dht.linkto=0x646174612d6e6f766f2d7265706c69636174652d3100 v2:~$ ls -l /gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 ---------T. 2 root root 0 Oct 17 11:28 /gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 v2:~$ getfattr -d -m . -e hex /gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 # file: gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.gfid=0xc7f6cc63ae404d1aaa6ffe97f7912036 trusted.glusterfs.dht.linkto=0x646174612d6e6f766f2d7265706c69636174652d3100 ==== SECOND REPLICA SET (v0-v2) === v2:~$ ls -l /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 -rw-rw----. 2 root root 536870912 Nov 27 14:15 /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 v2:~$ getfattr -d -m . -e hex /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 # file: gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.afr.dirty=0x000000000000000000000000 trusted.bit-rot.version=0x02000000000000005b4a3790000750d5 trusted.gfid=0x492d52d6e3d14ed4918d9cdab7a135e0 v3:~$ ls -l /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 -rw-rw----. 2 root root 536870912 Nov 27 14:15 /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 v3:~$ getfattr -d -m . -e hex /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 # file: gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.afr.dirty=0x000000000000000000000000 trusted.bit-rot.version=0x04000000000000005bda0242000c7cdd trusted.gfid=0x492d52d6e3d14ed4918d9cdab7a135e0 v0:~$ ls -l /gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 -rw-rw----. 2 root root 0 Oct 17 11:28 /gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 v0:~$ getfattr -d -m . -e hex /gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 # file: gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.afr.dirty=0x000000000000000000000000 trusted.bit-rot.version=0x03000000000000005bda02410001b61a trusted.gfid=0x492d52d6e3d14ed4918d9cdab7a135e0 ==================================== So the shard has GFID " 0xc7f6cc63ae404d1aaa6ffe97f7912036" on the first replica set, and GFID "0x492d52d6e3d14ed4918d9cdab7a135e0" on the second replica set. How do I go about correcting this? Many thanks, Doug -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181204/72542b8e/attachment.html>