Gluster users - Oct 2017 - gfid entries in volume heal info that do not heal

Hi Jim,

Can you check whether the same hardlinks are present on both the bricks &
both of them have the link count 2?
If the link count is 2 then "find <brickpath> -samefile
<brickpath/.glusterfs/<first two bits of gfid>/<next 2 bits of
gfid>/<full
gfid>"
should give you the file path.

Regards,
Karthik

On Tue, Oct 24, 2017 at 3:28 AM, Jim Kinney <jim.kinney at gmail.com>
wrote:
> I'm not so lucky. ALL of mine show 2 links and none have the attr data
> that supplies the path to the original.
>
> I have the inode from stat. Looking now to dig out the path/filename from
> xfs_db on the specific inodes individually.
>
> Is the hash of the filename or <path>/filename and if so relative to
> where? /, <path from top of brick>, ?
>
> On Mon, 2017-10-23 at 18:54 +0000, Matt Waymack wrote:
>
> In my case I was able to delete the hard links in the .glusterfs folders
> of the bricks and it seems to have done the trick, thanks!
>
>
>
> *From:* Karthik Subrahmanya [mailto:ksubrahm at redhat.com]
> *Sent:* Monday, October 23, 2017 1:52 AM
> *To:* Jim Kinney <jim.kinney at gmail.com>; Matt Waymack <mwaymack
at nsgdv.com>
> *Cc:* gluster-users <Gluster-users at gluster.org>
> *Subject:* Re: [Gluster-users] gfid entries in volume heal info that do
> not heal
>
>
>
> Hi Jim & Matt,
>
> Can you also check for the link count in the stat output of those hardlink
> entries in the .glusterfs folder on the bricks.
> If the link count is 1 on all the bricks for those entries, then they are
> orphaned entries and you can delete those hardlinks.
>
> To be on the safer side have a backup before deleting any of the entries.
>
> Regards,
>
> Karthik
>
>
>
> On Fri, Oct 20, 2017 at 3:18 AM, Jim Kinney <jim.kinney at gmail.com>
wrote:
>
> I've been following this particular thread as I have a similar issue
> (RAID6 array failed out with 3 dead drives at once while a 12 TB load was
> being copied into one mounted space - what a mess)
>
>
>
> I have >700K GFID entries that have no path data:
>
> Example:
>
> getfattr -d -e hex -m . .glusterfs/00/00/0000a5ef-
> 5af7-401b-84b5-ff2a51c10421
>
> # file: .glusterfs/00/00/0000a5ef-5af7-401b-84b5-ff2a51c10421
>
> security.selinux=0x73797374656d5f753a6f626a6563
> 745f723a756e6c6162656c65645f743a733000
>
> trusted.bit-rot.version=0x020000000000000059b1b316000270e7
>
> trusted.gfid=0x0000a5ef5af7401b84b5ff2a51c10421
>
>
>
> [root at bmidata1 brick]# getfattr -d -n trusted.glusterfs.pathinfo -e hex
> -m . .glusterfs/00/00/0000a5ef-5af7-401b-84b5-ff2a51c10421
>
> .glusterfs/00/00/0000a5ef-5af7-401b-84b5-ff2a51c10421:
> trusted.glusterfs.pathinfo: No such attribute
>
>
>
> I had to totally rebuild the dead RAID array and did a copy from the live
> one before activating gluster on the rebuilt system. I accidentally copied
> over the .glusterfs folder from the working side
>
> (replica 2 only for now - adding arbiter node as soon as I can get this
> one cleaned up).
>
>
>
> I've run the methods from "http://docs.gluster.org/en/
> latest/Troubleshooting/gfid-to-path/" with no results using random
GFIDs.
> A full systemic run using the script from method 3 crashes with "too
many
> nested links" error (or something similar).
>
>
>
> When I run gluster volume heal volname info, I get 700K+ GFIDs. Oh.
> gluster 3.8.4 on Centos 7.3
>
>
>
> Should I just remove the contents of the .glusterfs folder on both and
> restart gluster and run a ls/stat on every file?
>
>
>
>
>
> When I run a heal, it no longer has a decreasing number of files to heal
> so that's an improvement over the last 2-3 weeks :-)
>
>
>
> On Tue, 2017-10-17 at 14:34 +0000, Matt Waymack wrote:
>
> Attached is the heal log for the volume as well as the shd log.
>
>
>
>
>
>
>
> Run these commands on all the bricks of the replica pair to get the attrs
set on the backend.
>
>
>
>
>
>
>
> [root at tpc-cent-glus1-081017 ~]# getfattr -d -e hex -m .
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
>
> getfattr: Removing leading '/' from absolute path names
>
> # file: exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
>
>
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
>
> trusted.afr.dirty=0x000000000000000000000000
>
> trusted.afr.gv0-client-2=0x000000000000000100000000
>
> trusted.gfid=0x108694dbc0394b7cbd3dad6a15d811a2
>
>
trusted.gfid2path.9a2f5ada22eb9c45=0x38633262623330322d323466332d346463622d393630322d3839356136396461363131662f435f564f4c2d623030312d693637342d63642d63772e6d6435
>
>
>
> [root at tpc-cent-glus2-081017 ~]# getfattr -d -e hex -m .
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
>
> getfattr: Removing leading '/' from absolute path names
>
> # file: exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
>
>
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
>
> trusted.afr.dirty=0x000000000000000000000000
>
> trusted.afr.gv0-client-2=0x000000000000000100000000
>
> trusted.gfid=0x108694dbc0394b7cbd3dad6a15d811a2
>
>
trusted.gfid2path.9a2f5ada22eb9c45=0x38633262623330322d323466332d346463622d393630322d3839356136396461363131662f435f564f4c2d623030312d693637342d63642d63772e6d6435
>
>
>
> [root at tpc-arbiter1-100617 ~]# getfattr -d -e hex -m .
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
>
> getfattr:
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2: No such file
or directory
>
>
>
>
>
> [root at tpc-cent-glus1-081017 ~]# getfattr -d -e hex -m .
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
>
> getfattr: Removing leading '/' from absolute path names
>
> # file: exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
>
>
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
>
> trusted.afr.dirty=0x000000000000000000000000
>
> trusted.afr.gv0-client-11=0x000000000000000100000000
>
> trusted.gfid=0xe0c56bf78bfe46cabde1e46b92d33df3
>
>
trusted.gfid2path.be3ba24c3ef95ff2=0x63323366353834652d353566652d343033382d393131622d3866373063656334616136662f435f564f4c2d623030332d69313331342d63642d636d2d63722e6d6435
>
>
>
> [root at tpc-cent-glus2-081017 ~]# getfattr -d -e hex -m .
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
>
> getfattr: Removing leading '/' from absolute path names
>
> # file: exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
>
>
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
>
> trusted.afr.dirty=0x000000000000000000000000
>
> trusted.afr.gv0-client-11=0x000000000000000100000000
>
> trusted.gfid=0xe0c56bf78bfe46cabde1e46b92d33df3
>
>
trusted.gfid2path.be3ba24c3ef95ff2=0x63323366353834652d353566652d343033382d393131622d3866373063656334616136662f435f564f4c2d623030332d69313331342d63642d636d2d63722e6d6435
>
>
>
> [root at tpc-arbiter1-100617 ~]# getfattr -d -e hex -m .
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
>
> getfattr:
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3: No such file
or directory
>
>
>
>
>
>
>
> And the output of "gluster volume heal <volname> info
split-brain"
>
>
>
>
>
>
>
> [root at tpc-cent-glus1-081017 ~]# gluster volume heal gv0 info split-brain
>
> Brick tpc-cent-glus1-081017:/exp/b1/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b1/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-arbiter1-100617:/exp/b1/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-cent-glus1-081017:/exp/b2/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b2/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-arbiter1-100617:/exp/b2/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-cent-glus1-081017:/exp/b3/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b3/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-arbiter1-100617:/exp/b3/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-cent-glus1-081017:/exp/b4/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b4/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-arbiter1-100617:/exp/b4/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> -Matt
>
>
>
> From: Karthik Subrahmanya [mailto:ksubrahm at redhat.com <ksubrahm at
redhat.com>]
>
> Sent: Tuesday, October 17, 2017 1:26 AM
>
> To: Matt Waymack <mwaymack at nsgdv.com>
>
> Cc: gluster-users <Gluster-users at gluster.org>
>
> Subject: Re: [Gluster-users] gfid entries in volume heal info that do not
heal
>
>
>
> Hi Matt,
>
>
>
> Run these commands on all the bricks of the replica pair to get the attrs
set on the backend.
>
>
>
> On the bricks of first replica set:
>
> getfattr -d -e hex -m . <brick
path>/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
>
> On the fourth replica set:
>
> getfattr -d -e hex -m . <brick
path>/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
>
> Also run the "gluster volume heal <volname>" once and send
the shd log.
>
> And the output of "gluster volume heal <volname> info
split-brain"
>
> Regards,
>
> Karthik
>
>
>
> On Mon, Oct 16, 2017 at 9:51 PM, Matt Waymack <mailto:mwaymack at
nsgdv.com <mwaymack at nsgdv.com>> wrote:
>
> OK, so here?s my output of the volume info and the heal info. I have not
yet tracked down physical location of these files, any tips to finding them
would be appreciated, but I?m definitely just wanting them gone.  I forgot to
mention earlier that the cluster is running 3.12 and was upgraded from 3.10;
these files were likely stuck like this when it was on 3.10.
>
>
>
> [root at tpc-cent-glus1-081017 ~]# gluster volume info gv0
>
>
>
> Volume Name: gv0
>
> Type: Distributed-Replicate
>
> Volume ID: 8f07894d-e3ab-4a65-bda1-9d9dd46db007
>
> Status: Started
>
> Snapshot Count: 0
>
> Number of Bricks: 4 x (2 + 1) = 12
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: tpc-cent-glus1-081017:/exp/b1/gv0
>
> Brick2: tpc-cent-glus2-081017:/exp/b1/gv0
>
> Brick3: tpc-arbiter1-100617:/exp/b1/gv0 (arbiter)
>
> Brick4: tpc-cent-glus1-081017:/exp/b2/gv0
>
> Brick5: tpc-cent-glus2-081017:/exp/b2/gv0
>
> Brick6: tpc-arbiter1-100617:/exp/b2/gv0 (arbiter)
>
> Brick7: tpc-cent-glus1-081017:/exp/b3/gv0
>
> Brick8: tpc-cent-glus2-081017:/exp/b3/gv0
>
> Brick9: tpc-arbiter1-100617:/exp/b3/gv0 (arbiter)
>
> Brick10: tpc-cent-glus1-081017:/exp/b4/gv0
>
> Brick11: tpc-cent-glus2-081017:/exp/b4/gv0
>
> Brick12: tpc-arbiter1-100617:/exp/b4/gv0 (arbiter)
>
> Options Reconfigured:
>
> nfs.disable: on
>
> transport.address-family: inet
>
>
>
> [root at tpc-cent-glus1-081017 ~]# gluster volume heal gv0 info
>
> Brick tpc-cent-glus1-081017:/exp/b1/gv0
>
> <gfid:108694db-c039-4b7c-bd3d-ad6a15d811a2>
>
> <gfid:6d5ade20-8996-4de2-95d5-20ef98004742>
>
> <gfid:bc6cdc3d-5c46-4597-a7eb-282b21e9bdd5>
>
> <gfid:3c2ff4d1-3662-4214-8f21-f8f47dbdbf06>
>
> <gfid:053e2fb1-bc89-476e-a529-90dffa39963c>
>
>
>
> <removed to save scrolling>
>
>
>
> Status: Connected
>
> Number of entries: 118
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b1/gv0
>
> <gfid:108694db-c039-4b7c-bd3d-ad6a15d811a2>
>
> <gfid:6d5ade20-8996-4de2-95d5-20ef98004742>
>
> <gfid:bc6cdc3d-5c46-4597-a7eb-282b21e9bdd5>
>
> <gfid:3c2ff4d1-3662-4214-8f21-f8f47dbdbf06>
>
> <gfid:053e2fb1-bc89-476e-a529-90dffa39963c>
>
>
>
> <removed to save scrolling>
>
>
>
> Status: Connected
>
> Number of entries: 118
>
>
>
> Brick tpc-arbiter1-100617:/exp/b1/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-cent-glus1-081017:/exp/b2/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b2/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-arbiter1-100617:/exp/b2/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-cent-glus1-081017:/exp/b3/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b3/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-arbiter1-100617:/exp/b3/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-cent-glus1-081017:/exp/b4/gv0
>
> <gfid:e0c56bf7-8bfe-46ca-bde1-e46b92d33df3>
>
> <gfid:6f0a0549-8669-46de-8823-d6677fdca8e3>
>
> <gfid:d0e2fb2a-21b5-4ea8-a578-0801280b2530>
>
> <gfid:48bff79c-7bc2-4dc5-8b7f-4401b27fdf5a>
>
> <gfid:5902593d-a059-4ec7-b18b-7a2ab5c49a50>
>
> <gfid:cb821178-4621-4fcf-90f3-5b5c2ad7f756>
>
> <gfid:6aea0805-8dd1-437c-b922-52c9d11e488a>
>
> <gfid:f4076a37-2e2f-4d7a-90dd-0a3560a4bdff>
>
> <gfid:51ff7386-a550-4971-957c-b42c4d915e9f>
>
> <gfid:4309f7b8-3a9d-4bc8-ba2b-799f8a02611b>
>
> <gfid:b76746ec-6d7d-4ea3-a001-c96672a4d47e>
>
> <gfid:f8de26e7-d17d-41e0-adcd-e7d24ed74ac8>
>
> <gfid:8e2c4540-e0b4-4006-bb5d-aacd57f8f21b>
>
> <gfid:183ebefb-b827-4cbc-b42b-bfd136d5cabb>
>
> <gfid:88d492fe-bfbd-4463-ba55-0582d0ad671b>
>
> <gfid:e3a6c068-d48b-44b5-9480-245a69648a9b>
>
> <gfid:4aab9c6a-22d2-469a-a688-7b0a8784f4b1>
>
> <gfid:c6d182f2-7e46-4502-a0d2-b92824caa4de>
>
> <gfid:eb546f93-e9d6-4a59-ac35-6139b5c40919>
>
> <gfid:6043e381-7edf-4569-bc37-e27dd13549d2>
>
> <gfid:52090dc7-7a3c-40f9-9c54-3395f5158eab>
>
> <gfid:ecceee46-4310-421e-b56e-5fe46bd5263c>
>
> <gfid:354aea57-4b40-47fc-8ede-1d7e3b7501b4>
>
> <gfid:d43284d4-86aa-42ff-98b8-f6340b407d9d>
>
> Status: Connected
>
> Number of entries: 24
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b4/gv0
>
> <gfid:e0c56bf7-8bfe-46ca-bde1-e46b92d33df3>
>
> <gfid:6f0a0549-8669-46de-8823-d6677fdca8e3>
>
> <gfid:d0e2fb2a-21b5-4ea8-a578-0801280b2530>
>
> <gfid:48bff79c-7bc2-4dc5-8b7f-4401b27fdf5a>
>
> <gfid:5902593d-a059-4ec7-b18b-7a2ab5c49a50>
>
> <gfid:cb821178-4621-4fcf-90f3-5b5c2ad7f756>
>
> <gfid:6aea0805-8dd1-437c-b922-52c9d11e488a>
>
> <gfid:f4076a37-2e2f-4d7a-90dd-0a3560a4bdff>
>
> <gfid:51ff7386-a550-4971-957c-b42c4d915e9f>
>
> <gfid:4309f7b8-3a9d-4bc8-ba2b-799f8a02611b>
>
> <gfid:b76746ec-6d7d-4ea3-a001-c96672a4d47e>
>
> <gfid:f8de26e7-d17d-41e0-adcd-e7d24ed74ac8>
>
> <gfid:8e2c4540-e0b4-4006-bb5d-aacd57f8f21b>
>
> <gfid:183ebefb-b827-4cbc-b42b-bfd136d5cabb>
>
> <gfid:88d492fe-bfbd-4463-ba55-0582d0ad671b>
>
> <gfid:e3a6c068-d48b-44b5-9480-245a69648a9b>
>
> <gfid:4aab9c6a-22d2-469a-a688-7b0a8784f4b1>
>
> <gfid:c6d182f2-7e46-4502-a0d2-b92824caa4de>
>
> <gfid:eb546f93-e9d6-4a59-ac35-6139b5c40919>
>
> <gfid:6043e381-7edf-4569-bc37-e27dd13549d2>
>
> <gfid:52090dc7-7a3c-40f9-9c54-3395f5158eab>
>
> <gfid:ecceee46-4310-421e-b56e-5fe46bd5263c>
>
> <gfid:354aea57-4b40-47fc-8ede-1d7e3b7501b4>
>
> <gfid:d43284d4-86aa-42ff-98b8-f6340b407d9d>
>
> Status: Connected
>
> Number of entries: 24
>
>
>
> Brick tpc-arbiter1-100617:/exp/b4/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Thank you for your help!
>
>
>
> From: Karthik Subrahmanya [mailto:mailto:ksubrahm at redhat.com
<mailto:ksubrahm at redhat.com>]
>
> Sent: Monday, October 16, 2017 10:27 AM
>
> To: Matt Waymack <mailto:mwaymack at nsgdv.com <mwaymack at
nsgdv.com>>
>
> Cc: gluster-users <mailto:Gluster-users at gluster.org <Gluster-users
at gluster.org>>
>
> Subject: Re: [Gluster-users] gfid entries in volume heal info that do not
heal
>
>
>
> Hi Matt,
>
>
>
> The files might be in split brain. Could you please send the outputs of
these?
>
> gluster volume info <volname>
>
> gluster volume heal <volname> info
>
> And also the getfattr output of the files which are in the heal info output
from all the bricks of that replica pair.
>
> getfattr -d -e hex -m . <file path on brick>
>
>
>
> Thanks &  Regards
>
> Karthik
>
>
>
> On 16-Oct-2017 8:16 PM, "Matt Waymack" <mailto:mwaymack at
nsgdv.com <mwaymack at nsgdv.com>> wrote:
>
> Hi all,
>
>
>
> I have a volume where the output of volume heal info shows several gfid
entries to be healed, but they?ve been there for weeks and have not healed.  Any
normal file that shows up on the heal info does get healed as expected, but
these gfid entries do not.  Is there any way to remove these orphaned entries
from the volume so they are no longer stuck in the heal process?
>
>
>
> Thank you!
>
>
>
> _______________________________________________
>
> Gluster-users mailing list
>
> mailto:Gluster-users at gluster.org <Gluster-users at gluster.org>
>
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
>
> _______________________________________________
>
> Gluster-users mailing list
>
> Gluster-users at gluster.org
>
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171024/f156d546/attachment.html>

I have 14,734 GFIDS that are different. All the different ones are only
on the brick that was live during the outage and concurrent file copy-
in. The brick that was down at that time has no GFIDs that are not also
on the up brick.
As the bricks are 10TB, the find is going to be a long running process.
I'm running several finds at once with gnu parallel but it will still
take some time. Can't bring the up machine offline as it's in use. At
least I have 24 cores to work with. 
I've only tested with one GFID but the file it referenced _IS_ on the
down machine even though it has no GFID in the .glusterfs structure.
On Tue, 2017-10-24 at 12:35 +0530, Karthik Subrahmanya
wrote:
> Hi Jim,
> 
> Can you check whether the same hardlinks are present on both the
> bricks & both of them have the link count 2?
> If the link count is 2 then  "find <brickpath> -samefile
> <brickpath/.glusterfs/<first two
>  bits of gfid>/<next 2 bits of gfid>/<full gfid>"
> should give you the file path.
> 
> Regards,
> Karthik
> 
> On Tue, Oct 24, 2017 at 3:28 AM, Jim Kinney <jim.kinney at gmail.com>
> wrote:
> > 
> > 
> > 
> > I'm not so lucky. ALL of mine show 2 links and none have the attr
> > data that supplies the path to the original.
> > 
> > I have the inode from stat. Looking now to dig out the
> > path/filename from  xfs_db on the specific inodes individually.
> > 
> > Is the hash of the filename or <path>/filename and if so
relative
> > to where? /, <path from top of brick>, ?
> > 
> > On Mon, 2017-10-23 at 18:54 +0000, Matt Waymack wrote:
> > > In my case I was able to delete the hard links in the .glusterfs
> > > folders of the bricks and it seems to have done the trick,
> > > thanks!
> > >  
> > > 
> > > From: Karthik Subrahmanya [mailto:ksubrahm at redhat.com]
> > > 
> > > 
> > > Sent: Monday, October 23, 2017 1:52 AM
> > > 
> > > To: Jim Kinney <jim.kinney at gmail.com>; Matt Waymack
<mwaymack at nsg
> > > dv.com>
> > > 
> > > Cc: gluster-users <Gluster-users at gluster.org>
> > > 
> > > Subject: Re: [Gluster-users] gfid entries in volume heal info
> > > that do not heal
> > >  
> > > 
> > > 
> > > 
> > > Hi Jim & Matt,
> > > 
> > > Can you also check for the link count in the stat output of those
> > > hardlink entries in the .glusterfs folder on the bricks.
> > > 
> > > If the link count is 1 on all the bricks for those entries, then
> > > they are orphaned entries and you can delete those hardlinks.
> > > 
> > > 
> > > To be on the safer side have a backup before deleting any of the
> > > entries.
> > > 
> > > 
> > > Regards,
> > > 
> > > 
> > > Karthik
> > > 
> > > 
> > > 
> > >  
> > > 
> > > On Fri, Oct 20, 2017 at 3:18 AM, Jim Kinney <jim.kinney at
gmail.com
> > > > wrote:
> > > > 
> > > > I've been following this particular thread as I have a
similar
> > > > issue (RAID6 array failed out with 3 dead drives at once
while
> > > > a 12 TB load was being copied into one mounted space - what
a
> > > > mess)
> > > > 
> > > > 
> > > >  
> > > > 
> > > > 
> > > > I have >700K GFID entries that have no path data:
> > > > 
> > > > 
> > > > Example:
> > > > 
> > > > 
> > > > getfattr -d -e hex -m . .glusterfs/00/00/0000a5ef-5af7-401b-
> > > > 84b5-ff2a51c10421
> > > > 
> > > > 
> > > > # file:
.glusterfs/00/00/0000a5ef-5af7-401b-84b5-ff2a51c10421
> > > > 
> > > > 
> > > >
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c61
> > > > 62656c65645f743a733000
> > > > 
> > > > 
> > > > trusted.bit-rot.version=0x020000000000000059b1b316000270e7
> > > > 
> > > > 
> > > > trusted.gfid=0x0000a5ef5af7401b84b5ff2a51c10421
> > > > 
> > > > 
> > > >  
> > > > 
> > > > 
> > > > [root at bmidata1 brick]# getfattr -d -n
> > > > trusted.glusterfs.pathinfo -e hex -m .
> > > > .glusterfs/00/00/0000a5ef-5af7-401b-84b5-ff2a51c10421
> > > > 
> > > > 
> > > > .glusterfs/00/00/0000a5ef-5af7-401b-84b5-ff2a51c10421:
> > > > trusted.glusterfs.pathinfo: No such attribute
> > > > 
> > > > 
> > > >  
> > > > 
> > > > 
> > > > I had to totally rebuild the dead RAID array and did a copy
> > > > from the live one before activating gluster on the rebuilt
> > > > system. I accidentally copied over the .glusterfs folder
from
> > > > the working side
> > > > 
> > > > 
> > > > (replica 2 only for now - adding arbiter node as soon as I
can
> > > > get this one cleaned up).
> > > > 
> > > > 
> > > > 
> > > >  
> > > > 
> > > > 
> > > > I've run the methods from
"http://docs.gluster.org/en/latest/Tr
> > > > oubleshooting/gfid-to-path/" with no results using
random
> > > > GFIDs. A full systemic
> > > >  run using the script from method 3 crashes with "too
many
> > > > nested links" error (or something similar).
> > > > 
> > > > 
> > > >  
> > > > 
> > > > 
> > > > When I run gluster volume heal volname info, I get 700K+
GFIDs.
> > > > Oh. gluster 3.8.4 on Centos 7.3
> > > > 
> > > > 
> > > >  
> > > > 
> > > > 
> > > > Should I just remove the contents of the .glusterfs folder
on
> > > > both and restart gluster and run a ls/stat on every file?
> > > > 
> > > > 
> > > >  
> > > > 
> > > > 
> > > >  
> > > > 
> > > > 
> > > > When I run a heal, it no longer has a decreasing number of
> > > > files to heal so that's an improvement over the last 2-3
weeks
> > > > :-)
> > > > 
> > > > 
> > > > 
> > > > 
> > > >  
> > > > 
> > > > 
> > > > On Tue, 2017-10-17 at 14:34 +0000, Matt Waymack wrote:
> > > > 
> > > > 
> > > > 
> > > > > 
> > > > > Attached is the heal log for the volume as well as the
shd
> > > > > log. 
> > > > >  
> > > > > >  
> > > > > > >  
> > > > > > > Run these commands on all the bricks of the
replica pair
> > > > > > > to get the attrs set on the backend.
> > > > > > 
> > > > > >  
> > > > > 
> > > > >  
> > > > >  
> > > > > [root at tpc-cent-glus1-081017 ~]# getfattr -d -e hex
-m .
> > > > > /exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-
> > > > > ad6a15d811a2
> > > > > getfattr: Removing leading '/' from absolute
path names
> > > > > # file:
exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-
> > > > > ad6a15d811a2
> > > > >
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c
> > > > > 6162656c65645f743a733000
> > > > > trusted.afr.dirty=0x000000000000000000000000
> > > > > trusted.afr.gv0-client-2=0x000000000000000100000000
> > > > > trusted.gfid=0x108694dbc0394b7cbd3dad6a15d811a2
> > > > >
trusted.gfid2path.9a2f5ada22eb9c45=0x38633262623330322d323466
> > > > >
332d346463622d393630322d3839356136396461363131662f435f564f4c2
> > > > > d623030312d693637342d63642d63772e6d6435
> > > > >  
> > > > > [root at tpc-cent-glus2-081017 ~]# getfattr -d -e hex
-m .
> > > > > /exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-
> > > > > ad6a15d811a2
> > > > > getfattr: Removing leading '/' from absolute
path names
> > > > > # file:
exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-
> > > > > ad6a15d811a2
> > > > >
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c
> > > > > 6162656c65645f743a733000
> > > > > trusted.afr.dirty=0x000000000000000000000000
> > > > > trusted.afr.gv0-client-2=0x000000000000000100000000
> > > > > trusted.gfid=0x108694dbc0394b7cbd3dad6a15d811a2
> > > > >
trusted.gfid2path.9a2f5ada22eb9c45=0x38633262623330322d323466
> > > > >
332d346463622d393630322d3839356136396461363131662f435f564f4c2
> > > > > d623030312d693637342d63642d63772e6d6435
> > > > >  
> > > > > [root at tpc-arbiter1-100617 ~]# getfattr -d -e hex -m
.
> > > > > /exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-
> > > > > ad6a15d811a2
> > > > > getfattr:
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-
> > > > > bd3d-ad6a15d811a2: No such file or directory
> > > > >  
> > > > >  
> > > > > [root at tpc-cent-glus1-081017 ~]# getfattr -d -e hex
-m .
> > > > > /exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-
> > > > > e46b92d33df3
> > > > > getfattr: Removing leading '/' from absolute
path names
> > > > > # file:
exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-
> > > > > e46b92d33df3
> > > > >
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c
> > > > > 6162656c65645f743a733000
> > > > > trusted.afr.dirty=0x000000000000000000000000
> > > > > trusted.afr.gv0-client-11=0x000000000000000100000000
> > > > > trusted.gfid=0xe0c56bf78bfe46cabde1e46b92d33df3
> > > > >
trusted.gfid2path.be3ba24c3ef95ff2=0x63323366353834652d353566
> > > > >
652d343033382d393131622d3866373063656334616136662f435f564f4c2
> > > > > d623030332d69313331342d63642d636d2d63722e6d6435
> > > > >  
> > > > > [root at tpc-cent-glus2-081017 ~]# getfattr -d -e hex
-m .
> > > > > /exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-
> > > > > e46b92d33df3
> > > > > getfattr: Removing leading '/' from absolute
path names
> > > > > # file:
exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-
> > > > > e46b92d33df3
> > > > >
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c
> > > > > 6162656c65645f743a733000
> > > > > trusted.afr.dirty=0x000000000000000000000000
> > > > > trusted.afr.gv0-client-11=0x000000000000000100000000
> > > > > trusted.gfid=0xe0c56bf78bfe46cabde1e46b92d33df3
> > > > >
trusted.gfid2path.be3ba24c3ef95ff2=0x63323366353834652d353566
> > > > >
652d343033382d393131622d3866373063656334616136662f435f564f4c2
> > > > > d623030332d69313331342d63642d636d2d63722e6d6435
> > > > >  
> > > > > [root at tpc-arbiter1-100617 ~]# getfattr -d -e hex -m
.
> > > > > /exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-
> > > > > e46b92d33df3
> > > > > getfattr:
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-
> > > > > bde1-e46b92d33df3: No such file or directory
> > > > >  
> > > > > >  
> > > > > > >  
> > > > > > > And the output of "gluster volume heal
<volname> info
> > > > > > > split-brain"
> > > > > > 
> > > > > >  
> > > > > 
> > > > >  
> > > > >  
> > > > > [root at tpc-cent-glus1-081017 ~]# gluster volume heal
gv0 info
> > > > > split-brain
> > > > > Brick tpc-cent-glus1-081017:/exp/b1/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-cent-glus2-081017:/exp/b1/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-arbiter1-100617:/exp/b1/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-cent-glus1-081017:/exp/b2/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-cent-glus2-081017:/exp/b2/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-arbiter1-100617:/exp/b2/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-cent-glus1-081017:/exp/b3/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-cent-glus2-081017:/exp/b3/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-arbiter1-100617:/exp/b3/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-cent-glus1-081017:/exp/b4/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-cent-glus2-081017:/exp/b4/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-arbiter1-100617:/exp/b4/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > -Matt
> > > > >  
> > > > > From: Karthik Subrahmanya [mailto:ksubrahm at
redhat.com]
> > > > > Sent: Tuesday, October 17, 2017 1:26 AM
> > > > > To: Matt Waymack <mwaymack at nsgdv.com>
> > > > > Cc: gluster-users <Gluster-users at gluster.org>
> > > > > Subject: Re: [Gluster-users] gfid entries in volume
heal info
> > > > > that do not heal
> > > > >  
> > > > > Hi Matt,
> > > > >  
> > > > > Run these commands on all the bricks of the replica
pair to
> > > > > get the attrs set on the backend.
> > > > >  
> > > > > On the bricks of first replica set:
> > > > > getfattr -d -e hex -m . <brick
> > > > >
path>/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
> > > > > On the fourth replica set:
> > > > > getfattr -d -e hex -m . <brick
> > > > >
path>/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
> > > > > Also run the "gluster volume heal
<volname>" once and send
> > > > > the shd log.
> > > > > And the output of "gluster volume heal
<volname> info split-
> > > > > brain"
> > > > > Regards,
> > > > > Karthik
> > > > >  
> > > > > On Mon, Oct 16, 2017 at 9:51 PM, Matt Waymack
<mailto:mwaymac
> > > > > k at nsgdv.com> wrote:
> > > > > OK, so here?s my output of the volume info and the heal
info.
> > > > > I have not yet tracked down physical location of these
files,
> > > > > any tips to finding them would be appreciated, but I?m
> > > > > definitely just wanting them gone.  I forgot to mention
> > > > > earlier that the cluster is running 3.12 and was
upgraded
> > > > > from 3.10; these files were likely stuck like this when
it
> > > > > was on 3.10.
> > > > >  
> > > > > [root at tpc-cent-glus1-081017 ~]# gluster volume info
gv0
> > > > >  
> > > > > Volume Name: gv0
> > > > > Type: Distributed-Replicate
> > > > > Volume ID: 8f07894d-e3ab-4a65-bda1-9d9dd46db007
> > > > > Status: Started
> > > > > Snapshot Count: 0
> > > > > Number of Bricks: 4 x (2 + 1) = 12
> > > > > Transport-type: tcp
> > > > > Bricks:
> > > > > Brick1: tpc-cent-glus1-081017:/exp/b1/gv0
> > > > > Brick2: tpc-cent-glus2-081017:/exp/b1/gv0
> > > > > Brick3: tpc-arbiter1-100617:/exp/b1/gv0 (arbiter)
> > > > > Brick4: tpc-cent-glus1-081017:/exp/b2/gv0
> > > > > Brick5: tpc-cent-glus2-081017:/exp/b2/gv0
> > > > > Brick6: tpc-arbiter1-100617:/exp/b2/gv0 (arbiter)
> > > > > Brick7: tpc-cent-glus1-081017:/exp/b3/gv0
> > > > > Brick8: tpc-cent-glus2-081017:/exp/b3/gv0
> > > > > Brick9: tpc-arbiter1-100617:/exp/b3/gv0 (arbiter)
> > > > > Brick10: tpc-cent-glus1-081017:/exp/b4/gv0
> > > > > Brick11: tpc-cent-glus2-081017:/exp/b4/gv0
> > > > > Brick12: tpc-arbiter1-100617:/exp/b4/gv0 (arbiter)
> > > > > Options Reconfigured:
> > > > > nfs.disable: on
> > > > > transport.address-family: inet
> > > > >  
> > > > > [root at tpc-cent-glus1-081017 ~]# gluster volume heal
gv0 info
> > > > > Brick tpc-cent-glus1-081017:/exp/b1/gv0
> > > > > <gfid:108694db-c039-4b7c-bd3d-ad6a15d811a2>
> > > > > <gfid:6d5ade20-8996-4de2-95d5-20ef98004742>
> > > > > <gfid:bc6cdc3d-5c46-4597-a7eb-282b21e9bdd5>
> > > > > <gfid:3c2ff4d1-3662-4214-8f21-f8f47dbdbf06>
> > > > > <gfid:053e2fb1-bc89-476e-a529-90dffa39963c>
> > > > >  
> > > > > <removed to save scrolling>
> > > > >  
> > > > > Status: Connected
> > > > > Number of entries: 118
> > > > >  
> > > > > Brick tpc-cent-glus2-081017:/exp/b1/gv0
> > > > > <gfid:108694db-c039-4b7c-bd3d-ad6a15d811a2>
> > > > > <gfid:6d5ade20-8996-4de2-95d5-20ef98004742>
> > > > > <gfid:bc6cdc3d-5c46-4597-a7eb-282b21e9bdd5>
> > > > > <gfid:3c2ff4d1-3662-4214-8f21-f8f47dbdbf06>
> > > > > <gfid:053e2fb1-bc89-476e-a529-90dffa39963c>
> > > > >  
> > > > > <removed to save scrolling>
> > > > >  
> > > > > Status: Connected
> > > > > Number of entries: 118
> > > > >  
> > > > > Brick tpc-arbiter1-100617:/exp/b1/gv0
> > > > > Status: Connected
> > > > > Number of entries: 0
> > > > >  
> > > > > Brick tpc-cent-glus1-081017:/exp/b2/gv0
> > > > > Status: Connected
> > > > > Number of entries: 0
> > > > >  
> > > > > Brick tpc-cent-glus2-081017:/exp/b2/gv0
> > > > > Status: Connected
> > > > > Number of entries: 0
> > > > >  
> > > > > Brick tpc-arbiter1-100617:/exp/b2/gv0
> > > > > Status: Connected
> > > > > Number of entries: 0
> > > > >  
> > > > > Brick tpc-cent-glus1-081017:/exp/b3/gv0
> > > > > Status: Connected
> > > > > Number of entries: 0
> > > > >  
> > > > > Brick tpc-cent-glus2-081017:/exp/b3/gv0
> > > > > Status: Connected
> > > > > Number of entries: 0
> > > > >  
> > > > > Brick tpc-arbiter1-100617:/exp/b3/gv0
> > > > > Status: Connected
> > > > > Number of entries: 0
> > > > >  
> > > > > Brick tpc-cent-glus1-081017:/exp/b4/gv0
> > > > > <gfid:e0c56bf7-8bfe-46ca-bde1-e46b92d33df3>
> > > > > <gfid:6f0a0549-8669-46de-8823-d6677fdca8e3>
> > > > > <gfid:d0e2fb2a-21b5-4ea8-a578-0801280b2530>
> > > > > <gfid:48bff79c-7bc2-4dc5-8b7f-4401b27fdf5a>
> > > > > <gfid:5902593d-a059-4ec7-b18b-7a2ab5c49a50>
> > > > > <gfid:cb821178-4621-4fcf-90f3-5b5c2ad7f756>
> > > > > <gfid:6aea0805-8dd1-437c-b922-52c9d11e488a>
> > > > > <gfid:f4076a37-2e2f-4d7a-90dd-0a3560a4bdff>
> > > > > <gfid:51ff7386-a550-4971-957c-b42c4d915e9f>
> > > > > <gfid:4309f7b8-3a9d-4bc8-ba2b-799f8a02611b>
> > > > > <gfid:b76746ec-6d7d-4ea3-a001-c96672a4d47e>
> > > > > <gfid:f8de26e7-d17d-41e0-adcd-e7d24ed74ac8>
> > > > > <gfid:8e2c4540-e0b4-4006-bb5d-aacd57f8f21b>
> > > > > <gfid:183ebefb-b827-4cbc-b42b-bfd136d5cabb>
> > > > > <gfid:88d492fe-bfbd-4463-ba55-0582d0ad671b>
> > > > > <gfid:e3a6c068-d48b-44b5-9480-245a69648a9b>
> > > > > <gfid:4aab9c6a-22d2-469a-a688-7b0a8784f4b1>
> > > > > <gfid:c6d182f2-7e46-4502-a0d2-b92824caa4de>
> > > > > <gfid:eb546f93-e9d6-4a59-ac35-6139b5c40919>
> > > > > <gfid:6043e381-7edf-4569-bc37-e27dd13549d2>
> > > > > <gfid:52090dc7-7a3c-40f9-9c54-3395f5158eab>
> > > > > <gfid:ecceee46-4310-421e-b56e-5fe46bd5263c>
> > > > > <gfid:354aea57-4b40-47fc-8ede-1d7e3b7501b4>
> > > > > <gfid:d43284d4-86aa-42ff-98b8-f6340b407d9d>
> > > > > Status: Connected
> > > > > Number of entries: 24
> > > > >  
> > > > > Brick tpc-cent-glus2-081017:/exp/b4/gv0
> > > > > <gfid:e0c56bf7-8bfe-46ca-bde1-e46b92d33df3>
> > > > > <gfid:6f0a0549-8669-46de-8823-d6677fdca8e3>
> > > > > <gfid:d0e2fb2a-21b5-4ea8-a578-0801280b2530>
> > > > > <gfid:48bff79c-7bc2-4dc5-8b7f-4401b27fdf5a>
> > > > > <gfid:5902593d-a059-4ec7-b18b-7a2ab5c49a50>
> > > > > <gfid:cb821178-4621-4fcf-90f3-5b5c2ad7f756>
> > > > > <gfid:6aea0805-8dd1-437c-b922-52c9d11e488a>
> > > > > <gfid:f4076a37-2e2f-4d7a-90dd-0a3560a4bdff>
> > > > > <gfid:51ff7386-a550-4971-957c-b42c4d915e9f>
> > > > > <gfid:4309f7b8-3a9d-4bc8-ba2b-799f8a02611b>
> > > > > <gfid:b76746ec-6d7d-4ea3-a001-c96672a4d47e>
> > > > > <gfid:f8de26e7-d17d-41e0-adcd-e7d24ed74ac8>
> > > > > <gfid:8e2c4540-e0b4-4006-bb5d-aacd57f8f21b>
> > > > > <gfid:183ebefb-b827-4cbc-b42b-bfd136d5cabb>
> > > > > <gfid:88d492fe-bfbd-4463-ba55-0582d0ad671b>
> > > > > <gfid:e3a6c068-d48b-44b5-9480-245a69648a9b>
> > > > > <gfid:4aab9c6a-22d2-469a-a688-7b0a8784f4b1>
> > > > > <gfid:c6d182f2-7e46-4502-a0d2-b92824caa4de>
> > > > > <gfid:eb546f93-e9d6-4a59-ac35-6139b5c40919>
> > > > > <gfid:6043e381-7edf-4569-bc37-e27dd13549d2>
> > > > > <gfid:52090dc7-7a3c-40f9-9c54-3395f5158eab>
> > > > > <gfid:ecceee46-4310-421e-b56e-5fe46bd5263c>
> > > > > <gfid:354aea57-4b40-47fc-8ede-1d7e3b7501b4>
> > > > > <gfid:d43284d4-86aa-42ff-98b8-f6340b407d9d>
> > > > > Status: Connected
> > > > > Number of entries: 24
> > > > >  
> > > > > Brick tpc-arbiter1-100617:/exp/b4/gv0
> > > > > Status: Connected
> > > > > Number of entries: 0
> > > > >  
> > > > > Thank you for your help!
> > > > >  
> > > > > From: Karthik Subrahmanya [mailto:mailto:ksubrahm at
redhat.com]
> > > > >  
> > > > > Sent: Monday, October 16, 2017 10:27 AM
> > > > > To: Matt Waymack <mailto:mwaymack at nsgdv.com>
> > > > > Cc: gluster-users <mailto:Gluster-users at
gluster.org>
> > > > > Subject: Re: [Gluster-users] gfid entries in volume
heal info
> > > > > that do not heal
> > > > >  
> > > > > Hi Matt, 
> > > > >  
> > > > > The files might be in split brain. Could you please
send the
> > > > > outputs of these? 
> > > > > gluster volume info <volname>
> > > > > gluster volume heal <volname> info
> > > > > And also the getfattr output of the files which are in
the
> > > > > heal info output from all the bricks of that replica
pair.
> > > > > getfattr -d -e hex -m . <file path on brick>
> > > > >  
> > > > > Thanks &  Regards
> > > > > Karthik
> > > > >  
> > > > > On 16-Oct-2017 8:16 PM, "Matt Waymack"
<mailto:mwaymack at nsgdv
> > > > > .com> wrote:
> > > > > Hi all,
> > > > >  
> > > > > I have a volume where the output of volume heal info
shows
> > > > > several gfid entries to be healed, but they?ve been
there for
> > > > > weeks and have not healed.  Any normal file that shows
up on
> > > > > the heal info does get healed as expected, but these
gfid
> > > > > entries do not.  Is there any way to remove these
orphaned
> > > > > entries from the volume so they are no longer stuck in
the
> > > > > heal process?
> > > > >  
> > > > > Thank you!
> > > > >  
> > > > > _______________________________________________
> > > > > Gluster-users mailing list
> > > > > mailto:Gluster-users at gluster.org
> > > > > http://lists.gluster.org/mailman/listinfo/gluster-users
> > > > >  
> > > > >  
> > > > > 
> > > > > 
> > > > > _______________________________________________
> > > > > Gluster-users mailing list
> > > > > Gluster-users at gluster.org
> > > > > http://lists.gluster.org/mailman/listinfo/gluster-users
> > > 
> > >  
> > > 
> > > 
> > > 
> > > 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171024/b2c07429/attachment.html>

Hey Jim,

Please let me know what I understood is correct or not.
You have 14,734 GFIDs in .glusterfs which are only on the brick which was
up during the failure and it does not have any referenced file in that
brick.
The down brick has the file  inside the brick path, but not the GFID
hardlink in the .glusterfs folder.
Did I get it right?

Could you also give me these information as well?
1. What is the link count for those GFIDs in the up brick?
2. If the link count is 2 or more do you have a file path for those GFIDs
in the up brick? (use the find command)
3. Do you have the GFID hardlink or you have the file in the down brick?

Let me explain how the things work.
When a file gets created from the mount, the file will be created inside
the brick & a hardlink to that file will be created inside the .glusterfs
folder with its GFID.
So the link count for the file will be 2 (unless you create more hardlinks
manually). So after the failure happened I guess you should have both the
file & hardlink in the up brick,
and when you do the lookup on the mount for that file, it should create the
file & the hardlink on the brick which was down.

Regards,
Karthik

On Tue, Oct 24, 2017 at 10:29 PM, Jim Kinney <jim.kinney at gmail.com>
wrote:
> I have 14,734 GFIDS that are different. All the different ones are only on
> the brick that was live during the outage and concurrent file copy-in. The
> brick that was down at that time has no GFIDs that are not also on the up
> brick.
>
> As the bricks are 10TB, the find is going to be a long running process.
> I'm running several finds at once with gnu parallel but it will still
take
> some time. Can't bring the up machine offline as it's in use. At
least I
> have 24 cores to work with.
>
> I've only tested with one GFID but the file it referenced _IS_ on the
down
> machine even though it has no GFID in the .glusterfs structure.
>
> On Tue, 2017-10-24 at 12:35 +0530, Karthik Subrahmanya wrote:
>
> Hi Jim,
>
> Can you check whether the same hardlinks are present on both the bricks
&
> both of them have the link count 2?
> If the link count is 2 then "find <brickpath> -samefile
> <brickpath/.glusterfs/<first two bits of gfid>/<next 2 bits of
gfid>/<full
> gfid>"
> should give you the file path.
>
> Regards,
> Karthik
>
> On Tue, Oct 24, 2017 at 3:28 AM, Jim Kinney <jim.kinney at gmail.com>
wrote:
>
> I'm not so lucky. ALL of mine show 2 links and none have the attr data
> that supplies the path to the original.
>
> I have the inode from stat. Looking now to dig out the path/filename from
> xfs_db on the specific inodes individually.
>
> Is the hash of the filename or <path>/filename and if so relative to
> where? /, <path from top of brick>, ?
>
> On Mon, 2017-10-23 at 18:54 +0000, Matt Waymack wrote:
>
> In my case I was able to delete the hard links in the .glusterfs folders
> of the bricks and it seems to have done the trick, thanks!
>
>
>
> *From:* Karthik Subrahmanya [mailto:ksubrahm at redhat.com]
> *Sent:* Monday, October 23, 2017 1:52 AM
> *To:* Jim Kinney <jim.kinney at gmail.com>; Matt Waymack <mwaymack
at nsgdv.com>
> *Cc:* gluster-users <Gluster-users at gluster.org>
> *Subject:* Re: [Gluster-users] gfid entries in volume heal info that do
> not heal
>
>
>
> Hi Jim & Matt,
>
> Can you also check for the link count in the stat output of those hardlink
> entries in the .glusterfs folder on the bricks.
> If the link count is 1 on all the bricks for those entries, then they are
> orphaned entries and you can delete those hardlinks.
>
> To be on the safer side have a backup before deleting any of the entries.
>
> Regards,
>
> Karthik
>
>
>
> On Fri, Oct 20, 2017 at 3:18 AM, Jim Kinney <jim.kinney at gmail.com>
wrote:
>
> I've been following this particular thread as I have a similar issue
> (RAID6 array failed out with 3 dead drives at once while a 12 TB load was
> being copied into one mounted space - what a mess)
>
>
>
> I have >700K GFID entries that have no path data:
>
> Example:
>
> getfattr -d -e hex -m . .glusterfs/00/00/0000a5ef-5af7
> -401b-84b5-ff2a51c10421
>
> # file: .glusterfs/00/00/0000a5ef-5af7-401b-84b5-ff2a51c10421
>
> security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6
> c6162656c65645f743a733000
>
> trusted.bit-rot.version=0x020000000000000059b1b316000270e7
>
> trusted.gfid=0x0000a5ef5af7401b84b5ff2a51c10421
>
>
>
> [root at bmidata1 brick]# getfattr -d -n trusted.glusterfs.pathinfo -e hex
> -m . .glusterfs/00/00/0000a5ef-5af7-401b-84b5-ff2a51c10421
>
> .glusterfs/00/00/0000a5ef-5af7-401b-84b5-ff2a51c10421:
> trusted.glusterfs.pathinfo: No such attribute
>
>
>
> I had to totally rebuild the dead RAID array and did a copy from the live
> one before activating gluster on the rebuilt system. I accidentally copied
> over the .glusterfs folder from the working side
>
> (replica 2 only for now - adding arbiter node as soon as I can get this
> one cleaned up).
>
>
>
> I've run the methods from "http://docs.gluster.org/en/la
> test/Troubleshooting/gfid-to-path/" with no results using random
GFIDs. A
> full systemic run using the script from method 3 crashes with "too
many
> nested links" error (or something similar).
>
>
>
> When I run gluster volume heal volname info, I get 700K+ GFIDs. Oh.
> gluster 3.8.4 on Centos 7.3
>
>
>
> Should I just remove the contents of the .glusterfs folder on both and
> restart gluster and run a ls/stat on every file?
>
>
>
>
>
> When I run a heal, it no longer has a decreasing number of files to heal
> so that's an improvement over the last 2-3 weeks :-)
>
>
>
> On Tue, 2017-10-17 at 14:34 +0000, Matt Waymack wrote:
>
> Attached is the heal log for the volume as well as the shd log.
>
>
>
>
>
>
>
> Run these commands on all the bricks of the replica pair to get the attrs
set on the backend.
>
>
>
>
>
>
>
> [root at tpc-cent-glus1-081017 ~]# getfattr -d -e hex -m .
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
>
> getfattr: Removing leading '/' from absolute path names
>
> # file: exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
>
>
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
>
> trusted.afr.dirty=0x000000000000000000000000
>
> trusted.afr.gv0-client-2=0x000000000000000100000000
>
> trusted.gfid=0x108694dbc0394b7cbd3dad6a15d811a2
>
>
trusted.gfid2path.9a2f5ada22eb9c45=0x38633262623330322d323466332d346463622d393630322d3839356136396461363131662f435f564f4c2d623030312d693637342d63642d63772e6d6435
>
>
>
> [root at tpc-cent-glus2-081017 ~]# getfattr -d -e hex -m .
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
>
> getfattr: Removing leading '/' from absolute path names
>
> # file: exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
>
>
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
>
> trusted.afr.dirty=0x000000000000000000000000
>
> trusted.afr.gv0-client-2=0x000000000000000100000000
>
> trusted.gfid=0x108694dbc0394b7cbd3dad6a15d811a2
>
>
trusted.gfid2path.9a2f5ada22eb9c45=0x38633262623330322d323466332d346463622d393630322d3839356136396461363131662f435f564f4c2d623030312d693637342d63642d63772e6d6435
>
>
>
> [root at tpc-arbiter1-100617 ~]# getfattr -d -e hex -m .
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
>
> getfattr:
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2: No such file
or directory
>
>
>
>
>
> [root at tpc-cent-glus1-081017 ~]# getfattr -d -e hex -m .
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
>
> getfattr: Removing leading '/' from absolute path names
>
> # file: exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
>
>
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
>
> trusted.afr.dirty=0x000000000000000000000000
>
> trusted.afr.gv0-client-11=0x000000000000000100000000
>
> trusted.gfid=0xe0c56bf78bfe46cabde1e46b92d33df3
>
>
trusted.gfid2path.be3ba24c3ef95ff2=0x63323366353834652d353566652d343033382d393131622d3866373063656334616136662f435f564f4c2d623030332d69313331342d63642d636d2d63722e6d6435
>
>
>
> [root at tpc-cent-glus2-081017 ~]# getfattr -d -e hex -m .
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
>
> getfattr: Removing leading '/' from absolute path names
>
> # file: exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
>
>
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
>
> trusted.afr.dirty=0x000000000000000000000000
>
> trusted.afr.gv0-client-11=0x000000000000000100000000
>
> trusted.gfid=0xe0c56bf78bfe46cabde1e46b92d33df3
>
>
trusted.gfid2path.be3ba24c3ef95ff2=0x63323366353834652d353566652d343033382d393131622d3866373063656334616136662f435f564f4c2d623030332d69313331342d63642d636d2d63722e6d6435
>
>
>
> [root at tpc-arbiter1-100617 ~]# getfattr -d -e hex -m .
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
>
> getfattr:
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3: No such file
or directory
>
>
>
>
>
>
>
> And the output of "gluster volume heal <volname> info
split-brain"
>
>
>
>
>
>
>
> [root at tpc-cent-glus1-081017 ~]# gluster volume heal gv0 info split-brain
>
> Brick tpc-cent-glus1-081017:/exp/b1/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b1/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-arbiter1-100617:/exp/b1/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-cent-glus1-081017:/exp/b2/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b2/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-arbiter1-100617:/exp/b2/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-cent-glus1-081017:/exp/b3/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b3/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-arbiter1-100617:/exp/b3/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-cent-glus1-081017:/exp/b4/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b4/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> Brick tpc-arbiter1-100617:/exp/b4/gv0
>
> Status: Connected
>
> Number of entries in split-brain: 0
>
>
>
> -Matt
>
>
>
> From: Karthik Subrahmanya [mailto:ksubrahm at redhat.com <ksubrahm at
redhat.com>]
>
> Sent: Tuesday, October 17, 2017 1:26 AM
>
> To: Matt Waymack <mwaymack at nsgdv.com>
>
> Cc: gluster-users <Gluster-users at gluster.org>
>
> Subject: Re: [Gluster-users] gfid entries in volume heal info that do not
heal
>
>
>
> Hi Matt,
>
>
>
> Run these commands on all the bricks of the replica pair to get the attrs
set on the backend.
>
>
>
> On the bricks of first replica set:
>
> getfattr -d -e hex -m . <brick
path>/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
>
> On the fourth replica set:
>
> getfattr -d -e hex -m . <brick
path>/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
>
> Also run the "gluster volume heal <volname>" once and send
the shd log.
>
> And the output of "gluster volume heal <volname> info
split-brain"
>
> Regards,
>
> Karthik
>
>
>
> On Mon, Oct 16, 2017 at 9:51 PM, Matt Waymack <mailto:mwaymack at
nsgdv.com <mwaymack at nsgdv.com>> wrote:
>
> OK, so here?s my output of the volume info and the heal info. I have not
yet tracked down physical location of these files, any tips to finding them
would be appreciated, but I?m definitely just wanting them gone.  I forgot to
mention earlier that the cluster is running 3.12 and was upgraded from 3.10;
these files were likely stuck like this when it was on 3.10.
>
>
>
> [root at tpc-cent-glus1-081017 ~]# gluster volume info gv0
>
>
>
> Volume Name: gv0
>
> Type: Distributed-Replicate
>
> Volume ID: 8f07894d-e3ab-4a65-bda1-9d9dd46db007
>
> Status: Started
>
> Snapshot Count: 0
>
> Number of Bricks: 4 x (2 + 1) = 12
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: tpc-cent-glus1-081017:/exp/b1/gv0
>
> Brick2: tpc-cent-glus2-081017:/exp/b1/gv0
>
> Brick3: tpc-arbiter1-100617:/exp/b1/gv0 (arbiter)
>
> Brick4: tpc-cent-glus1-081017:/exp/b2/gv0
>
> Brick5: tpc-cent-glus2-081017:/exp/b2/gv0
>
> Brick6: tpc-arbiter1-100617:/exp/b2/gv0 (arbiter)
>
> Brick7: tpc-cent-glus1-081017:/exp/b3/gv0
>
> Brick8: tpc-cent-glus2-081017:/exp/b3/gv0
>
> Brick9: tpc-arbiter1-100617:/exp/b3/gv0 (arbiter)
>
> Brick10: tpc-cent-glus1-081017:/exp/b4/gv0
>
> Brick11: tpc-cent-glus2-081017:/exp/b4/gv0
>
> Brick12: tpc-arbiter1-100617:/exp/b4/gv0 (arbiter)
>
> Options Reconfigured:
>
> nfs.disable: on
>
> transport.address-family: inet
>
>
>
> [root at tpc-cent-glus1-081017 ~]# gluster volume heal gv0 info
>
> Brick tpc-cent-glus1-081017:/exp/b1/gv0
>
> <gfid:108694db-c039-4b7c-bd3d-ad6a15d811a2>
>
> <gfid:6d5ade20-8996-4de2-95d5-20ef98004742>
>
> <gfid:bc6cdc3d-5c46-4597-a7eb-282b21e9bdd5>
>
> <gfid:3c2ff4d1-3662-4214-8f21-f8f47dbdbf06>
>
> <gfid:053e2fb1-bc89-476e-a529-90dffa39963c>
>
>
>
> <removed to save scrolling>
>
>
>
> Status: Connected
>
> Number of entries: 118
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b1/gv0
>
> <gfid:108694db-c039-4b7c-bd3d-ad6a15d811a2>
>
> <gfid:6d5ade20-8996-4de2-95d5-20ef98004742>
>
> <gfid:bc6cdc3d-5c46-4597-a7eb-282b21e9bdd5>
>
> <gfid:3c2ff4d1-3662-4214-8f21-f8f47dbdbf06>
>
> <gfid:053e2fb1-bc89-476e-a529-90dffa39963c>
>
>
>
> <removed to save scrolling>
>
>
>
> Status: Connected
>
> Number of entries: 118
>
>
>
> Brick tpc-arbiter1-100617:/exp/b1/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-cent-glus1-081017:/exp/b2/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b2/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-arbiter1-100617:/exp/b2/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-cent-glus1-081017:/exp/b3/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b3/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-arbiter1-100617:/exp/b3/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-cent-glus1-081017:/exp/b4/gv0
>
> <gfid:e0c56bf7-8bfe-46ca-bde1-e46b92d33df3>
>
> <gfid:6f0a0549-8669-46de-8823-d6677fdca8e3>
>
> <gfid:d0e2fb2a-21b5-4ea8-a578-0801280b2530>
>
> <gfid:48bff79c-7bc2-4dc5-8b7f-4401b27fdf5a>
>
> <gfid:5902593d-a059-4ec7-b18b-7a2ab5c49a50>
>
> <gfid:cb821178-4621-4fcf-90f3-5b5c2ad7f756>
>
> <gfid:6aea0805-8dd1-437c-b922-52c9d11e488a>
>
> <gfid:f4076a37-2e2f-4d7a-90dd-0a3560a4bdff>
>
> <gfid:51ff7386-a550-4971-957c-b42c4d915e9f>
>
> <gfid:4309f7b8-3a9d-4bc8-ba2b-799f8a02611b>
>
> <gfid:b76746ec-6d7d-4ea3-a001-c96672a4d47e>
>
> <gfid:f8de26e7-d17d-41e0-adcd-e7d24ed74ac8>
>
> <gfid:8e2c4540-e0b4-4006-bb5d-aacd57f8f21b>
>
> <gfid:183ebefb-b827-4cbc-b42b-bfd136d5cabb>
>
> <gfid:88d492fe-bfbd-4463-ba55-0582d0ad671b>
>
> <gfid:e3a6c068-d48b-44b5-9480-245a69648a9b>
>
> <gfid:4aab9c6a-22d2-469a-a688-7b0a8784f4b1>
>
> <gfid:c6d182f2-7e46-4502-a0d2-b92824caa4de>
>
> <gfid:eb546f93-e9d6-4a59-ac35-6139b5c40919>
>
> <gfid:6043e381-7edf-4569-bc37-e27dd13549d2>
>
> <gfid:52090dc7-7a3c-40f9-9c54-3395f5158eab>
>
> <gfid:ecceee46-4310-421e-b56e-5fe46bd5263c>
>
> <gfid:354aea57-4b40-47fc-8ede-1d7e3b7501b4>
>
> <gfid:d43284d4-86aa-42ff-98b8-f6340b407d9d>
>
> Status: Connected
>
> Number of entries: 24
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b4/gv0
>
> <gfid:e0c56bf7-8bfe-46ca-bde1-e46b92d33df3>
>
> <gfid:6f0a0549-8669-46de-8823-d6677fdca8e3>
>
> <gfid:d0e2fb2a-21b5-4ea8-a578-0801280b2530>
>
> <gfid:48bff79c-7bc2-4dc5-8b7f-4401b27fdf5a>
>
> <gfid:5902593d-a059-4ec7-b18b-7a2ab5c49a50>
>
> <gfid:cb821178-4621-4fcf-90f3-5b5c2ad7f756>
>
> <gfid:6aea0805-8dd1-437c-b922-52c9d11e488a>
>
> <gfid:f4076a37-2e2f-4d7a-90dd-0a3560a4bdff>
>
> <gfid:51ff7386-a550-4971-957c-b42c4d915e9f>
>
> <gfid:4309f7b8-3a9d-4bc8-ba2b-799f8a02611b>
>
> <gfid:b76746ec-6d7d-4ea3-a001-c96672a4d47e>
>
> <gfid:f8de26e7-d17d-41e0-adcd-e7d24ed74ac8>
>
> <gfid:8e2c4540-e0b4-4006-bb5d-aacd57f8f21b>
>
> <gfid:183ebefb-b827-4cbc-b42b-bfd136d5cabb>
>
> <gfid:88d492fe-bfbd-4463-ba55-0582d0ad671b>
>
> <gfid:e3a6c068-d48b-44b5-9480-245a69648a9b>
>
> <gfid:4aab9c6a-22d2-469a-a688-7b0a8784f4b1>
>
> <gfid:c6d182f2-7e46-4502-a0d2-b92824caa4de>
>
> <gfid:eb546f93-e9d6-4a59-ac35-6139b5c40919>
>
> <gfid:6043e381-7edf-4569-bc37-e27dd13549d2>
>
> <gfid:52090dc7-7a3c-40f9-9c54-3395f5158eab>
>
> <gfid:ecceee46-4310-421e-b56e-5fe46bd5263c>
>
> <gfid:354aea57-4b40-47fc-8ede-1d7e3b7501b4>
>
> <gfid:d43284d4-86aa-42ff-98b8-f6340b407d9d>
>
> Status: Connected
>
> Number of entries: 24
>
>
>
> Brick tpc-arbiter1-100617:/exp/b4/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Thank you for your help!
>
>
>
> From: Karthik Subrahmanya [mailto:mailto:ksubrahm at redhat.com
<mailto:ksubrahm at redhat.com>]
>
> Sent: Monday, October 16, 2017 10:27 AM
>
> To: Matt Waymack <mailto:mwaymack at nsgdv.com <mwaymack at
nsgdv.com>>
>
> Cc: gluster-users <mailto:Gluster-users at gluster.org <Gluster-users
at gluster.org>>
>
> Subject: Re: [Gluster-users] gfid entries in volume heal info that do not
heal
>
>
>
> Hi Matt,
>
>
>
> The files might be in split brain. Could you please send the outputs of
these?
>
> gluster volume info <volname>
>
> gluster volume heal <volname> info
>
> And also the getfattr output of the files which are in the heal info output
from all the bricks of that replica pair.
>
> getfattr -d -e hex -m . <file path on brick>
>
>
>
> Thanks &  Regards
>
> Karthik
>
>
>
> On 16-Oct-2017 8:16 PM, "Matt Waymack" <mailto:mwaymack at
nsgdv.com <mwaymack at nsgdv.com>> wrote:
>
> Hi all,
>
>
>
> I have a volume where the output of volume heal info shows several gfid
entries to be healed, but they?ve been there for weeks and have not healed.  Any
normal file that shows up on the heal info does get healed as expected, but
these gfid entries do not.  Is there any way to remove these orphaned entries
from the volume so they are no longer stuck in the heal process?
>
>
>
> Thank you!
>
>
>
> _______________________________________________
>
> Gluster-users mailing list
>
> mailto:Gluster-users at gluster.org <Gluster-users at gluster.org>
>
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
>
> _______________________________________________
>
> Gluster-users mailing list
>
> Gluster-users at gluster.org
>
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171025/3f8e9d78/attachment.html>

That took a while!
I have the following stats:
4085169 files in both bricks3162940 files only have a single hard link.
All of the files exist on both servers. bmidata2 (below) WAS running
when bmidata1 died.
gluster volume heal clifford  statistics heal-countGathering count of
entries to be healed on volume clifford has been successful 
Brick bmidata1:/data/glusterfs/clifford/brick/brickNumber of entries: 0
Brick bmidata2:/data/glusterfs/clifford/brick/brickNumber of entries:
296252
Brick bmidata1:/data/glusterfs/clifford3/brick/brickNumber of entries:
1
Brick bmidata2:/data/glusterfs/clifford3/brick/brickNumber of entries:
182407
Why those numbers are so much smaller than the data from a stat run
through the entire brick, I have no idea.
The 20TB of space for this mount point is composed of 4 10TB brick in a
2x2.
As this is all from a large copy in from a backup source, I'm thinking
of rerunning rsync to overwrite files with same create/modify times on
the mount to realign things (maybe?)
I ran a giant ls/stat on the mount but nothing changed. Ran it again
with no changes.
gluster-health-report 
Loaded reports: glusterd-op-version, georep, gfid-mismatch-dht-report,
glusterd-peer-disconnect, disk_usage, errors_in_logs, coredump,
glusterd, glusterd_volume_version_cksum_errors, kernel_issues,
errors_in_logs, ifconfig, nic-health, process_status
[     OK] Disk used percentage  path=/  percentage=7[     OK] Disk used
percentage  path=/var  percentage=7[     OK] Disk used
percentage  path=/tmp  percentage=7[  ERROR] Report
failure  report=report_check_errors_in_glusterd_log[     OK] All peers
are in connected state  connected_count=1  total_peer_count=1[     OK]
no gfid mismatch[ NOT OK] Failed to check op-version[ NOT OK] The
maximum size of core files created is NOT set to unlimited.[  ERROR]
Report failure  report=report_check_worker_restarts[  ERROR] Report
failure  report=report_non_participating_bricks[     OK] Glusterd is
running  uptime_sec=5177509[  ERROR] Report
failure  report=report_check_version_or_cksum_errors_in_glusterd_log[  
ERROR] Report failure  report=report_check_errors_in_glusterd_log[ NOT
OK] Recieve errors in "ifconfig enp131s0" output[ NOT OK] Recieve
errors in "ifconfig eth0" output[ NOT OK] Recieve errors in
"ifconfig
eth3" output[ NOT OK] Recieve errors in "ifconfig mlx_ib0"
output[ NOT
OK] Transmission errors in "ifconfig mlx_ib0" output[ NOT OK] Errors
seen in "cat /proc/net/dev -- eth0" output[ NOT OK] Errors seen in
"cat
/proc/net/dev -- eth3" output[ NOT OK] Errors seen in "cat
/proc/net/dev -- mlx_ib0" output[ NOT OK] Errors seen in "cat
/proc/net/dev -- enp131s0" outputHigh CPU usage by Self-heal
NOTE: Bmidata2 up for over 300 days. due for reboot. 
On Tue, 2017-10-24 at 12:35 +0530, Karthik Subrahmanya
wrote:
> Hi Jim,
> 
> Can you check whether the same hardlinks are present on both the
> bricks & both of them have the link count 2?
> If the link count is 2 then  "find <brickpath> -samefile
> <brickpath/.glusterfs/<first two
>  bits of gfid>/<next 2 bits of gfid>/<full gfid>"
> should give you the file path.
> 
> Regards,
> Karthik
> 
> On Tue, Oct 24, 2017 at 3:28 AM, Jim Kinney <jim.kinney at gmail.com>
> wrote:
> > 
> > 
> > 
> > I'm not so lucky. ALL of mine show 2 links and none have the attr
> > data that supplies the path to the original.
> > 
> > I have the inode from stat. Looking now to dig out the
> > path/filename from  xfs_db on the specific inodes individually.
> > 
> > Is the hash of the filename or <path>/filename and if so
relative
> > to where? /, <path from top of brick>, ?
> > 
> > On Mon, 2017-10-23 at 18:54 +0000, Matt Waymack wrote:
> > > In my case I was able to delete the hard links in the .glusterfs
> > > folders of the bricks and it seems to have done the trick,
> > > thanks!
> > >  
> > > 
> > > From: Karthik Subrahmanya [mailto:ksubrahm at redhat.com]
> > > 
> > > 
> > > Sent: Monday, October 23, 2017 1:52 AM
> > > 
> > > To: Jim Kinney <jim.kinney at gmail.com>; Matt Waymack
<mwaymack at nsg
> > > dv.com>
> > > 
> > > Cc: gluster-users <Gluster-users at gluster.org>
> > > 
> > > Subject: Re: [Gluster-users] gfid entries in volume heal info
> > > that do not heal
> > >  
> > > 
> > > 
> > > 
> > > Hi Jim & Matt,
> > > 
> > > Can you also check for the link count in the stat output of those
> > > hardlink entries in the .glusterfs folder on the bricks.
> > > 
> > > If the link count is 1 on all the bricks for those entries, then
> > > they are orphaned entries and you can delete those hardlinks.
> > > 
> > > 
> > > To be on the safer side have a backup before deleting any of the
> > > entries.
> > > 
> > > 
> > > Regards,
> > > 
> > > 
> > > Karthik
> > > 
> > > 
> > > 
> > >  
> > > 
> > > On Fri, Oct 20, 2017 at 3:18 AM, Jim Kinney <jim.kinney at
gmail.com
> > > > wrote:
> > > > 
> > > > I've been following this particular thread as I have a
similar
> > > > issue (RAID6 array failed out with 3 dead drives at once
while
> > > > a 12 TB load was being copied into one mounted space - what
a
> > > > mess)
> > > > 
> > > > 
> > > >  
> > > > 
> > > > 
> > > > I have >700K GFID entries that have no path data:
> > > > 
> > > > 
> > > > Example:
> > > > 
> > > > 
> > > > getfattr -d -e hex -m . .glusterfs/00/00/0000a5ef-5af7-401b-
> > > > 84b5-ff2a51c10421
> > > > 
> > > > 
> > > > # file:
.glusterfs/00/00/0000a5ef-5af7-401b-84b5-ff2a51c10421
> > > > 
> > > > 
> > > >
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c61
> > > > 62656c65645f743a733000
> > > > 
> > > > 
> > > > trusted.bit-rot.version=0x020000000000000059b1b316000270e7
> > > > 
> > > > 
> > > > trusted.gfid=0x0000a5ef5af7401b84b5ff2a51c10421
> > > > 
> > > > 
> > > >  
> > > > 
> > > > 
> > > > [root at bmidata1 brick]# getfattr -d -n
> > > > trusted.glusterfs.pathinfo -e hex -m .
> > > > .glusterfs/00/00/0000a5ef-5af7-401b-84b5-ff2a51c10421
> > > > 
> > > > 
> > > > .glusterfs/00/00/0000a5ef-5af7-401b-84b5-ff2a51c10421:
> > > > trusted.glusterfs.pathinfo: No such attribute
> > > > 
> > > > 
> > > >  
> > > > 
> > > > 
> > > > I had to totally rebuild the dead RAID array and did a copy
> > > > from the live one before activating gluster on the rebuilt
> > > > system. I accidentally copied over the .glusterfs folder
from
> > > > the working side
> > > > 
> > > > 
> > > > (replica 2 only for now - adding arbiter node as soon as I
can
> > > > get this one cleaned up).
> > > > 
> > > > 
> > > > 
> > > >  
> > > > 
> > > > 
> > > > I've run the methods from
"http://docs.gluster.org/en/latest/Tr
> > > > oubleshooting/gfid-to-path/" with no results using
random
> > > > GFIDs. A full systemic
> > > >  run using the script from method 3 crashes with "too
many
> > > > nested links" error (or something similar).
> > > > 
> > > > 
> > > >  
> > > > 
> > > > 
> > > > When I run gluster volume heal volname info, I get 700K+
GFIDs.
> > > > Oh. gluster 3.8.4 on Centos 7.3
> > > > 
> > > > 
> > > >  
> > > > 
> > > > 
> > > > Should I just remove the contents of the .glusterfs folder
on
> > > > both and restart gluster and run a ls/stat on every file?
> > > > 
> > > > 
> > > >  
> > > > 
> > > > 
> > > >  
> > > > 
> > > > 
> > > > When I run a heal, it no longer has a decreasing number of
> > > > files to heal so that's an improvement over the last 2-3
weeks
> > > > :-)
> > > > 
> > > > 
> > > > 
> > > > 
> > > >  
> > > > 
> > > > 
> > > > On Tue, 2017-10-17 at 14:34 +0000, Matt Waymack wrote:
> > > > 
> > > > 
> > > > 
> > > > > 
> > > > > Attached is the heal log for the volume as well as the
shd
> > > > > log. 
> > > > >  
> > > > > >  
> > > > > > >  
> > > > > > > Run these commands on all the bricks of the
replica pair
> > > > > > > to get the attrs set on the backend.
> > > > > > 
> > > > > >  
> > > > > 
> > > > >  
> > > > >  
> > > > > [root at tpc-cent-glus1-081017 ~]# getfattr -d -e hex
-m .
> > > > > /exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-
> > > > > ad6a15d811a2
> > > > > getfattr: Removing leading '/' from absolute
path names
> > > > > # file:
exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-
> > > > > ad6a15d811a2
> > > > >
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c
> > > > > 6162656c65645f743a733000
> > > > > trusted.afr.dirty=0x000000000000000000000000
> > > > > trusted.afr.gv0-client-2=0x000000000000000100000000
> > > > > trusted.gfid=0x108694dbc0394b7cbd3dad6a15d811a2
> > > > >
trusted.gfid2path.9a2f5ada22eb9c45=0x38633262623330322d323466
> > > > >
332d346463622d393630322d3839356136396461363131662f435f564f4c2
> > > > > d623030312d693637342d63642d63772e6d6435
> > > > >  
> > > > > [root at tpc-cent-glus2-081017 ~]# getfattr -d -e hex
-m .
> > > > > /exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-
> > > > > ad6a15d811a2
> > > > > getfattr: Removing leading '/' from absolute
path names
> > > > > # file:
exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-
> > > > > ad6a15d811a2
> > > > >
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c
> > > > > 6162656c65645f743a733000
> > > > > trusted.afr.dirty=0x000000000000000000000000
> > > > > trusted.afr.gv0-client-2=0x000000000000000100000000
> > > > > trusted.gfid=0x108694dbc0394b7cbd3dad6a15d811a2
> > > > >
trusted.gfid2path.9a2f5ada22eb9c45=0x38633262623330322d323466
> > > > >
332d346463622d393630322d3839356136396461363131662f435f564f4c2
> > > > > d623030312d693637342d63642d63772e6d6435
> > > > >  
> > > > > [root at tpc-arbiter1-100617 ~]# getfattr -d -e hex -m
.
> > > > > /exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-
> > > > > ad6a15d811a2
> > > > > getfattr:
/exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-
> > > > > bd3d-ad6a15d811a2: No such file or directory
> > > > >  
> > > > >  
> > > > > [root at tpc-cent-glus1-081017 ~]# getfattr -d -e hex
-m .
> > > > > /exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-
> > > > > e46b92d33df3
> > > > > getfattr: Removing leading '/' from absolute
path names
> > > > > # file:
exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-
> > > > > e46b92d33df3
> > > > >
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c
> > > > > 6162656c65645f743a733000
> > > > > trusted.afr.dirty=0x000000000000000000000000
> > > > > trusted.afr.gv0-client-11=0x000000000000000100000000
> > > > > trusted.gfid=0xe0c56bf78bfe46cabde1e46b92d33df3
> > > > >
trusted.gfid2path.be3ba24c3ef95ff2=0x63323366353834652d353566
> > > > >
652d343033382d393131622d3866373063656334616136662f435f564f4c2
> > > > > d623030332d69313331342d63642d636d2d63722e6d6435
> > > > >  
> > > > > [root at tpc-cent-glus2-081017 ~]# getfattr -d -e hex
-m .
> > > > > /exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-
> > > > > e46b92d33df3
> > > > > getfattr: Removing leading '/' from absolute
path names
> > > > > # file:
exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-
> > > > > e46b92d33df3
> > > > >
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c
> > > > > 6162656c65645f743a733000
> > > > > trusted.afr.dirty=0x000000000000000000000000
> > > > > trusted.afr.gv0-client-11=0x000000000000000100000000
> > > > > trusted.gfid=0xe0c56bf78bfe46cabde1e46b92d33df3
> > > > >
trusted.gfid2path.be3ba24c3ef95ff2=0x63323366353834652d353566
> > > > >
652d343033382d393131622d3866373063656334616136662f435f564f4c2
> > > > > d623030332d69313331342d63642d636d2d63722e6d6435
> > > > >  
> > > > > [root at tpc-arbiter1-100617 ~]# getfattr -d -e hex -m
.
> > > > > /exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-
> > > > > e46b92d33df3
> > > > > getfattr:
/exp/b4/gv0/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-
> > > > > bde1-e46b92d33df3: No such file or directory
> > > > >  
> > > > > >  
> > > > > > >  
> > > > > > > And the output of "gluster volume heal
<volname> info
> > > > > > > split-brain"
> > > > > > 
> > > > > >  
> > > > > 
> > > > >  
> > > > >  
> > > > > [root at tpc-cent-glus1-081017 ~]# gluster volume heal
gv0 info
> > > > > split-brain
> > > > > Brick tpc-cent-glus1-081017:/exp/b1/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-cent-glus2-081017:/exp/b1/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-arbiter1-100617:/exp/b1/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-cent-glus1-081017:/exp/b2/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-cent-glus2-081017:/exp/b2/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-arbiter1-100617:/exp/b2/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-cent-glus1-081017:/exp/b3/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-cent-glus2-081017:/exp/b3/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-arbiter1-100617:/exp/b3/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-cent-glus1-081017:/exp/b4/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-cent-glus2-081017:/exp/b4/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > Brick tpc-arbiter1-100617:/exp/b4/gv0
> > > > > Status: Connected
> > > > > Number of entries in split-brain: 0
> > > > >  
> > > > > -Matt
> > > > >  
> > > > > From: Karthik Subrahmanya [mailto:ksubrahm at
redhat.com]
> > > > > Sent: Tuesday, October 17, 2017 1:26 AM
> > > > > To: Matt Waymack <mwaymack at nsgdv.com>
> > > > > Cc: gluster-users <Gluster-users at gluster.org>
> > > > > Subject: Re: [Gluster-users] gfid entries in volume
heal info
> > > > > that do not heal
> > > > >  
> > > > > Hi Matt,
> > > > >  
> > > > > Run these commands on all the bricks of the replica
pair to
> > > > > get the attrs set on the backend.
> > > > >  
> > > > > On the bricks of first replica set:
> > > > > getfattr -d -e hex -m . <brick
> > > > >
path>/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2
> > > > > On the fourth replica set:
> > > > > getfattr -d -e hex -m . <brick
> > > > >
path>/.glusterfs/e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3
> > > > > Also run the "gluster volume heal
<volname>" once and send
> > > > > the shd log.
> > > > > And the output of "gluster volume heal
<volname> info split-
> > > > > brain"
> > > > > Regards,
> > > > > Karthik
> > > > >  
> > > > > On Mon, Oct 16, 2017 at 9:51 PM, Matt Waymack
<mailto:mwaymac
> > > > > k at nsgdv.com> wrote:
> > > > > OK, so here?s my output of the volume info and the heal
info.
> > > > > I have not yet tracked down physical location of these
files,
> > > > > any tips to finding them would be appreciated, but I?m
> > > > > definitely just wanting them gone.  I forgot to mention
> > > > > earlier that the cluster is running 3.12 and was
upgraded
> > > > > from 3.10; these files were likely stuck like this when
it
> > > > > was on 3.10.
> > > > >  
> > > > > [root at tpc-cent-glus1-081017 ~]# gluster volume info
gv0
> > > > >  
> > > > > Volume Name: gv0
> > > > > Type: Distributed-Replicate
> > > > > Volume ID: 8f07894d-e3ab-4a65-bda1-9d9dd46db007
> > > > > Status: Started
> > > > > Snapshot Count: 0
> > > > > Number of Bricks: 4 x (2 + 1) = 12
> > > > > Transport-type: tcp
> > > > > Bricks:
> > > > > Brick1: tpc-cent-glus1-081017:/exp/b1/gv0
> > > > > Brick2: tpc-cent-glus2-081017:/exp/b1/gv0
> > > > > Brick3: tpc-arbiter1-100617:/exp/b1/gv0 (arbiter)
> > > > > Brick4: tpc-cent-glus1-081017:/exp/b2/gv0
> > > > > Brick5: tpc-cent-glus2-081017:/exp/b2/gv0
> > > > > Brick6: tpc-arbiter1-100617:/exp/b2/gv0 (arbiter)
> > > > > Brick7: tpc-cent-glus1-081017:/exp/b3/gv0
> > > > > Brick8: tpc-cent-glus2-081017:/exp/b3/gv0
> > > > > Brick9: tpc-arbiter1-100617:/exp/b3/gv0 (arbiter)
> > > > > Brick10: tpc-cent-glus1-081017:/exp/b4/gv0
> > > > > Brick11: tpc-cent-glus2-081017:/exp/b4/gv0
> > > > > Brick12: tpc-arbiter1-100617:/exp/b4/gv0 (arbiter)
> > > > > Options Reconfigured:
> > > > > nfs.disable: on
> > > > > transport.address-family: inet
> > > > >  
> > > > > [root at tpc-cent-glus1-081017 ~]# gluster volume heal
gv0 info
> > > > > Brick tpc-cent-glus1-081017:/exp/b1/gv0
> > > > > <gfid:108694db-c039-4b7c-bd3d-ad6a15d811a2>
> > > > > <gfid:6d5ade20-8996-4de2-95d5-20ef98004742>
> > > > > <gfid:bc6cdc3d-5c46-4597-a7eb-282b21e9bdd5>
> > > > > <gfid:3c2ff4d1-3662-4214-8f21-f8f47dbdbf06>
> > > > > <gfid:053e2fb1-bc89-476e-a529-90dffa39963c>
> > > > >  
> > > > > <removed to save scrolling>
> > > > >  
> > > > > Status: Connected
> > > > > Number of entries: 118
> > > > >  
> > > > > Brick tpc-cent-glus2-081017:/exp/b1/gv0
> > > > > <gfid:108694db-c039-4b7c-bd3d-ad6a15d811a2>
> > > > > <gfid:6d5ade20-8996-4de2-95d5-20ef98004742>
> > > > > <gfid:bc6cdc3d-5c46-4597-a7eb-282b21e9bdd5>
> > > > > <gfid:3c2ff4d1-3662-4214-8f21-f8f47dbdbf06>
> > > > > <gfid:053e2fb1-bc89-476e-a529-90dffa39963c>
> > > > >  
> > > > > <removed to save scrolling>
> > > > >  
> > > > > Status: Connected
> > > > > Number of entries: 118
> > > > >  
> > > > > Brick tpc-arbiter1-100617:/exp/b1/gv0
> > > > > Status: Connected
> > > > > Number of entries: 0
> > > > >  
> > > > > Brick tpc-cent-glus1-081017:/exp/b2/gv0
> > > > > Status: Connected
> > > > > Number of entries: 0
> > > > >  
> > > > > Brick tpc-cent-glus2-081017:/exp/b2/gv0
> > > > > Status: Connected
> > > > > Number of entries: 0
> > > > >  
> > > > > Brick tpc-arbiter1-100617:/exp/b2/gv0
> > > > > Status: Connected
> > > > > Number of entries: 0
> > > > >  
> > > > > Brick tpc-cent-glus1-081017:/exp/b3/gv0
> > > > > Status: Connected
> > > > > Number of entries: 0
> > > > >  
> > > > > Brick tpc-cent-glus2-081017:/exp/b3/gv0
> > > > > Status: Connected
> > > > > Number of entries: 0
> > > > >  
> > > > > Brick tpc-arbiter1-100617:/exp/b3/gv0
> > > > > Status: Connected
> > > > > Number of entries: 0
> > > > >  
> > > > > Brick tpc-cent-glus1-081017:/exp/b4/gv0
> > > > > <gfid:e0c56bf7-8bfe-46ca-bde1-e46b92d33df3>
> > > > > <gfid:6f0a0549-8669-46de-8823-d6677fdca8e3>
> > > > > <gfid:d0e2fb2a-21b5-4ea8-a578-0801280b2530>
> > > > > <gfid:48bff79c-7bc2-4dc5-8b7f-4401b27fdf5a>
> > > > > <gfid:5902593d-a059-4ec7-b18b-7a2ab5c49a50>
> > > > > <gfid:cb821178-4621-4fcf-90f3-5b5c2ad7f756>
> > > > > <gfid:6aea0805-8dd1-437c-b922-52c9d11e488a>
> > > > > <gfid:f4076a37-2e2f-4d7a-90dd-0a3560a4bdff>
> > > > > <gfid:51ff7386-a550-4971-957c-b42c4d915e9f>
> > > > > <gfid:4309f7b8-3a9d-4bc8-ba2b-799f8a02611b>
> > > > > <gfid:b76746ec-6d7d-4ea3-a001-c96672a4d47e>
> > > > > <gfid:f8de26e7-d17d-41e0-adcd-e7d24ed74ac8>
> > > > > <gfid:8e2c4540-e0b4-4006-bb5d-aacd57f8f21b>
> > > > > <gfid:183ebefb-b827-4cbc-b42b-bfd136d5cabb>
> > > > > <gfid:88d492fe-bfbd-4463-ba55-0582d0ad671b>
> > > > > <gfid:e3a6c068-d48b-44b5-9480-245a69648a9b>
> > > > > <gfid:4aab9c6a-22d2-469a-a688-7b0a8784f4b1>
> > > > > <gfid:c6d182f2-7e46-4502-a0d2-b92824caa4de>
> > > > > <gfid:eb546f93-e9d6-4a59-ac35-6139b5c40919>
> > > > > <gfid:6043e381-7edf-4569-bc37-e27dd13549d2>
> > > > > <gfid:52090dc7-7a3c-40f9-9c54-3395f5158eab>
> > > > > <gfid:ecceee46-4310-421e-b56e-5fe46bd5263c>
> > > > > <gfid:354aea57-4b40-47fc-8ede-1d7e3b7501b4>
> > > > > <gfid:d43284d4-86aa-42ff-98b8-f6340b407d9d>
> > > > > Status: Connected
> > > > > Number of entries: 24
> > > > >  
> > > > > Brick tpc-cent-glus2-081017:/exp/b4/gv0
> > > > > <gfid:e0c56bf7-8bfe-46ca-bde1-e46b92d33df3>
> > > > > <gfid:6f0a0549-8669-46de-8823-d6677fdca8e3>
> > > > > <gfid:d0e2fb2a-21b5-4ea8-a578-0801280b2530>
> > > > > <gfid:48bff79c-7bc2-4dc5-8b7f-4401b27fdf5a>
> > > > > <gfid:5902593d-a059-4ec7-b18b-7a2ab5c49a50>
> > > > > <gfid:cb821178-4621-4fcf-90f3-5b5c2ad7f756>
> > > > > <gfid:6aea0805-8dd1-437c-b922-52c9d11e488a>
> > > > > <gfid:f4076a37-2e2f-4d7a-90dd-0a3560a4bdff>
> > > > > <gfid:51ff7386-a550-4971-957c-b42c4d915e9f>
> > > > > <gfid:4309f7b8-3a9d-4bc8-ba2b-799f8a02611b>
> > > > > <gfid:b76746ec-6d7d-4ea3-a001-c96672a4d47e>
> > > > > <gfid:f8de26e7-d17d-41e0-adcd-e7d24ed74ac8>
> > > > > <gfid:8e2c4540-e0b4-4006-bb5d-aacd57f8f21b>
> > > > > <gfid:183ebefb-b827-4cbc-b42b-bfd136d5cabb>
> > > > > <gfid:88d492fe-bfbd-4463-ba55-0582d0ad671b>
> > > > > <gfid:e3a6c068-d48b-44b5-9480-245a69648a9b>
> > > > > <gfid:4aab9c6a-22d2-469a-a688-7b0a8784f4b1>
> > > > > <gfid:c6d182f2-7e46-4502-a0d2-b92824caa4de>
> > > > > <gfid:eb546f93-e9d6-4a59-ac35-6139b5c40919>
> > > > > <gfid:6043e381-7edf-4569-bc37-e27dd13549d2>
> > > > > <gfid:52090dc7-7a3c-40f9-9c54-3395f5158eab>
> > > > > <gfid:ecceee46-4310-421e-b56e-5fe46bd5263c>
> > > > > <gfid:354aea57-4b40-47fc-8ede-1d7e3b7501b4>
> > > > > <gfid:d43284d4-86aa-42ff-98b8-f6340b407d9d>
> > > > > Status: Connected
> > > > > Number of entries: 24
> > > > >  
> > > > > Brick tpc-arbiter1-100617:/exp/b4/gv0
> > > > > Status: Connected
> > > > > Number of entries: 0
> > > > >  
> > > > > Thank you for your help!
> > > > >  
> > > > > From: Karthik Subrahmanya [mailto:mailto:ksubrahm at
redhat.com]
> > > > >  
> > > > > Sent: Monday, October 16, 2017 10:27 AM
> > > > > To: Matt Waymack <mailto:mwaymack at nsgdv.com>
> > > > > Cc: gluster-users <mailto:Gluster-users at
gluster.org>
> > > > > Subject: Re: [Gluster-users] gfid entries in volume
heal info
> > > > > that do not heal
> > > > >  
> > > > > Hi Matt, 
> > > > >  
> > > > > The files might be in split brain. Could you please
send the
> > > > > outputs of these? 
> > > > > gluster volume info <volname>
> > > > > gluster volume heal <volname> info
> > > > > And also the getfattr output of the files which are in
the
> > > > > heal info output from all the bricks of that replica
pair.
> > > > > getfattr -d -e hex -m . <file path on brick>
> > > > >  
> > > > > Thanks &  Regards
> > > > > Karthik
> > > > >  
> > > > > On 16-Oct-2017 8:16 PM, "Matt Waymack"
<mailto:mwaymack at nsgdv
> > > > > .com> wrote:
> > > > > Hi all,
> > > > >  
> > > > > I have a volume where the output of volume heal info
shows
> > > > > several gfid entries to be healed, but they?ve been
there for
> > > > > weeks and have not healed.  Any normal file that shows
up on
> > > > > the heal info does get healed as expected, but these
gfid
> > > > > entries do not.  Is there any way to remove these
orphaned
> > > > > entries from the volume so they are no longer stuck in
the
> > > > > heal process?
> > > > >  
> > > > > Thank you!
> > > > >  
> > > > > _______________________________________________
> > > > > Gluster-users mailing list
> > > > > mailto:Gluster-users at gluster.org
> > > > > http://lists.gluster.org/mailman/listinfo/gluster-users
> > > > >  
> > > > >  
> > > > > 
> > > > > 
> > > > > _______________________________________________
> > > > > Gluster-users mailing list
> > > > > Gluster-users at gluster.org
> > > > > http://lists.gluster.org/mailman/listinfo/gluster-users
> > > 
> > >  
> > > 
> > > 
> > > 
> > > 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171106/43678f94/attachment.html>