thr3ads.net - Gluster users - [Gluster-users] gfid entries in volume heal info that do not heal [Oct 2017]

If this information is useful, please help other people find it:
Share via:

Karthik Subrahmanya

2017-Oct-16 15:27 UTC

[Gluster-users] gfid entries in volume heal info that do not heal

Hi Matt,

The files might be in split brain. Could you please send the outputs of
these?
gluster volume info <volname>
gluster volume heal <volname> info
And also the getfattr output of the files which are in the heal info output
from all the bricks of that replica pair.
getfattr -d -e hex -m . <file path on brick>

Thanks &  Regards
Karthik

On 16-Oct-2017 8:16 PM, "Matt Waymack" <mwaymack at nsgdv.com>
wrote:

Hi all,



I have a volume where the output of volume heal info shows several gfid
entries to be healed, but they?ve been there for weeks and have not
healed.  Any normal file that shows up on the heal info does get healed as
expected, but these gfid entries do not.  Is there any way to remove these
orphaned entries from the volume so they are no longer stuck in the heal
process?



Thank you!

_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171016/06fa261c/attachment.html>

Matt Waymack

2017-Oct-16 16:21 UTC

head link

[Gluster-users] gfid entries in volume heal info that do not heal

OK, so here?s my output of the volume info and the heal info. I have not yet
tracked down physical location of these files, any tips to finding them would be
appreciated, but I?m definitely just wanting them gone.  I forgot to mention
earlier that the cluster is running 3.12 and was upgraded from 3.10; these files
were likely stuck like this when it was on 3.10.

[root at tpc-cent-glus1-081017 ~]# gluster volume info gv0

Volume Name: gv0
Type: Distributed-Replicate
Volume ID: 8f07894d-e3ab-4a65-bda1-9d9dd46db007
Status: Started
Snapshot Count: 0
Number of Bricks: 4 x (2 + 1) = 12
Transport-type: tcp
Bricks:
Brick1: tpc-cent-glus1-081017:/exp/b1/gv0
Brick2: tpc-cent-glus2-081017:/exp/b1/gv0
Brick3: tpc-arbiter1-100617:/exp/b1/gv0 (arbiter)
Brick4: tpc-cent-glus1-081017:/exp/b2/gv0
Brick5: tpc-cent-glus2-081017:/exp/b2/gv0
Brick6: tpc-arbiter1-100617:/exp/b2/gv0 (arbiter)
Brick7: tpc-cent-glus1-081017:/exp/b3/gv0
Brick8: tpc-cent-glus2-081017:/exp/b3/gv0
Brick9: tpc-arbiter1-100617:/exp/b3/gv0 (arbiter)
Brick10: tpc-cent-glus1-081017:/exp/b4/gv0
Brick11: tpc-cent-glus2-081017:/exp/b4/gv0
Brick12: tpc-arbiter1-100617:/exp/b4/gv0 (arbiter)
Options Reconfigured:
nfs.disable: on
transport.address-family: inet

[root at tpc-cent-glus1-081017 ~]# gluster volume heal gv0 info
Brick tpc-cent-glus1-081017:/exp/b1/gv0
<gfid:108694db-c039-4b7c-bd3d-ad6a15d811a2>
<gfid:6d5ade20-8996-4de2-95d5-20ef98004742>
<gfid:bc6cdc3d-5c46-4597-a7eb-282b21e9bdd5>
<gfid:3c2ff4d1-3662-4214-8f21-f8f47dbdbf06>
<gfid:053e2fb1-bc89-476e-a529-90dffa39963c>

<removed to save scrolling>

Status: Connected
Number of entries: 118

Brick tpc-cent-glus2-081017:/exp/b1/gv0
<gfid:108694db-c039-4b7c-bd3d-ad6a15d811a2>
<gfid:6d5ade20-8996-4de2-95d5-20ef98004742>
<gfid:bc6cdc3d-5c46-4597-a7eb-282b21e9bdd5>
<gfid:3c2ff4d1-3662-4214-8f21-f8f47dbdbf06>
<gfid:053e2fb1-bc89-476e-a529-90dffa39963c>

<removed to save scrolling>

Status: Connected
Number of entries: 118

Brick tpc-arbiter1-100617:/exp/b1/gv0
Status: Connected
Number of entries: 0

Brick tpc-cent-glus1-081017:/exp/b2/gv0
Status: Connected
Number of entries: 0

Brick tpc-cent-glus2-081017:/exp/b2/gv0
Status: Connected
Number of entries: 0

Brick tpc-arbiter1-100617:/exp/b2/gv0
Status: Connected
Number of entries: 0

Brick tpc-cent-glus1-081017:/exp/b3/gv0
Status: Connected
Number of entries: 0

Brick tpc-cent-glus2-081017:/exp/b3/gv0
Status: Connected
Number of entries: 0

Brick tpc-arbiter1-100617:/exp/b3/gv0
Status: Connected
Number of entries: 0

Brick tpc-cent-glus1-081017:/exp/b4/gv0
<gfid:e0c56bf7-8bfe-46ca-bde1-e46b92d33df3>
<gfid:6f0a0549-8669-46de-8823-d6677fdca8e3>
<gfid:d0e2fb2a-21b5-4ea8-a578-0801280b2530>
<gfid:48bff79c-7bc2-4dc5-8b7f-4401b27fdf5a>
<gfid:5902593d-a059-4ec7-b18b-7a2ab5c49a50>
<gfid:cb821178-4621-4fcf-90f3-5b5c2ad7f756>
<gfid:6aea0805-8dd1-437c-b922-52c9d11e488a>
<gfid:f4076a37-2e2f-4d7a-90dd-0a3560a4bdff>
<gfid:51ff7386-a550-4971-957c-b42c4d915e9f>
<gfid:4309f7b8-3a9d-4bc8-ba2b-799f8a02611b>
<gfid:b76746ec-6d7d-4ea3-a001-c96672a4d47e>
<gfid:f8de26e7-d17d-41e0-adcd-e7d24ed74ac8>
<gfid:8e2c4540-e0b4-4006-bb5d-aacd57f8f21b>
<gfid:183ebefb-b827-4cbc-b42b-bfd136d5cabb>
<gfid:88d492fe-bfbd-4463-ba55-0582d0ad671b>
<gfid:e3a6c068-d48b-44b5-9480-245a69648a9b>
<gfid:4aab9c6a-22d2-469a-a688-7b0a8784f4b1>
<gfid:c6d182f2-7e46-4502-a0d2-b92824caa4de>
<gfid:eb546f93-e9d6-4a59-ac35-6139b5c40919>
<gfid:6043e381-7edf-4569-bc37-e27dd13549d2>
<gfid:52090dc7-7a3c-40f9-9c54-3395f5158eab>
<gfid:ecceee46-4310-421e-b56e-5fe46bd5263c>
<gfid:354aea57-4b40-47fc-8ede-1d7e3b7501b4>
<gfid:d43284d4-86aa-42ff-98b8-f6340b407d9d>
Status: Connected
Number of entries: 24

Brick tpc-cent-glus2-081017:/exp/b4/gv0
<gfid:e0c56bf7-8bfe-46ca-bde1-e46b92d33df3>
<gfid:6f0a0549-8669-46de-8823-d6677fdca8e3>
<gfid:d0e2fb2a-21b5-4ea8-a578-0801280b2530>
<gfid:48bff79c-7bc2-4dc5-8b7f-4401b27fdf5a>
<gfid:5902593d-a059-4ec7-b18b-7a2ab5c49a50>
<gfid:cb821178-4621-4fcf-90f3-5b5c2ad7f756>
<gfid:6aea0805-8dd1-437c-b922-52c9d11e488a>
<gfid:f4076a37-2e2f-4d7a-90dd-0a3560a4bdff>
<gfid:51ff7386-a550-4971-957c-b42c4d915e9f>
<gfid:4309f7b8-3a9d-4bc8-ba2b-799f8a02611b>
<gfid:b76746ec-6d7d-4ea3-a001-c96672a4d47e>
<gfid:f8de26e7-d17d-41e0-adcd-e7d24ed74ac8>
<gfid:8e2c4540-e0b4-4006-bb5d-aacd57f8f21b>
<gfid:183ebefb-b827-4cbc-b42b-bfd136d5cabb>
<gfid:88d492fe-bfbd-4463-ba55-0582d0ad671b>
<gfid:e3a6c068-d48b-44b5-9480-245a69648a9b>
<gfid:4aab9c6a-22d2-469a-a688-7b0a8784f4b1>
<gfid:c6d182f2-7e46-4502-a0d2-b92824caa4de>
<gfid:eb546f93-e9d6-4a59-ac35-6139b5c40919>
<gfid:6043e381-7edf-4569-bc37-e27dd13549d2>
<gfid:52090dc7-7a3c-40f9-9c54-3395f5158eab>
<gfid:ecceee46-4310-421e-b56e-5fe46bd5263c>
<gfid:354aea57-4b40-47fc-8ede-1d7e3b7501b4>
<gfid:d43284d4-86aa-42ff-98b8-f6340b407d9d>
Status: Connected
Number of entries: 24

Brick tpc-arbiter1-100617:/exp/b4/gv0
Status: Connected
Number of entries: 0

Thank you for your help!

From: Karthik Subrahmanya [mailto:ksubrahm at redhat.com]
Sent: Monday, October 16, 2017 10:27 AM
To: Matt Waymack <mwaymack at nsgdv.com>
Cc: gluster-users <Gluster-users at gluster.org>
Subject: Re: [Gluster-users] gfid entries in volume heal info that do not heal

Hi Matt,

The files might be in split brain. Could you please send the outputs of these?
gluster volume info <volname>
gluster volume heal <volname> info
And also the getfattr output of the files which are in the heal info output from
all the bricks of that replica pair.
getfattr -d -e hex -m . <file path on brick>

Thanks &  Regards
Karthik

On 16-Oct-2017 8:16 PM, "Matt Waymack" <mwaymack at
nsgdv.com<mailto:mwaymack at nsgdv.com>> wrote:
Hi all,

I have a volume where the output of volume heal info shows several gfid entries
to be healed, but they?ve been there for weeks and have not healed.  Any normal
file that shows up on the heal info does get healed as expected, but these gfid
entries do not.  Is there any way to remove these orphaned entries from the
volume so they are no longer stuck in the heal process?

Thank you!

_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171016/1a1d8f74/attachment.html>

Karthik Subrahmanya

2017-Oct-17 06:26 UTC

head link

[Gluster-users] gfid entries in volume heal info that do not heal

Hi Matt,

Run these commands on all the bricks of the replica pair to get the attrs
set on the backend.

On the bricks of first replica set:
getfattr -d -e hex -m . <brick path>/.glusterfs/10/86/
108694db-c039-4b7c-bd3d-ad6a15d811a2

On the fourth replica set:
getfattr -d -e hex -m . <brick path>/.glusterfs/
e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3

Also run the "gluster volume heal <volname>" once and send the
shd log.
And the output of "gluster volume heal <volname> info
split-brain"

Regards,
Karthik

On Mon, Oct 16, 2017 at 9:51 PM, Matt Waymack <mwaymack at nsgdv.com>
wrote:
> OK, so here?s my output of the volume info and the heal info. I have not
> yet tracked down physical location of these files, any tips to finding them
> would be appreciated, but I?m definitely just wanting them gone.  I forgot
> to mention earlier that the cluster is running 3.12 and was upgraded from
> 3.10; these files were likely stuck like this when it was on 3.10.
>
>
>
> [root at tpc-cent-glus1-081017 ~]# gluster volume info gv0
>
>
>
> Volume Name: gv0
>
> Type: Distributed-Replicate
>
> Volume ID: 8f07894d-e3ab-4a65-bda1-9d9dd46db007
>
> Status: Started
>
> Snapshot Count: 0
>
> Number of Bricks: 4 x (2 + 1) = 12
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: tpc-cent-glus1-081017:/exp/b1/gv0
>
> Brick2: tpc-cent-glus2-081017:/exp/b1/gv0
>
> Brick3: tpc-arbiter1-100617:/exp/b1/gv0 (arbiter)
>
> Brick4: tpc-cent-glus1-081017:/exp/b2/gv0
>
> Brick5: tpc-cent-glus2-081017:/exp/b2/gv0
>
> Brick6: tpc-arbiter1-100617:/exp/b2/gv0 (arbiter)
>
> Brick7: tpc-cent-glus1-081017:/exp/b3/gv0
>
> Brick8: tpc-cent-glus2-081017:/exp/b3/gv0
>
> Brick9: tpc-arbiter1-100617:/exp/b3/gv0 (arbiter)
>
> Brick10: tpc-cent-glus1-081017:/exp/b4/gv0
>
> Brick11: tpc-cent-glus2-081017:/exp/b4/gv0
>
> Brick12: tpc-arbiter1-100617:/exp/b4/gv0 (arbiter)
>
> Options Reconfigured:
>
> nfs.disable: on
>
> transport.address-family: inet
>
>
>
> [root at tpc-cent-glus1-081017 ~]# gluster volume heal gv0 info
>
> Brick tpc-cent-glus1-081017:/exp/b1/gv0
>
> <gfid:108694db-c039-4b7c-bd3d-ad6a15d811a2>
>
> <gfid:6d5ade20-8996-4de2-95d5-20ef98004742>
>
> <gfid:bc6cdc3d-5c46-4597-a7eb-282b21e9bdd5>
>
> <gfid:3c2ff4d1-3662-4214-8f21-f8f47dbdbf06>
>
> <gfid:053e2fb1-bc89-476e-a529-90dffa39963c>
>
>
>
> <removed to save scrolling>
>
>
>
> Status: Connected
>
> Number of entries: 118
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b1/gv0
>
> <gfid:108694db-c039-4b7c-bd3d-ad6a15d811a2>
>
> <gfid:6d5ade20-8996-4de2-95d5-20ef98004742>
>
> <gfid:bc6cdc3d-5c46-4597-a7eb-282b21e9bdd5>
>
> <gfid:3c2ff4d1-3662-4214-8f21-f8f47dbdbf06>
>
> <gfid:053e2fb1-bc89-476e-a529-90dffa39963c>
>
>
>
> <removed to save scrolling>
>
>
>
> Status: Connected
>
> Number of entries: 118
>
>
>
> Brick tpc-arbiter1-100617:/exp/b1/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-cent-glus1-081017:/exp/b2/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b2/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-arbiter1-100617:/exp/b2/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-cent-glus1-081017:/exp/b3/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b3/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-arbiter1-100617:/exp/b3/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick tpc-cent-glus1-081017:/exp/b4/gv0
>
> <gfid:e0c56bf7-8bfe-46ca-bde1-e46b92d33df3>
>
> <gfid:6f0a0549-8669-46de-8823-d6677fdca8e3>
>
> <gfid:d0e2fb2a-21b5-4ea8-a578-0801280b2530>
>
> <gfid:48bff79c-7bc2-4dc5-8b7f-4401b27fdf5a>
>
> <gfid:5902593d-a059-4ec7-b18b-7a2ab5c49a50>
>
> <gfid:cb821178-4621-4fcf-90f3-5b5c2ad7f756>
>
> <gfid:6aea0805-8dd1-437c-b922-52c9d11e488a>
>
> <gfid:f4076a37-2e2f-4d7a-90dd-0a3560a4bdff>
>
> <gfid:51ff7386-a550-4971-957c-b42c4d915e9f>
>
> <gfid:4309f7b8-3a9d-4bc8-ba2b-799f8a02611b>
>
> <gfid:b76746ec-6d7d-4ea3-a001-c96672a4d47e>
>
> <gfid:f8de26e7-d17d-41e0-adcd-e7d24ed74ac8>
>
> <gfid:8e2c4540-e0b4-4006-bb5d-aacd57f8f21b>
>
> <gfid:183ebefb-b827-4cbc-b42b-bfd136d5cabb>
>
> <gfid:88d492fe-bfbd-4463-ba55-0582d0ad671b>
>
> <gfid:e3a6c068-d48b-44b5-9480-245a69648a9b>
>
> <gfid:4aab9c6a-22d2-469a-a688-7b0a8784f4b1>
>
> <gfid:c6d182f2-7e46-4502-a0d2-b92824caa4de>
>
> <gfid:eb546f93-e9d6-4a59-ac35-6139b5c40919>
>
> <gfid:6043e381-7edf-4569-bc37-e27dd13549d2>
>
> <gfid:52090dc7-7a3c-40f9-9c54-3395f5158eab>
>
> <gfid:ecceee46-4310-421e-b56e-5fe46bd5263c>
>
> <gfid:354aea57-4b40-47fc-8ede-1d7e3b7501b4>
>
> <gfid:d43284d4-86aa-42ff-98b8-f6340b407d9d>
>
> Status: Connected
>
> Number of entries: 24
>
>
>
> Brick tpc-cent-glus2-081017:/exp/b4/gv0
>
> <gfid:e0c56bf7-8bfe-46ca-bde1-e46b92d33df3>
>
> <gfid:6f0a0549-8669-46de-8823-d6677fdca8e3>
>
> <gfid:d0e2fb2a-21b5-4ea8-a578-0801280b2530>
>
> <gfid:48bff79c-7bc2-4dc5-8b7f-4401b27fdf5a>
>
> <gfid:5902593d-a059-4ec7-b18b-7a2ab5c49a50>
>
> <gfid:cb821178-4621-4fcf-90f3-5b5c2ad7f756>
>
> <gfid:6aea0805-8dd1-437c-b922-52c9d11e488a>
>
> <gfid:f4076a37-2e2f-4d7a-90dd-0a3560a4bdff>
>
> <gfid:51ff7386-a550-4971-957c-b42c4d915e9f>
>
> <gfid:4309f7b8-3a9d-4bc8-ba2b-799f8a02611b>
>
> <gfid:b76746ec-6d7d-4ea3-a001-c96672a4d47e>
>
> <gfid:f8de26e7-d17d-41e0-adcd-e7d24ed74ac8>
>
> <gfid:8e2c4540-e0b4-4006-bb5d-aacd57f8f21b>
>
> <gfid:183ebefb-b827-4cbc-b42b-bfd136d5cabb>
>
> <gfid:88d492fe-bfbd-4463-ba55-0582d0ad671b>
>
> <gfid:e3a6c068-d48b-44b5-9480-245a69648a9b>
>
> <gfid:4aab9c6a-22d2-469a-a688-7b0a8784f4b1>
>
> <gfid:c6d182f2-7e46-4502-a0d2-b92824caa4de>
>
> <gfid:eb546f93-e9d6-4a59-ac35-6139b5c40919>
>
> <gfid:6043e381-7edf-4569-bc37-e27dd13549d2>
>
> <gfid:52090dc7-7a3c-40f9-9c54-3395f5158eab>
>
> <gfid:ecceee46-4310-421e-b56e-5fe46bd5263c>
>
> <gfid:354aea57-4b40-47fc-8ede-1d7e3b7501b4>
>
> <gfid:d43284d4-86aa-42ff-98b8-f6340b407d9d>
>
> Status: Connected
>
> Number of entries: 24
>
>
>
> Brick tpc-arbiter1-100617:/exp/b4/gv0
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Thank you for your help!
>
>
>
> *From:* Karthik Subrahmanya [mailto:ksubrahm at redhat.com]
> *Sent:* Monday, October 16, 2017 10:27 AM
> *To:* Matt Waymack <mwaymack at nsgdv.com>
> *Cc:* gluster-users <Gluster-users at gluster.org>
> *Subject:* Re: [Gluster-users] gfid entries in volume heal info that do
> not heal
>
>
>
> Hi Matt,
>
>
>
> The files might be in split brain. Could you please send the outputs of
> these?
>
> gluster volume info <volname>
>
> gluster volume heal <volname> info
>
> And also the getfattr output of the files which are in the heal info
> output from all the bricks of that replica pair.
>
> getfattr -d -e hex -m . <file path on brick>
>
>
>
> Thanks &  Regards
>
> Karthik
>
>
>
> On 16-Oct-2017 8:16 PM, "Matt Waymack" <mwaymack at
nsgdv.com> wrote:
>
> Hi all,
>
>
>
> I have a volume where the output of volume heal info shows several gfid
> entries to be healed, but they?ve been there for weeks and have not
> healed.  Any normal file that shows up on the heal info does get healed as
> expected, but these gfid entries do not.  Is there any way to remove these
> orphaned entries from the volume so they are no longer stuck in the heal
> process?
>
>
>
> Thank you!
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171017/8f6f4777/attachment.html>

Reasonably Related Threads

Search for more apparently analagous threads

Gluster users - Oct 2017 - gfid entries in volume heal info that do not heal

[Gluster-users] gfid entries in volume heal info that do not heal

[Gluster-users] gfid entries in volume heal info that do not heal

[Gluster-users] gfid entries in volume heal info that do not heal

Reasonably Related Threads