Peter Auyeung
2014-Nov-08 02:49 UTC
[Gluster-users] [Gluster-devel] info heal-failed shown as gfid
I have a node down while gfs still open for writing. Got tons of heal-failed on a replicated volume showing as gfid. Tried gfid-resolver and got the following: # ./gfid-resolver.sh /brick02/gfs/ 88417c43-7d0f-4ec5-8fcd-f696617b5bc1 88417c43-7d0f-4ec5-8fcd-f696617b5bc1 == File: 11/07/14 18:47:19 [ /root/scripts ] Any one has clue how to resolve and fix these heal-failed entries?? # gluster volume heal sas02 info heal-failed Gathering list of heal failed entries on volume sas02 has been successful Brick glusterprod001.bo.shopzilla.sea:/brick02/gfs Number of entries: 0 Brick glusterprod002.bo.shopzilla.sea:/brick02/gfs Number of entries: 1024 at path on brick ----------------------------------- 2014-11-08 01:22:23 <gfid:88417c43-7d0f-4ec5-8fcd-f696617b5bc1> 2014-11-08 01:22:23 <gfid:d0d9e3aa-45ef-4a5c-8ccb-1a5859ff658c> 2014-11-08 01:22:23 <gfid:81a8867c-f0ef-4390-ad84-a4b1b3393842> 2014-11-08 01:22:23 <gfid:f7ed8fd8-f8bc-4c0c-8803-98a8e31ee46e> 2014-11-08 01:22:23 <gfid:df451eec-e3e3-4e9a-9cb4-4991c68f0d62> 2014-11-08 01:22:23 <gfid:4e759749-bf6c-4355-af48-17ca64c20428> 2014-11-08 01:22:23 <gfid:72c5a979-3106-4130-b3a2-32a18cef7a39> 2014-11-08 01:22:23 <gfid:15db8683-733c-4af4-ab97-e35af581c06f> 2014-11-08 01:22:23 <gfid:ff86e3b4-b7ad-46a6-aa97-2437679a4c7f> 2014-11-08 01:22:23 <gfid:574223c0-b705-4672-b387-6c5aa7f63d31> 2014-11-08 01:22:23 <gfid:ca522f4f-3fc7-4dfa-823f-888e2269085e> 2014-11-08 01:22:23 <gfid:56feab84-b4fa-414a-8173-1472913ed50c> 2014-11-08 01:22:23 <gfid:c66bfc5e-8631-4422-aca4-c9f8f99c2ef2> 2014-11-08 01:22:23 <gfid:69a20ce2-27dc-41fa-9883-83c1a2842ba8> 2014-11-08 01:22:23 <gfid:1ab58d44-570e-4ac2-82b6-568dcfe9e530> 2014-11-08 01:22:23 <gfid:face55f7-e8dc-48e9-a3fb-1aa126f0e829> 2014-11-08 01:22:23 <gfid:a8fb721d-aaae-4bef-8fcc-d7da950a8c2f> 2014-11-08 01:22:23 <gfid:45a35318-d208-4bb2-9826-3b37083ae4ab> 2014-11-08 01:22:23 <gfid:d8174f82-15b2-4318-8953-2873c38bb628> 2014-11-08 01:22:23 <gfid:689c26f7-2442-4c84-be98-e75a81d916ac> 2014-11-08 01:22:23 <gfid:acd5e0b4-e12b-4af2-878e-1d197ef90061> 2014-11-08 01:22:23 <gfid:90260d6f-cfe2-4949-a7e1-15c1cc3a914d> 2014-11-08 01:22:23 <gfid:5bdd3c63-1873-4cfc-89c3-7ac21b9bb684> 2014-11-08 01:22:23 <gfid:48b3f29b-fd3c-4147-b849-bac5ec3dd22b> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141108/4fe80227/attachment.html>
Vijay Bellur
2014-Nov-10 05:51 UTC
[Gluster-users] [Gluster-devel] info heal-failed shown as gfid
On 11/08/2014 08:19 AM, Peter Auyeung wrote:> I have a node down while gfs still open for writing. > > Got tons of heal-failed on a replicated volume showing as gfid. > > Tried gfid-resolver and got the following: > > # ./gfid-resolver.sh /brick02/gfs/ 88417c43-7d0f-4ec5-8fcd-f696617b5bc1 > 88417c43-7d0f-4ec5-8fcd-f696617b5bc1 == File: 11/07/14 18:47:19 > [ /root/scripts ] > > Any one has clue how to resolve and fix these heal-failed entries?? >Procedure detailed in [1] might be useful. -Vijay [1] https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md
Joe Julian
2014-Nov-10 06:50 UTC
[Gluster-users] [Gluster-devel] info heal-failed shown as gfid
Heal-failed can be for any reason that's not defined as split brain. The only place I've been able to find clues is in the log files. Look at the timestamp on the heal-failed output and match it to log entries in glustershd logs. On November 7, 2014 6:49:36 PM PST, Peter Auyeung <pauyeung at connexity.com> wrote:>I have a node down while gfs still open for writing. > >Got tons of heal-failed on a replicated volume showing as gfid. > >Tried gfid-resolver and got the following: > ># ./gfid-resolver.sh /brick02/gfs/ 88417c43-7d0f-4ec5-8fcd-f696617b5bc1 >88417c43-7d0f-4ec5-8fcd-f696617b5bc1 == File: 11/07/14 >18:47:19 [ /root/scripts ] > >Any one has clue how to resolve and fix these heal-failed entries?? > > ># gluster volume heal sas02 info heal-failed >Gathering list of heal failed entries on volume sas02 has been >successful > >Brick glusterprod001.bo.shopzilla.sea:/brick02/gfs >Number of entries: 0 > >Brick glusterprod002.bo.shopzilla.sea:/brick02/gfs >Number of entries: 1024 >at path on brick >----------------------------------- >2014-11-08 01:22:23 <gfid:88417c43-7d0f-4ec5-8fcd-f696617b5bc1> >2014-11-08 01:22:23 <gfid:d0d9e3aa-45ef-4a5c-8ccb-1a5859ff658c> >2014-11-08 01:22:23 <gfid:81a8867c-f0ef-4390-ad84-a4b1b3393842> >2014-11-08 01:22:23 <gfid:f7ed8fd8-f8bc-4c0c-8803-98a8e31ee46e> >2014-11-08 01:22:23 <gfid:df451eec-e3e3-4e9a-9cb4-4991c68f0d62> >2014-11-08 01:22:23 <gfid:4e759749-bf6c-4355-af48-17ca64c20428> >2014-11-08 01:22:23 <gfid:72c5a979-3106-4130-b3a2-32a18cef7a39> >2014-11-08 01:22:23 <gfid:15db8683-733c-4af4-ab97-e35af581c06f> >2014-11-08 01:22:23 <gfid:ff86e3b4-b7ad-46a6-aa97-2437679a4c7f> >2014-11-08 01:22:23 <gfid:574223c0-b705-4672-b387-6c5aa7f63d31> >2014-11-08 01:22:23 <gfid:ca522f4f-3fc7-4dfa-823f-888e2269085e> >2014-11-08 01:22:23 <gfid:56feab84-b4fa-414a-8173-1472913ed50c> >2014-11-08 01:22:23 <gfid:c66bfc5e-8631-4422-aca4-c9f8f99c2ef2> >2014-11-08 01:22:23 <gfid:69a20ce2-27dc-41fa-9883-83c1a2842ba8> >2014-11-08 01:22:23 <gfid:1ab58d44-570e-4ac2-82b6-568dcfe9e530> >2014-11-08 01:22:23 <gfid:face55f7-e8dc-48e9-a3fb-1aa126f0e829> >2014-11-08 01:22:23 <gfid:a8fb721d-aaae-4bef-8fcc-d7da950a8c2f> >2014-11-08 01:22:23 <gfid:45a35318-d208-4bb2-9826-3b37083ae4ab> >2014-11-08 01:22:23 <gfid:d8174f82-15b2-4318-8953-2873c38bb628> >2014-11-08 01:22:23 <gfid:689c26f7-2442-4c84-be98-e75a81d916ac> >2014-11-08 01:22:23 <gfid:acd5e0b4-e12b-4af2-878e-1d197ef90061> >2014-11-08 01:22:23 <gfid:90260d6f-cfe2-4949-a7e1-15c1cc3a914d> >2014-11-08 01:22:23 <gfid:5bdd3c63-1873-4cfc-89c3-7ac21b9bb684> >2014-11-08 01:22:23 <gfid:48b3f29b-fd3c-4147-b849-bac5ec3dd22b> > > >------------------------------------------------------------------------ > >_______________________________________________ >Gluster-devel mailing list >Gluster-devel at gluster.org >http://supercolony.gluster.org/mailman/listinfo/gluster-devel-- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141109/4314da97/attachment.html>