Joe Julian
2022-Aug-12 16:27 UTC
[Gluster-users] Directory in split brain does not heal - Gfs 9.2
It could work, but I never imagined, back then, that *directories* could get in split-brain. The most likely reason for that split is that there's a gfid mismatch on one of the replicas. I'd go to the brick with the odd gfid, move that directory out of the brick path, then do a "find folder" on the client mount to rebuild the directory structure. Check the directory to make sure all the files are right before deleting the moved odd one. If you need to fix anything, just copy from the moved directory to a client-mount on the same machine. On 8/12/22 8:12 AM, Ilias Chasapakis forumZFD wrote:> > Dear fellow gluster users, > > we are facing a problem with our replica 3 setup. Glusterfs version is > 9.2. > > We have a problem with a directory that is in split-brain and we > cannot manage to heal with: > >> gluster volume heal gfsVol split-brain latest-mtime /folder >> > The command throws the following error: "failed:Transport endpoint is > not connected." > > So the split brain directory entry remains and and so the whole > healing process is not completing and other entries get stuck. > > I saw there is a python script available > https://github.com/joejulian/glusterfs-splitbrain > <https://github.com/joejulian/glusterfs-splitbrain> Would that be a > good solution to try? To be honest we are a bit concerned with > deleting the gfid and the files from the brick manually as it seems it > can create inconsistencies and break things... I can of course give > you more information about our setup and situation, but if you already > have some tip, that would be fantastic. > > Best regards > > Ilias > > -- > ?forumZFD > Entschieden f?r Frieden | Committed to Peace > > Ilias Chasapakis > Referent IT | IT Consultant > > Forum Ziviler Friedensdienst e.V. | Forum Civil Peace Service > Am K?lner Brett 8 | 50825 K?ln | Germany > > Tel 0221 91273243 | Fax 0221 91273299 |http://www.forumZFD.de > > Vorstand nach ? 26 BGB, einzelvertretungsberechtigt | Executive Board: > Oliver Knabe (Vorsitz | Chair), Jens von Bargen, Alexander Mauz > VR 17651 Amtsgericht K?ln > > Spenden | Donations: IBAN DE37 3702 0500 0008 2401 01 BIC BFSWDE33XXX > > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge:https://meet.google.com/cpu-eiue-hvk > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20220812/9136bef6/attachment.html>
Péter Károly JUHÁSZ
2022-Aug-12 17:25 UTC
[Gluster-users] Directory in split brain does not heal - Gfs 9.2
This always helped for me in this kind of situations: http://docs.gluster.org/Troubleshooting/resolving-splitbrain/ Joe Julian <joe at julianfamily.org> ? 2022?8?12??? 18:33???> It could work, but I never imagined, back then, that *directories* could > get in split-brain. > > The most likely reason for that split is that there's a gfid mismatch on > one of the replicas. I'd go to the brick with the odd gfid, move that > directory out of the brick path, then do a "find folder" on the client > mount to rebuild the directory structure. Check the directory to make sure > all the files are right before deleting the moved odd one. > > If you need to fix anything, just copy from the moved directory to a > client-mount on the same machine. > On 8/12/22 8:12 AM, Ilias Chasapakis forumZFD wrote: > > Dear fellow gluster users, > > we are facing a problem with our replica 3 setup. Glusterfs version is 9.2. > > We have a problem with a directory that is in split-brain and we cannot > manage to heal with: > > gluster volume heal gfsVol split-brain latest-mtime /folder > > The command throws the following error: "failed:Transport endpoint is not > connected." > > So the split brain directory entry remains and and so the whole healing > process is not completing and other entries get stuck. > > I saw there is a python script available > https://github.com/joejulian/glusterfs-splitbrain Would that be a good > solution to try? To be honest we are a bit concerned with deleting the gfid > and the files from the brick manually as it seems it can create > inconsistencies and break things... I can of course give you more > information about our setup and situation, but if you already have some > tip, that would be fantastic. > > Best regards > > Ilias > > -- > ?forumZFD > Entschieden f?r Frieden | Committed to Peace > > Ilias Chasapakis > Referent IT | IT Consultant > > Forum Ziviler Friedensdienst e.V. | Forum Civil Peace Service > Am K?lner Brett 8 | 50825 K?ln | Germany > > Tel 0221 91273243 | Fax 0221 91273299 | http://www.forumZFD.de > > Vorstand nach ? 26 BGB, einzelvertretungsberechtigt | Executive Board: > Oliver Knabe (Vorsitz | Chair), Jens von Bargen, Alexander Mauz > VR 17651 Amtsgericht K?ln > > Spenden | Donations: IBAN DE37 3702 0500 0008 2401 01 BIC BFSWDE33XXX > > > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://meet.google.com/cpu-eiue-hvk > Gluster-users mailing listGluster-users at gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users > > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://meet.google.com/cpu-eiue-hvk > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20220812/1a3d11d2/attachment.html>