Hello Ravi, thanks a lot for your reply. The Data on ovirt-node03 is the one which i want. Here are the infos collected by following the howto: https://github.com/GlusterFS/glusterfs/blob/master/doc/debugging/split-brain.md [root at ovirt-node03 ~]# gluster volume heal RaidVolB info split-brain Gathering list of split brain entries on volume RaidVolB has been successful Brick ovirt-node03.example.local:/raidvol/volb/brick Number of entries: 0 Brick ovirt-node04.example.local:/raidvol/volb/brick Number of entries: 14 at path on brick ----------------------------------- 2015-01-27 17:33:00 <gfid:1c15d0cb-1cca-4627-841c-395f7b712f73> 2015-01-27 17:34:01 <gfid:1c15d0cb-1cca-4627-841c-395f7b712f73> 2015-01-27 17:35:04 <gfid:1c15d0cb-1cca-4627-841c-395f7b712f73> 2015-01-27 17:36:05 <gfid:cd411b57-6078-4f3c-80d1-0ac1455186a6>/ids 2015-01-27 17:37:06 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids 2015-01-27 17:37:07 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids 2015-01-27 17:38:08 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids 2015-01-27 17:38:21 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids 2015-01-27 17:39:22 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids 2015-01-27 17:40:23 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids 2015-01-27 17:41:24 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids 2015-01-27 17:42:25 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids 2015-01-27 17:43:26 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids 2015-01-27 17:44:27 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids [root at ovirt-node03 ~]# gluster volume heal RaidVolB info Brick ovirt-node03.example.local:/raidvol/volb/brick/ Number of entries: 0 Brick ovirt-node04.example.local:/raidvol/volb/brick/ Number of entries: 0 [root at ovirt-node03 ~]# getfattr -d -m . -e hex /raidvol/volb/brick/1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids getfattr: Removing leading '/' from absolute path names # file: raidvol/volb/brick/1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids trusted.afr.RaidVolB-client-0=0x000000000000000000000000 trusted.afr.RaidVolB-client-1=0x000000000000000000000000 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0x1c15d0cb1cca4627841c395f7b712f73 The "Resetting the relevant changelogs to resolve the split-brain: " part of the howto is now a little complictaed. Do i have a data or meta split brain now? I guess i have a data split brain in my case, right? What are my next setfattr commands nowin my case if i want to keep the data from node03? Thanks a lot! Mario On Wed, Jan 28, 2015 at 9:44 AM, Ravishankar N <ravishankar at redhat.com> wrote:> > On 01/28/2015 02:02 PM, Ml Ml wrote: >> >> I want to either take the file from node03 or node04. i really don?t >> >mind. Can i not just tell gluster that it should use one node as the >> >?current? one? > > Policy based split-brain resolution [1] which does just that, has been > merged in master and should be available in glusterfs 3.7. > For the moment, you would have to modify the xattrs on the one of the bricks > and trigger heal. You can see > https://github.com/GlusterFS/glusterfs/blob/master/doc/debugging/split-brain.md > on how to do it. > > Hope this helps, > Ravi > > [1] http://review.gluster.org/#/c/9377/ >
Ravishankar N
2015-Jan-28 15:57 UTC
[Gluster-users] ... i was able to produce a split brain...
On 01/28/2015 08:34 PM, Ml Ml wrote:> Hello Ravi, > > thanks a lot for your reply. > > The Data on ovirt-node03 is the one which i want. > > Here are the infos collected by following the howto: > https://github.com/GlusterFS/glusterfs/blob/master/doc/debugging/split-brain.md > > > > [root at ovirt-node03 ~]# gluster volume heal RaidVolB info split-brain > Gathering list of split brain entries on volume RaidVolB has been successful > > Brick ovirt-node03.example.local:/raidvol/volb/brick > Number of entries: 0 > > Brick ovirt-node04.example.local:/raidvol/volb/brick > Number of entries: 14 > at path on brick > ----------------------------------- > 2015-01-27 17:33:00 <gfid:1c15d0cb-1cca-4627-841c-395f7b712f73> > 2015-01-27 17:34:01 <gfid:1c15d0cb-1cca-4627-841c-395f7b712f73> > 2015-01-27 17:35:04 <gfid:1c15d0cb-1cca-4627-841c-395f7b712f73> > 2015-01-27 17:36:05 <gfid:cd411b57-6078-4f3c-80d1-0ac1455186a6>/ids > 2015-01-27 17:37:06 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids > 2015-01-27 17:37:07 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids > 2015-01-27 17:38:08 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids > 2015-01-27 17:38:21 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids > 2015-01-27 17:39:22 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids > 2015-01-27 17:40:23 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids > 2015-01-27 17:41:24 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids > 2015-01-27 17:42:25 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids > 2015-01-27 17:43:26 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids > 2015-01-27 17:44:27 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids > > [root at ovirt-node03 ~]# gluster volume heal RaidVolB info > Brick ovirt-node03.example.local:/raidvol/volb/brick/ > Number of entries: 0 > > Brick ovirt-node04.example.local:/raidvol/volb/brick/ > Number of entries: 0Hi Mario, Is "/1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids" a file or a directory? Whatever it is, it should be shown in the output of heal info /heal info split-brain command of both nodes. But I see it being listed only under node03. Also, heal info is showing zero entries for both nodes which is strange. Are node03 and node04 bricks of the same replica pair? Can you share 'gluster volume info` of RaidVolB? How did you infer that there is a split-brain? Does accessing the file(s) from the mount give input/output error?> > [root at ovirt-node03 ~]# getfattr -d -m . -e hex > /raidvol/volb/brick/1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids > getfattr: Removing leading '/' from absolute path names > # file: raidvol/volb/brick/1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids > trusted.afr.RaidVolB-client-0=0x000000000000000000000000 > trusted.afr.RaidVolB-client-1=0x000000000000000000000000 > trusted.afr.dirty=0x000000000000000000000000 > trusted.gfid=0x1c15d0cb1cca4627841c395f7b712f73What is the getfattr output of this file on the other brick? The afr specific xattrs being all zeros certainly don't indicate the possibility of a split-brain> > The "Resetting the relevant changelogs to resolve the split-brain: " > part of the howto is now a little complictaed. Do i have a data or > meta split brain now? > I guess i have a data split brain in my case, right? > > What are my next setfattr commands nowin my case if i want to keep the > data from node03? > > Thanks a lot! > > Mario > > > On Wed, Jan 28, 2015 at 9:44 AM, Ravishankar N <ravishankar at redhat.com> wrote: >> On 01/28/2015 02:02 PM, Ml Ml wrote: >>> I want to either take the file from node03 or node04. i really don?t >>>> mind. Can i not just tell gluster that it should use one node as the >>>> ?current? one? >> Policy based split-brain resolution [1] which does just that, has been >> merged in master and should be available in glusterfs 3.7. >> For the moment, you would have to modify the xattrs on the one of the bricks >> and trigger heal. You can see >> https://github.com/GlusterFS/glusterfs/blob/master/doc/debugging/split-brain.md >> on how to do it. >> >> Hope this helps, >> Ravi >> >> [1] http://review.gluster.org/#/c/9377/ >>