Carl L Hoffman
2015-Jun-02 03:40 UTC
[Gluster-users] split brain on / just after installation
Hello - I was wondering if someone could please help me. I've just setup Gluster 3.6 on two Ubuntu 14.04 hosts. Gluster is setup to replicate two volumes (prod-volume, dev-volume) between the two hosts. Replication is working fine. The glustershd.log shows: [2015-06-02 03:28:04.495162] E [afr-self-heal-common.c:197:afr_sh_print_split_brain_log] 0-prod-volume-replicate-0: Unable to self-heal contents of '<gfid:00000000-0000-0000-0000-000000000001>' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 2 ] [ 2 0 ] ] and the prod-volume logs shows: [2015-06-02 02:54:28.286268] E [afr-self-heal-common.c:197:afr_sh_print_split_brain_log] 0-prod-volume-replicate-0: Unable to self-heal contents of '/' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 2 ] [ 2 0 ] ] [2015-06-02 02:54:28.287476] E [afr-self-heal-common.c:2212:afr_self_heal_completion_cbk] 0-prod-volume-replicate-0: background meta-data self-heal failed on / I've checked against https://github.com/gluster/glusterfs/blob/6c578c03f0d44913d264494de5df004544c96271/doc/features/heal-info-and-split-brain-resolution.md but I can't see any scenario that covers mine. The output of bluster volume heal prod-volume info is: Gathering Heal info on volume prod-volume has been successful Brick server1:/export/prodvol/brick Number of entries: 1 / Brick server2 Number of entries: 1 / and doesn't show anything in split-brain. But the output of gluster volume heal prod-volume info split brain shows: Gathering Heal info on volume prod-volume has been successful Brick server1:/export/prodvol/brick Number of entries: 6 at path on brick ----------------------------------- 2015-06-02 03:28:04 / 2015-06-02 03:18:04 / 2015-06-02 03:08:04 / 2015-06-02 02:58:04 / 2015-06-02 02:48:04 / 2015-06-02 02:48:04 / Brick server2:/export/prodvol/brick Number of entries: 5 at path on brick ----------------------------------- 2015-06-02 03:28:00 / 2015-06-02 03:18:00 / 2015-06-02 03:08:00 / 2015-06-02 02:58:00 / 2015-06-02 02:48:04 / And the number continues to grow. The count on server2 is always one behind server1. Could someone please help? Cheers,
Ravishankar N
2015-Jun-02 04:08 UTC
[Gluster-users] split brain on / just after installation
On 06/02/2015 09:10 AM, Carl L Hoffman wrote:> Hello - I was wondering if someone could please help me. > > I've just setup Gluster 3.6 on two Ubuntu 14.04 hosts. Gluster is setup to replicate two volumes (prod-volume, dev-volume) between the two hosts. Replication is working fine. The glustershd.log shows:Are you sure you are running gluster 3.6? The 'afr_sh_print_split_brain_log' message appears only in gluster 3.5 or lower.> > [2015-06-02 03:28:04.495162] E [afr-self-heal-common.c:197:afr_sh_print_split_brain_log] 0-prod-volume-replicate-0: Unable to self-heal contents of '<gfid:00000000-0000-0000-0000-000000000001>' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 2 ] [ 2 0 ] ] > > and the prod-volume logs shows: > > [2015-06-02 02:54:28.286268] E [afr-self-heal-common.c:197:afr_sh_print_split_brain_log] 0-prod-volume-replicate-0: Unable to self-heal contents of '/' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 2 ] [ 2 0 ] ] > [2015-06-02 02:54:28.287476] E [afr-self-heal-common.c:2212:afr_self_heal_completion_cbk] 0-prod-volume-replicate-0: background meta-data self-heal failed on / > > I've checked against https://github.com/gluster/glusterfs/blob/6c578c03f0d44913d264494de5df004544c96271/doc/features/heal-info-and-split-brain-resolution.md but I can't see any scenario that covers mine. The output of bluster volume heal prod-volume info is:Is the metadata same on both bricks on the root? (Compare `ls -ld /export/prodvol/brick` and `getfattr -d -m . -e hex /export/prodvol/brick` on both servers to see if anything is mismatching). -Ravi> > Gathering Heal info on volume prod-volume has been successful > > Brick server1:/export/prodvol/brick > Number of entries: 1 > / > > Brick server2 > Number of entries: 1 > / > > > and doesn't show anything in split-brain. > > But the output of gluster volume heal prod-volume info split brain shows: > > Gathering Heal info on volume prod-volume has been successful > > Brick server1:/export/prodvol/brick > Number of entries: 6 > at path on brick > ----------------------------------- > 2015-06-02 03:28:04 / > 2015-06-02 03:18:04 / > 2015-06-02 03:08:04 / > 2015-06-02 02:58:04 / > 2015-06-02 02:48:04 / > 2015-06-02 02:48:04 / > > Brick server2:/export/prodvol/brick > Number of entries: 5 > at path on brick > ----------------------------------- > 2015-06-02 03:28:00 / > 2015-06-02 03:18:00 / > 2015-06-02 03:08:00 / > 2015-06-02 02:58:00 / > 2015-06-02 02:48:04 / > > > And the number continues to grow. The count on server2 is always one behind server1. > > Could someone please help? > > Cheers, > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users