Dan Bretherton
2010-Nov-25 10:55 UTC
[Gluster-users] Unable to self-heal permissions/ownership of '/' (possible split-brain)
Hello all- I have set up two new servers each with two RAID 5 arrays. I am trying to distribute the load across the two RAID arrays on each server by creating a distributed, replicated volume. This is the output from "gluster volume info". Volume Name: marine2 Type: Distributed-Replicate Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 192.171.166.86:/vg1lv0/glusterfs Brick2: 192.171.166.87:/vg1lv0/glusterfs Brick3: 192.171.166.86:/vg2lv0/glusterfs Brick4: 192.171.166.87:/vg2lv0/glusterfs Options Reconfigured: performance.quick-read: off The volume seems to be working normally, but I am seeing the following messages repeated in my volume log file: [2010-11-25 10:27:06.867368] I [afr-common.c:672:afr_lookup_done] marine2-replicate-0: split brain detected during lookup of /. [2010-11-25 10:27:06.867454] I [afr-common.c:716:afr_lookup_done] marine2-replicate-0: background meta-data data self-heal triggered. path: / [2010-11-25 10:27:06.867614] I [afr-common.c:672:afr_lookup_done] marine2-replicate-1: split brain detected during lookup of /. [2010-11-25 10:27:06.867633] I [afr-common.c:716:afr_lookup_done] marine2-replicate-1: background meta-data data self-heal triggered. path: / [2010-11-25 10:27:06.868174] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] marine2-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2010-11-25 10:27:06.868265] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] marine2-replicate-1: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2010-11-25 10:27:06.868483] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] marine2-replicate-0: background meta-data data self-heal completed on / [2010-11-25 10:27:06.868541] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] marine2-replicate-1: background meta-data data self-heal completed on / I don't understand why a self heal is ever necessary on "/". These presumably are the brick directories (/vg1lv0/glusterfs etc), which are all owned by root and have the same permissions. Is this something I should be worried about, and if not how can I stop these warnings? -Dan.
Dan Bretherton
2010-Nov-25 11:20 UTC
[Gluster-users] Unable to self-heal permissions/ownership of '/' (possible split-brain)
On 25/11/10 10:55, Dan Bretherton wrote:> Hello all- > > I have set up two new servers each with two RAID 5 arrays. I am trying > to distribute the load across the two RAID arrays on each server by > creating a distributed, replicated volume. This is the output from > "gluster volume info". > > Volume Name: marine2 > Type: Distributed-Replicate > Status: Started > Number of Bricks: 2 x 2 = 4 > Transport-type: tcp > Bricks: > Brick1: 192.171.166.86:/vg1lv0/glusterfs > Brick2: 192.171.166.87:/vg1lv0/glusterfs > Brick3: 192.171.166.86:/vg2lv0/glusterfs > Brick4: 192.171.166.87:/vg2lv0/glusterfs > Options Reconfigured: > performance.quick-read: off > > The volume seems to be working normally, but I am seeing the following > messages repeated in my volume log file: > > [2010-11-25 10:27:06.867368] I [afr-common.c:672:afr_lookup_done] > marine2-replicate-0: split brain detected during lookup of /. > [2010-11-25 10:27:06.867454] I [afr-common.c:716:afr_lookup_done] > marine2-replicate-0: background meta-data data self-heal triggered. path: / > [2010-11-25 10:27:06.867614] I [afr-common.c:672:afr_lookup_done] > marine2-replicate-1: split brain detected during lookup of /. > [2010-11-25 10:27:06.867633] I [afr-common.c:716:afr_lookup_done] > marine2-replicate-1: background meta-data data self-heal triggered. path: / > [2010-11-25 10:27:06.868174] E > [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] marine2-replicate-0: > Unable to self-heal permissions/ownership of '/' (possible split-brain). > Please fix the file on all backend volumes > [2010-11-25 10:27:06.868265] E > [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] marine2-replicate-1: > Unable to self-heal permissions/ownership of '/' (possible split-brain). > Please fix the file on all backend volumes > [2010-11-25 10:27:06.868483] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > marine2-replicate-0: background meta-data data self-heal completed on / > [2010-11-25 10:27:06.868541] I > [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] > marine2-replicate-1: background meta-data data self-heal completed on / > > I don't understand why a self heal is ever necessary on "/". These > presumably are the brick directories (/vg1lv0/glusterfs etc), which are > all owned by root and have the same permissions. Is this something I > should be worried about, and if not how can I stop these warnings? > -Dan. >Sorry I have just seen this previous posting while trawling though the mail archives researching another problem (but I did Google search before posting mine earlier on): http://gluster.org/pipermail/gluster-users/2010-November/005780.html My getfattr output. There doesn't seem to have been any more discussion in the previous thread about this issue. [root at bdan4 ~]# getfattr -d -e hex -m trusted.afr /vg1lv0/glusterfs getfattr: Removing leading '/' from absolute path names # file: vg1lv0/glusterfs trusted.afr.marine2-client-0=0x000000000000000000000000 trusted.afr.marine2-client-1=0x000000000000000100000000 [root at bdan4 ~]# getfattr -d -e hex -m trusted.afr /vg2lv0/glusterfs getfattr: Removing leading '/' from absolute path names # file: vg2lv0/glusterfs trusted.afr.marine2-client-2=0x000000000000000000000000 trusted.afr.marine2-client-3=0x000000000000000100000000 [root at bdan5 ~]# getfattr -d -e hex -m trusted.afr /vg1lv0/glusterfs getfattr: Removing leading '/' from absolute path names # file: vg1lv0/glusterfs trusted.afr.marine2-client-0=0x000000000000000100000000 trusted.afr.marine2-client-1=0x000000000000000000000000 [root at bdan5 ~]# getfattr -d -e hex -m trusted.afr /vg2lv0/glusterfs getfattr: Removing leading '/' from absolute path names # file: vg2lv0/glusterfs trusted.afr.marine2-client-2=0x000000000000000100000000 trusted.afr.marine2-client-3=0x000000000000000000000000
Mark "Naoki" Rogers
2010-Nov-26 03:19 UTC
[Gluster-users] Unable to self-heal permissions/ownership of '/' (possible split-brain)
On 11/25/2010 07:55 PM, Dan Bretherton wrote:> The volume seems to be working normally, but I am seeing the following > messages repeated in my volume log file: > > [2010-11-25 10:27:06.867368] I [afr-common.c:672:afr_lookup_done] > marine2-replicate-0: split brain detected during lookup of /. > [2010-11-25 10:27:06.867454] I [afr-common.c:716:afr_lookup_done] > marine2-replicate-0: background meta-data data self-heal triggered. path: / > [2010-11-25 10:27:06.867614] I [afr-common.c:672:afr_lookup_done] > marine2-replicate-1: split brain detected during lookup of /. > [2010-11-25 10:27:06.867633] I [afr-common.c:716:afr_lookup_done] > marine2-replicate-1: background meta-data data self-heal triggered. path: / > [2010-11-25 10:27:06.868174] E > [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] marine2-replicate-0: > Unable to self-heal permissions/ownership of '/' (possible split-brain). > Please fix the file on all backend volumesI've been seeing the exact same thing with all 3.1 releases (not tested past qa8): [2010-11-25 03:49:17.920483] I [afr-common.c:672:afr_lookup_done] test-volume-replicate-0: split brain detected during lookup of /. [2010-11-25 03:49:17.926982] I [afr-common.c:716:afr_lookup_done] test-volume-replicate-0: background meta-data data self-heal triggered. path: / [2010-11-25 03:49:17.955846] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] test-volume-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2010-11-25 03:49:17.956445] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] test-volume-replicate-0: background meta-data data self-heal completed on / [2010-11-26 04:21:20.549679] I [afr-common.c:672:afr_lookup_done] test-volume-replicate-0: split brain detected during lookup of /. [2010-11-26 04:21:20.589245] I [afr-common.c:716:afr_lookup_done] test-volume-replicate-0: background meta-data data self-heal triggered. path: / [2010-11-26 04:21:20.591699] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] test-volume-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes