tatebet at nttdata.co.jp
2011-Mar-04 01:35 UTC
[Gluster-users] Unable to self-heal permissions/ownership of '/' (possible split-brain)
Hi everyone, I have a problem about GlusterFS-3.1.2. The following message occurred: "Unable to self-heal permissions/ownership of '/' (possible split-brain)" The volume seems to be working normally, but what kind of issue does this error cause? Please let us know how to fix it and the reason about this error. I found the same issue from the following, but I could not find the resolution. <http://www.mail-archive.com/gluster-users at gluster.org/msg04442.html> <http://gluster.org/pipermail/gluster-users/2010-November/005780.html> I am seeing the following messages repeated in my volume log file: [2011-03-01 18:12:54.862809] I [afr-common.c:672:afr_lookup_done] nas-d-volume-replicate-1: split brain detected during lookup of /. [2011-03-01 18:12:54.862839] I [afr-common.c:716:afr_lookup_done] nas-d-volume-replicate-1: background meta-data data self-heal triggered. path: / [2011-03-01 18:12:54.862990] I [afr-common.c:672:afr_lookup_done] nas-d-volume-replicate-2: split brain detected during lookup of /. [2011-03-01 18:12:54.863006] I [afr-common.c:716:afr_lookup_done] nas-d-volume-replicate-2: background meta-data data self-heal triggered. path: / [2011-03-01 18:12:54.863106] I [afr-common.c:672:afr_lookup_done] nas-d-volume-replicate-0: split brain detected during lookup of /. [2011-03-01 18:12:54.863120] I [afr-common.c:716:afr_lookup_done] nas-d-volume-replicate-0: background meta-data data self-heal triggered. path: / [2011-03-01 18:12:54.863486] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] nas-d-volume-replicate-1: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2011-03-01 18:12:54.863599] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] nas-d-volume-replicate-2: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2011-03-01 18:12:54.863742] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] nas-d-volume-replicate-1: background meta-data data self-heal completed on / [2011-03-01 18:12:54.863804] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] nas-d-volume-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2011-03-01 18:12:54.863864] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] nas-d-volume-replicate-2: background meta-data data self-heal completed on / [2011-03-01 18:12:54.863987] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] nas-d-volume-replicate-0: background meta-data data self-heal completed on / [2011-03-01 18:12:55.880727] I [afr-common.c:672:afr_lookup_done] nas-d-volume-replicate-1: split brain detected during lookup of /. [2011-03-01 18:12:55.880754] I [afr-common.c:716:afr_lookup_done] nas-d-volume-replicate-1: background meta-data data self-heal triggered. path: / [2011-03-01 18:12:55.880903] I [afr-common.c:672:afr_lookup_done] nas-d-volume-replicate-0: split brain detected during lookup of /. [2011-03-01 18:12:55.880950] I [afr-common.c:716:afr_lookup_done] nas-d-volume-replicate-0: background meta-data data self-heal triggered. path: / [2011-03-01 18:12:55.881100] I [afr-common.c:672:afr_lookup_done] nas-d-volume-replicate-2: split brain detected during lookup of /. [2011-03-01 18:12:55.881116] I [afr-common.c:716:afr_lookup_done] nas-d-volume-replicate-2: background meta-data data self-heal triggered. path: / [2011-03-01 18:12:55.881295] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] nas-d-volume-replicate-1: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2011-03-01 18:12:55.881543] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] nas-d-volume-replicate-1: background meta-data data self-heal completed on / [2011-03-01 18:12:55.881597] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] nas-d-volume-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2011-03-01 18:12:55.881784] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] nas-d-volume-replicate-0: background meta-data data self-heal completed on / [2011-03-01 18:12:55.881858] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] nas-d-volume-replicate-2: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2011-03-01 18:12:55.882045] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] nas-d-volume-replicate-2: background meta-data data self-heal completed on / I am seeing the following server's information and volume status: [root at nas-d01 ~]# gluster peer status Number of Peers: 8 Hostname: nas-d07.linda.localdomain Uuid: 7c314b62-5df9-4488-b517-2983eb09e895 State: Peer in Cluster (Connected) Hostname: nas-d09.linda.localdomain Uuid: edcba91f-65d1-4a42-a66a-67b18efb7534 State: Peer in Cluster (Connected) Hostname: nas-d06.linda.localdomain Uuid: 3e24be97-bc91-4462-8584-15d4df0099b1 State: Peer in Cluster (Connected) Hostname: nas-d04.linda.localdomain Uuid: ad46f49a-ddbf-4112-889f-d21628c8ff70 State: Peer in Cluster (Connected) Hostname: nas-d03.linda.localdomain Uuid: 9d22942b-14a1-4fa8-b1a4-513a3ddf7348 State: Peer in Cluster (Connected) Hostname: nas-d05.linda.localdomain Uuid: 6b419ddc-1f1a-41d0-b771-32b584fee3b2 State: Peer in Cluster (Connected) Hostname: nas-d02.linda.localdomain Uuid: f1194656-e5fc-4bd0-a784-c1f90cb4cff5 State: Peer in Cluster (Connected) Hostname: nas-d08.linda.localdomain Uuid: 1d4e8e0c-299f-4abe-82f0-d908bc8238a4 State: Peer in Cluster (Connected) [root at nas-d01 ~]# gluster volume info Volume Name: nas-d-volume Type: Distributed-Replicate Status: Started Number of Bricks: 3 x 3 = 9 Transport-type: tcp Bricks: Brick1: nas-d01.linda.localdomain:/glusterfsd/data Brick2: nas-d02.linda.localdomain:/glusterfsd/data Brick3: nas-d03.linda.localdomain:/glusterfsd/data Brick4: nas-d04.linda.localdomain:/glusterfsd/data Brick5: nas-d05.linda.localdomain:/glusterfsd/data Brick6: nas-d06.linda.localdomain:/glusterfsd/data Brick7: nas-d07.linda.localdomain:/glusterfsd/data Brick8: nas-d08.linda.localdomain:/glusterfsd/data Brick9: nas-d09.linda.localdomain:/glusterfsd/data [root at nas-d01 ~]# getfattr -d -e hex -m trusted.afr /glusterfsd/data/ getfattr: Removing leading '/' from absolute path names # file: glusterfsd/data trusted.afr.nas-d-volume-client-0=0x000000000000000000000000 trusted.afr.nas-d-volume-client-1=0x000000000100000000000000 trusted.afr.nas-d-volume-client-2=0x000000000000000000000000 [root at nas-d02 ~]# getfattr -d -e hex -m trusted.afr /glusterfsd/data/ getfattr: Removing leading '/' from absolute path names # file: glusterfsd/data trusted.afr.nas-d-volume-client-0=0x000000000200000000000000 trusted.afr.nas-d-volume-client-1=0x000000000000000000000000 trusted.afr.nas-d-volume-client-2=0x000000000100000000000000 [root at nas-d03 ~]# getfattr -d -e hex -m trusted.afr /glusterfsd/data/ getfattr: Removing leading '/' from absolute path names # file: glusterfsd/data trusted.afr.nas-d-volume-client-0=0x000000000000000000000000 trusted.afr.nas-d-volume-client-1=0x000000000200000000000000 trusted.afr.nas-d-volume-client-2=0x000000000000000000000000 [root at nas-d04 ~]# getfattr -d -e hex -m trusted.afr /glusterfsd/data/ getfattr: Removing leading '/' from absolute path names # file: glusterfsd/data trusted.afr.nas-d-volume-client-3=0x000000000000000000000000 trusted.afr.nas-d-volume-client-4=0x000000000000000000000000 trusted.afr.nas-d-volume-client-5=0x000000000000000000000000 [root at nas-d05 ~]# getfattr -d -e hex -m trusted.afr /glusterfsd/data/ getfattr: Removing leading '/' from absolute path names # file: glusterfsd/data trusted.afr.nas-d-volume-client-3=0x000000000000000000000000 trusted.afr.nas-d-volume-client-4=0x000000000000000000000000 trusted.afr.nas-d-volume-client-5=0x000000000200000000000000 [root at nas-d06 ~]# getfattr -d -e hex -m trusted.afr /glusterfsd/data/ getfattr: Removing leading '/' from absolute path names # file: glusterfsd/data trusted.afr.nas-d-volume-client-3=0x000000000100000000000000 trusted.afr.nas-d-volume-client-4=0x000000000100000000000000 trusted.afr.nas-d-volume-client-5=0x000000000000000000000000 [root at nas-d07 ~]# getfattr -d -e hex -m trusted.afr /glusterfsd/data/ getfattr: Removing leading '/' from absolute path names # file: glusterfsd/data trusted.afr.nas-d-volume-client-6=0x000000000000000000000000 trusted.afr.nas-d-volume-client-7=0x000000000100000000000000 trusted.afr.nas-d-volume-client-8=0x000000000100000000000000 [root at nas-d08 ~]# getfattr -d -e hex -m trusted.afr /glusterfsd/data/ getfattr: Removing leading '/' from absolute path names # file: glusterfsd/data trusted.afr.nas-d-volume-client-6=0x000000000200000100000000 trusted.afr.nas-d-volume-client-7=0x000000000000000000000000 trusted.afr.nas-d-volume-client-8=0x000000000200000000000000 [root at nas-d09 ~]# getfattr -d -e hex -m trusted.afr /glusterfsd/data/ getfattr: Removing leading '/' from absolute path names # file: glusterfsd/data trusted.afr.nas-d-volume-client-6=0x000000000000000100000000 trusted.afr.nas-d-volume-client-7=0x000000000000000000000000 trusted.afr.nas-d-volume-client-8=0x000000000000000000000000