Hi ,
[issue]: To ensure the glusterFS mount point work, a script will periodically
using HTTP put a file to subdirectory under mount point which is used as Apache
DocumentRoot. But after running some time, some errors show below:
[2015-01-14 09:18:40.915639] E
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-gv0-replicate-0: metadata self heal failed, on /mpdis
[2015-01-14 09:18:41.924584] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-gv0-replicate-0:
Unable to self-heal contents of '/' (possible split-brain). Please
delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 20
] [ 21 0 ] ]
[2015-01-14 09:18:41.925182] E
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-gv0-replicate-0: metadata self heal failed, on /
[2015-01-14 09:18:41.934827] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-gv0-replicate-0:
Unable to self-heal contents of '/mpdis' (possible split-brain). Please
delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 4
] [ 2 0 ] ]
[2015-01-14 09:18:41.935375] E
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-gv0-replicate-0: metadata self heal failed, on /mpdis
[2015-01-14 09:18:42.943742] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-gv0-replicate-0:
Unable to self-heal contents of '/' (possible split-brain). Please
delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 20
] [ 21 0 ] ]
[2015-01-14 09:18:42.944432] E
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-gv0-replicate-0: metadata self heal failed, on /
[2015-01-14 09:18:42.946664] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-gv0-replicate-0:
Unable to self-heal contents of '/mpdis' (possible split-brain). Please
delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 4
] [ 2 0 ] ]
[2015-01-14 09:18:42.947323] E
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-gv0-replicate-0: metadata self heal failed, on /mpdis
[2015-01-14 09:18:43.955929] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-gv0-replicate-0:
Unable to self-heal contents of '/' (possible split-brain). Please
delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 20
] [ 21 0 ] ]
[2015-01-14 09:18:43.956701] E
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-gv0-replicate-0: metadata self heal failed, on /
[2015-01-14 09:18:43.958874] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-gv0-replicate-0:
Unable to self-heal contents of '/mpdis' (possible split-brain). Please
delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 4
] [ 2 0 ] ]
Besides, I find Input/output error shown below when listing the files of under
mount point:
[root at dmf-wpst-2 mpdis]# ll
total 0
-rwxr-xr-x. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf1
-rw-r--r--. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf2
[root at dmf-wpst-2 mpdis]# ll
total 0
-rwxr-xr-x. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf1
-rw-r--r--. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf2
[root at dmf-wpst-2 mpdis]# ll
ls: cannot open directory .: Input/output error
[root at dmf-wpst-2 mpdis]# ll
ls: cannot access test.rep.00.00.00.00.dmf1: Input/output error
ls: cannot access test.rep.00.00.00.00.dmf2: Input/output error
total 0
?????????? ? ? ? ? ? test.rep.00.00.00.00.dmf1
?????????? ? ? ? ? ? test.rep.00.00.00.00.dmf2
Any tips about debugging further or getting this fixed up would be
appreciated.
[version]: 3.5.3
[environment]: two virtual server each has one brick :
root at dmf-wpst-2 mpdis]# gluster volume status
Status of volume: gv0
Gluster process
Port Online Pid
------------------------------------------------------------------------------
Brick dmf-ha-1-glusterfs:/export/vdb1/brick 49152 Y
332
Brick dmf-ha-2-glusterfs:/export/vdb1/brick 49154 Y
19396
Self-heal Daemon on localhost
N/A Y 19410
Self-heal Daemon on 10.175.123.246 N/A
Y 999
[root at dmf-wpst-1 mpdis]# gluster volume info
Volume Name: gv0
Type: Replicate
Volume ID: 51de44c3-f01e-486d-a6b7-10c7b7a270d7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: dmf-ha-1-glusterfs:/export/vdb1/brick
Brick2: dmf-ha-2-glusterfs:/export/vdb1/brick
Options Reconfigured:
nfs.disable: ON
network.ping-timeout: 2
storage.bd-aio: on
storage.linux-aio: on
cluster.eager-lock: on
performance.client-io-threads: on
performance.cache-refresh-timeout: 60
performance.io-thread-count: 64
performance.cache-size: 8GB
cluster.server-quorum-type: none
[mount-point info]:
1. mount command
glusterfs -p /var/run/glusterfs.pid --volfile-server=dmf-ha-1-glusterfs
--volfile-server=dmf-ha-2-glusterfs --volfile-id=gv0 /dmfcontents
2. mount point directory hierarchy
[root at dmf-wpst-2 /]# ls -ld /dmfcontents/
drwxr-xr-x. 5 root root 71 Jan 14 04:39 /dmfcontents/
[root at dmf-wpst-2 /]# ls -ld /dmfcontents/mpdis/
drwxr-xr-x. 2 apache apache 89 Jan 14 04:39 /dmfcontents/mpdis/
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150114/b2e56c4e/attachment.html>
Pranith Kumar Karampuri
2015-Jan-21 16:13 UTC
[Gluster-users] [gluster] possible split-brain issue
On 01/14/2015 04:48 PM, Jifeng Li wrote:> > Hi , > > [issue]: To ensure the glusterFS mount point work, a script will > periodically using HTTP put a file to subdirectory under mount point > which is used as Apache DocumentRoot. But after running some time, > some errors show below: > > [2015-01-14 09:18:40.915639] E > [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] > 0-gv0-replicate-0: metadata self heal failed, on /mpdis > > [2015-01-14 09:18:41.924584] E > [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] > 0-gv0-replicate-0: Unable to self-heal contents of '/' (possible > split-brain). Please delete the file from all but the preferred > subvolume.- Pending matrix: [ [ 0 20 ] [ 21 0 ] ] > > [2015-01-14 09:18:41.925182] E > [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] > 0-gv0-replicate-0: metadata self heal failed, on / > > [2015-01-14 09:18:41.934827] E > [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] > 0-gv0-replicate-0: Unable to self-heal contents of '/mpdis' (possible > split-brain). Please delete the file from all but the preferred > subvolume.- Pending matrix: [ [ 0 4 ] [ 2 0 ] ] > > [2015-01-14 09:18:41.935375] E > [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] > 0-gv0-replicate-0: metadata self heal failed, on /mpdis > > [2015-01-14 09:18:42.943742] E > [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] > 0-gv0-replicate-0: Unable to self-heal contents of '/' (possible > split-brain). Please delete the file from all but the preferred > subvolume.- Pending matrix: [ [ 0 20 ] [ 21 0 ] ] > > [2015-01-14 09:18:42.944432] E > [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] > 0-gv0-replicate-0: metadata self heal failed, on / > > [2015-01-14 09:18:42.946664] E > [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] > 0-gv0-replicate-0: Unable to self-heal contents of '/mpdis' (possible > split-brain). Please delete the file from all but the preferred > subvolume.- Pending matrix: [ [ 0 4 ] [ 2 0 ] ] > > [2015-01-14 09:18:42.947323] E > [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] > 0-gv0-replicate-0: metadata self heal failed, on /mpdis > > [2015-01-14 09:18:43.955929] E > [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] > 0-gv0-replicate-0: Unable to self-heal contents of '/' (possible > split-brain). Please delete the file from all but the preferred > subvolume.- Pending matrix: [ [ 0 20 ] [ 21 0 ] ] > > [2015-01-14 09:18:43.956701] E > [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] > 0-gv0-replicate-0: metadata self heal failed, on / > > [2015-01-14 09:18:43.958874] E > [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] > 0-gv0-replicate-0: Unable to self-heal contents of '/mpdis' (possible > split-brain). Please delete the file from all but the preferred > subvolume.- Pending matrix: [ [ 0 4 ] [ 2 0 ] ] > > Besides, I find Input/output error shown below when listing the > files of under mount point: > > [root at dmf-wpst-2 mpdis]# ll > > total 0 > > -rwxr-xr-x. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf1 > > -rw-r--r--. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf2 > > [root at dmf-wpst-2 mpdis]# ll > > total 0 > > -rwxr-xr-x. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf1 > > -rw-r--r--. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf2 > > [root at dmf-wpst-2 mpdis]# ll > > ls: cannot open directory .: Input/output error > > [root at dmf-wpst-2 mpdis]# ll > > ls: cannot access test.rep.00.00.00.00.dmf1: Input/output error > > ls: cannot access test.rep.00.00.00.00.dmf2: Input/output error > > total 0 > > ?????????? ? ? ? ? ? test.rep.00.00.00.00.dmf1 > > ?????????? ? ? ? ? ? test.rep.00.00.00.00.dmf2 > > * Any tips about debugging further or getting this fixed up would > be appreciated. * > > [version]: 3.5.3 > > [environment]: two virtual server each has one brick : > > root at dmf-wpst-2 mpdis]# gluster volume status > > Status of volume: gv0 > > Gluster process Port Online Pid > > ------------------------------------------------------------------------------ > > Brick dmf-ha-1-glusterfs:/export/vdb1/brick 49152 Y 332 > > Brick dmf-ha-2-glusterfs:/export/vdb1/brick 49154 Y 19396 > > Self-heal Daemon on localhost N/A Y 19410 > > Self-heal Daemon on 10.175.123.246 N/A Y 999 > > [root at dmf-wpst-1 mpdis]# gluster volume info > > Volume Name: gv0 > > Type: Replicate > > Volume ID: 51de44c3-f01e-486d-a6b7-10c7b7a270d7 > > Status: Started > > Number of Bricks: 1 x 2 = 2 > > Transport-type: tcp > > Bricks: > > Brick1: dmf-ha-1-glusterfs:/export/vdb1/brick > > Brick2: dmf-ha-2-glusterfs:/export/vdb1/brick > > Options Reconfigured: > > nfs.disable: ON > > network.ping-timeout: 2 > > storage.bd-aio: on > > storage.linux-aio: on > > cluster.eager-lock: on > > performance.client-io-threads: on > > performance.cache-refresh-timeout: 60 > > performance.io-thread-count: 64 > > performance.cache-size: 8GB > > cluster.server-quorum-type: none > > [mount-point info]: > > 1.mount command > > glusterfs -p /var/run/glusterfs.pid > --volfile-server=dmf-ha-1-glusterfs > --volfile-server=dmf-ha-2-glusterfs --volfile-id=gv0 /dmfcontents > > 2.mount point directory hierarchy > > [root at dmf-wpst-2 /]# ls -ld /dmfcontents/ > > drwxr-xr-x. 5 root root 71 Jan 14 04:39 /dmfcontents/ > > [root at dmf-wpst-2 /]# ls -ld /dmfcontents/mpdis/ > > drwxr-xr-x. 2 apache apache 89 Jan 14 04:39 > /dmfcontents/mpdis/ >hi Jifeng Li, Sorry for the delay in response. Could you post the output of: 'getfattr -d -m. -e hex <brick-path>' 'getfattr -d -m. -e hex <brick-path>/mpdis' 'getfattr -d -m. -e hex <brick-path>/mpdis/test.rep.00.00.00.00.dmf1' 'getfattr -d -m. -e hex <brick-path>/mpdis/test.rep.00.00.00.00.dmf2' Pranith> > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150121/26710104/attachment.html>