[Gluster-users] heaps split-brains during back-transfert

Geoffrey Letessier
2015-07-27T20:57:55Z

Dears,

For a couple of weeks (more than one month), our computing production is stopped
due to several -but amazing- troubles with GlusterFS.

After having noticed a big problem with incorrect quota size accounted for many
many files, i decided under the guidance of Gluster team support to upgrade my
storage cluster from version 3.5.3 to the latest (3.7.2-3) because these bugs
are theoretically fixed in this branch. Now, since i?ve done this upgrade, it?s
the amazing mess and i cannot restart the production.
Indeed :
	1 - RDMA protocol is not working and hang my system / shell commands; only TCP
protocol (over Infiniband) is more or less operational   - it?s not a blocking
point but?
	2 - read/write performance relatively low
	3 - thousands split-brains are appeared.

So, for the moment, i believe GlusterFS 3.7 is not actually production ready.

Concerning the third point: after having destroy all my volumes (RAID re-init,
new partition, GlusterFS volumes, etc.), recreate the main one, I tried to
back-transfert my data from archive/backup server info this new volume and I
note a lot of errors in my mount log file, as your can read in this extract:
[2015-07-26 22:35:16.962815] I [afr-self-heal-entry.c:565:afr_selfheal_entry_do]
0-vol_home-replicate-0: performing entry selfheal on
865083fa-984e-44bd-aacf-b8195789d9e0
[2015-07-26 22:35:16.965896] E
[afr-self-heal-entry.c:249:afr_selfheal_detect_gfid_and_type_mismatch]
0-vol_home-replicate-0: Gfid mismatch detected for
<865083fa-984e-44bd-aacf-b8195789d9e0/job.pbs>,
e944d444-66c5-40a4-9603-7c190ad86013 on vol_home-client-1 and
820f9bcc-a0f6-40e0-bcec-28a76b4195ea on vol_home-client-0. Skipping conservative
merge on the file.
[2015-07-26 22:35:16.975206] I [afr-self-heal-entry.c:565:afr_selfheal_entry_do]
0-vol_home-replicate-0: performing entry selfheal on
29382d8d-c507-4d2e-b74d-dbdcb791ca65
[2015-07-26 22:35:28.719935] E
[afr-self-heal-entry.c:249:afr_selfheal_detect_gfid_and_type_mismatch]
0-vol_home-replicate-0: Gfid mismatch detected for
<29382d8d-c507-4d2e-b74d-dbdcb791ca65/res_1BVK_r_u_1IBR_l_u_Cond.1IBR_l_u.1BVK_r_u.UB.global.dat.txt>,
951c5ffb-ca38-4630-93f3-8e4119ab0bd8 on vol_home-client-1 and
5ae663ca-e896-4b92-8ec5-5b15422ab861 on vol_home-client-0. Skipping conservative
merge on the file.
[2015-07-26 22:35:29.764891] I [afr-self-heal-entry.c:565:afr_selfheal_entry_do]
0-vol_home-replicate-0: performing entry selfheal on
865083fa-984e-44bd-aacf-b8195789d9e0
[2015-07-26 22:35:29.768339] E
[afr-self-heal-entry.c:249:afr_selfheal_detect_gfid_and_type_mismatch]
0-vol_home-replicate-0: Gfid mismatch detected for
<865083fa-984e-44bd-aacf-b8195789d9e0/job.pbs>,
e944d444-66c5-40a4-9603-7c190ad86013 on vol_home-client-1 and
820f9bcc-a0f6-40e0-bcec-28a76b4195ea on vol_home-client-0. Skipping conservative
merge on the file.
[2015-07-26 22:35:29.775037] I [afr-self-heal-entry.c:565:afr_selfheal_entry_do]
0-vol_home-replicate-0: performing entry selfheal on
29382d8d-c507-4d2e-b74d-dbdcb791ca65
[2015-07-26 22:35:29.776857] E
[afr-self-heal-entry.c:249:afr_selfheal_detect_gfid_and_type_mismatch]
0-vol_home-replicate-0: Gfid mismatch detected for
<29382d8d-c507-4d2e-b74d-dbdcb791ca65/res_1BVK_r_u_1IBR_l_u_Cond.1IBR_l_u.1BVK_r_u.UB.global.dat.txt>,
951c5ffb-ca38-4630-93f3-8e4119ab0bd8 on vol_home-client-1 and
5ae663ca-e896-4b92-8ec5-5b15422ab861 on vol_home-client-0. Skipping conservative
merge on the file.
[2015-07-26 22:35:29.800535] W [MSGID: 108008]
[afr-self-heal-name.c:353:afr_selfheal_name_gfid_mismatch_check]
0-vol_home-replicate-0: GFID mismatch for
<gfid:29382d8d-c507-4d2e-b74d-dbdcb791ca65>/res_1BVK_r_u_1IBR_l_u_Cond.1IBR_l_u.1BVK_r_u.UB.global.dat.txt
951c5ffb-ca38-4630-93f3-8e4119ab0bd8 on vol_home-client-1 and
5ae663ca-e896-4b92-8ec5-5b15422ab861 on vol_home-client-0

And when I try to browse some folders (still in mount log file):
[2015-07-27 09:00:19.005763] I [afr-self-heal-entry.c:565:afr_selfheal_entry_do]
0-vol_home-replicate-0: performing entry selfheal on
2ac27442-8be0-4985-b48f-3328a86a6686
[2015-07-27 09:00:22.322316] E
[afr-self-heal-entry.c:249:afr_selfheal_detect_gfid_and_type_mismatch]
0-vol_home-replicate-0: Gfid mismatch detected for
<2ac27442-8be0-4985-b48f-3328a86a6686/md0012588.gro>,
9c635868-054b-4a13-b974-0ba562991586 on vol_home-client-1 and
1943175c-b336-4b33-aa1c-74a1c51f17b9 on vol_home-client-0. Skipping conservative
merge on the file.
[2015-07-27 09:00:23.008771] I [afr-self-heal-entry.c:565:afr_selfheal_entry_do]
0-vol_home-replicate-0: performing entry selfheal on
2ac27442-8be0-4985-b48f-3328a86a6686
[2015-07-27 08:59:50.359187] W [MSGID: 108008]
[afr-self-heal-name.c:353:afr_selfheal_name_gfid_mismatch_check]
0-vol_home-replicate-0: GFID mismatch for
<gfid:2ac27442-8be0-4985-b48f-3328a86a6686>/md0012588.gro
9c635868-054b-4a13-b974-0ba562991586 on vol_home-client-1 and
1943175c-b336-4b33-aa1c-74a1c51f17b9 on vol_home-client-0
[2015-07-27 09:00:02.500419] W [MSGID: 108008]
[afr-self-heal-name.c:353:afr_selfheal_name_gfid_mismatch_check]
0-vol_home-replicate-0: GFID mismatch for
<gfid:2ac27442-8be0-4985-b48f-3328a86a6686>/md0012590.gro
b22aec09-2be3-41ea-a976-7b8d0e6f61f0 on vol_home-client-1 and
ec100f9e-ec48-4b29-b75e-a50ec6245de6 on vol_home-client-0
[2015-07-27 09:00:02.506925] W [MSGID: 108008]
[afr-self-heal-name.c:353:afr_selfheal_name_gfid_mismatch_check]
0-vol_home-replicate-0: GFID mismatch for
<gfid:2ac27442-8be0-4985-b48f-3328a86a6686>/md0009059.gro
0485c093-11ca-4829-b705-e259668ebd8c on vol_home-client-1 and
e83a492b-7f8c-4b32-a76e-343f984142fe on vol_home-client-0
[2015-07-27 09:00:23.001121] W [MSGID: 108008] [afr-read-txn.c:241:afr_read_txn]
0-vol_home-replicate-0: Unreadable subvolume -1 found with event generation 2.
(Possible split-brain)
[2015-07-27 09:00:26.231262] E
[afr-self-heal-entry.c:249:afr_selfheal_detect_gfid_and_type_mismatch]
0-vol_home-replicate-0: Gfid mismatch detected for
<2ac27442-8be0-4985-b48f-3328a86a6686/md0012588.gro>,
9c635868-054b-4a13-b974-0ba562991586 on vol_home-client-1 and
1943175c-b336-4b33-aa1c-74a1c51f17b9 on vol_home-client-0. Skipping conservative
merge on the file.

And, above all, browsing folder I get a lot of input/ouput errors.

Currently I have 6.2M inodes and roughly 30TB in my "new" volume.

For the moment, Quota is disable to increase the IO performance during the
back-transfert?

Your can also find in attachments:
	- an "ls" result
	- a split-brain research result
	- the volume information and status
	- a complete volume heal info

Hoping this can help your to help me to fix all my problems and reopen the
computing production.

Thanks in advance,
Geoffrey

PS: ? Erreur d?Entr?e/Sortie ? = ? Input / Output Error ?
------------------------------------------------------
Geoffrey Letessier
Responsable informatique & ing?nieur syst?me
UPR 9080 - CNRS - Laboratoire de Biochimie Th?orique
Institut de Biologie Physico-Chimique
13, rue Pierre et Marie Curie - 75005 Paris
Tel: 01 58 41 50 93 - eMail: geoffrey.letessier at ibpc.fr

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150727/5d0f756d/attachment-0006.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ls_example.txt
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150727/5d0f756d/attachment-0005.txt>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150727/5d0f756d/attachment-0007.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: split_brain__20150725.txt
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150727/5d0f756d/attachment-0006.txt>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150727/5d0f756d/attachment-0008.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: vol_home_healinfo.txt
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150727/5d0f756d/attachment-0007.txt>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150727/5d0f756d/attachment-0009.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: vol_home_info.txt
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150727/5d0f756d/attachment-0008.txt>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150727/5d0f756d/attachment-0010.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: vol_home_status.txt
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150727/5d0f756d/attachment-0009.txt>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150727/5d0f756d/attachment-0011.html>