Krutika Dhananjay
2016-Jan-22 12:07 UTC
[Gluster-users] File Corruption when adding bricks to live replica volumes
Hi, So I looked at the glustershd logs and there are no log messages that indicate that there was a reverse heal. For instance, see this: [2016-01-22 06:13:50.268068] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-datastore1-replicate-0: Completed data selfheal on 5a3dcdd1-8866-47a5-8432-7b625a2806c3. source=0 sinks=2 The above implies that the sink was "2" (index of the sink brick - vng - counting from 0), and the source was "0" (vnb in this case), which means the heal happened into the correct brick (last brick). Could you do the following: 1) Disable client-side healing: For this, you need to execute # gluster volume set datastore1 entry-self-heal off # gluster volume set datastore1 data-self-heal off # gluster volume set datastore1 metadata-self-heal off 2) Run your add-brick/remove-brick tests again (you know it best). 3) Share the resultant glustershd.log from all three machines. -Krutika ----- Original Message -----> From: "Lindsay Mathieson" <lindsay.mathieson at gmail.com> > To: "Krutika Dhananjay" <kdhananj at redhat.com>, "gluster-users" > <gluster-users at gluster.org> > Sent: Friday, January 22, 2016 11:55:27 AM > Subject: Re: [Gluster-users] File Corruption when adding bricks to live > replica volumes> On 21/01/16 20:03, Krutika Dhananjay wrote:> > Also could you please attach glustershd.log files and the client logs? >> Ok, I ran the procedure again just to be sure. This time I got 81 shards > being healed from the other bricks, but still got 2 bricks being "reverse > healed" from the new brick. A image check on the vm file failed with:> > ERROR: I/O error in check_refcounts_l2 > > > qemu-img: Check failed: Input/output error >> removing the brick resolved the problems.> glustershd.log from node vng attached (the new brick node), the fuse client > log was empty.> thanks, > -- > Lindsay Mathieson-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160122/49e9e385/attachment.html>
Lindsay Mathieson
2016-Jan-22 12:21 UTC
[Gluster-users] File Corruption when adding bricks to live replica volumes
On 22/01/2016 10:07 PM, Krutika Dhananjay wrote:> Could you do the following: > 1) Disable client-side healing:Will do. Wasn't aware there were different types of healing - is there client and server side heals? are there any docs on this? thanks, -- Lindsay Mathieson