Chaloulos, Klearchos (Nokia - GR/Athens)
2016-Feb-05 15:00 UTC
[Gluster-users] Different file in two bricks, no split-brain detected
Hello, I managed to get logs from two occurrences, please see anonymized logs attached. Occurrence1: The copied file in both bricks had the correct checksum, but the client so an erroneous checksum. Occurrence2: The copied file in one brick had the correct checksum and in the second brick has an erroneous checksum. For details check the notes.txt file in the tarballs. Do you have any idea what could be causing this behavior? Best regards, Klearchos From: Chaloulos, Klearchos (Nokia - GR/Athens) Sent: Monday, February 01, 2016 10:39 AM To: 'EXT Krutika Dhananjay' <kdhananj at redhat.com> Cc: gluster-users at gluster.org Subject: RE: [Gluster-users] Different file in two bricks, no split-brain detected Hello, Sorry for not replying, but lately the issue cannot be reproduced. If we have any new occurrences I?ll collect the logs and send them here. Klearchos From: EXT Krutika Dhananjay [mailto:kdhananj at redhat.com] Sent: Wednesday, January 27, 2016 7:12 AM To: Chaloulos, Klearchos (Nokia - GR/Athens) <klearchos.chaloulos at nokia.com<mailto:klearchos.chaloulos at nokia.com>> Cc: gluster-users at gluster.org<mailto:gluster-users at gluster.org> Subject: Re: [Gluster-users] Different file in two bricks, no split-brain detected Hi, Could you share the following pieces of information: 1) output of `gluster volume info <VOL>` 2) the client/mount logs 3) glustershd logs -Krutika ________________________________ From: "Klearchos Chaloulos (Nokia - GR/Athens)" <klearchos.chaloulos at nokia.com<mailto:klearchos.chaloulos at nokia.com>> To: gluster-users at gluster.org<mailto:gluster-users at gluster.org> Sent: Tuesday, January 26, 2016 9:57:38 PM Subject: [Gluster-users] Different file in two bricks, no split-brain detected Description of problem: My setup has 5 gluster volumes, and each of them has 2 bricks as backend. When I copy a large file (100MB) in a gluster volume, 9/10 times it works OK. But about 1 in 10 times the resulting md5 is wrong. After checking I found that the file in one brick has the correct md5sum, while the file in the other brick has a wrong md5sum. The size of the two files is the same. By running "cmp -l <correct_file> <wrong_file>" I found that the difference was in 49 bytes. So the files in the two bricks had the same size, but 49 files were different. Interestingly enough I saw the same number of 49 bytes being different at every check that I made. Do you know what might cause this behavior, has anyone seen something like this before? Is this a bug in glusterfs? Version-Release number of selected component (if applicable): glusterfs 3.7.5 built on Nov 19 2015 16:29:59 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. How reproducible: Not easy to reproduce, about 1 in 10 times in some environments, not reproducible at all in other environments. Steps to Reproduce: 1. scp <100MB file> <path in gluster volume> Actual results: 1. md5sum of destination should be the same as the source 2. If checksum of files is different between the two bricks, the command "gluster volume heal <vol-name> info split-brain" should return that the two bricks are in split-brain. Expected results: 1. 1 in 10 times the destination file has incorrect checksum. Size is the same, but 49 bytes are altered. 2. "gluster volume heal <vol-name> info split-brain" does not return that the bricks are in split-brain, even though the checksum of the file in the two bricks is different. The size of the file is the same in the two bricks. But 49 bytes are altered. Additional info: _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org<mailto:Gluster-users at gluster.org> http://www.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160205/c95a1b24/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: occurence2.tgz Type: application/x-compressed Size: 135723 bytes Desc: occurence2.tgz URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160205/c95a1b24/attachment.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: occurence1.tgz Type: application/x-compressed Size: 136010 bytes Desc: occurence1.tgz URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160205/c95a1b24/attachment-0001.bin>
Krutika Dhananjay
2016-Feb-08 10:30 UTC
[Gluster-users] Different file in two bricks, no split-brain detected
Hi, Could you disable quick-read, read-ahead and io-cache and run your test again and share the results: #gluster volume set <VOLNAME> performance.quick-read off #gluster volume set <VOLNAME> performance.read-ahead off #gluster volume set <VOLNAME> performance.io-cache off -Krutika ----- Original Message -----> From: "Klearchos Chaloulos (Nokia - GR/Athens)" > <klearchos.chaloulos at nokia.com> > To: "EXT Krutika Dhananjay" <kdhananj at redhat.com> > Cc: gluster-users at gluster.org > Sent: Friday, February 5, 2016 8:30:57 PM > Subject: RE: [Gluster-users] Different file in two bricks, no split-brain > detected> Hello,> I managed to get logs from two occurrences, please see anonymized logs > attached.> Occurrence1: The copied file in both bricks had the correct checksum, but the > client so an erroneous checksum.> Occurrence2: The copied file in one brick had the correct checksum and in the > second brick has an erroneous checksum.> For details check the notes.txt file in the tarballs.> Do you have any idea what could be causing this behavior?> Best regards,> Klearchos> From: Chaloulos, Klearchos (Nokia - GR/Athens) > Sent: Monday, February 01, 2016 10:39 AM > To: 'EXT Krutika Dhananjay' <kdhananj at redhat.com> > Cc: gluster-users at gluster.org > Subject: RE: [Gluster-users] Different file in two bricks, no split-brain > detected> Hello,> Sorry for not replying, but lately the issue cannot be reproduced. If we have > any new occurrences I?ll collect the logs and send them here.> Klearchos> From: EXT Krutika Dhananjay [ mailto:kdhananj at redhat.com ] > Sent: Wednesday, January 27, 2016 7:12 AM > To: Chaloulos, Klearchos (Nokia - GR/Athens) < klearchos.chaloulos at nokia.com > > > Cc: gluster-users at gluster.org > Subject: Re: [Gluster-users] Different file in two bricks, no split-brain > detected> Hi,> Could you share the following pieces of information:> 1) output of `gluster volume info <VOL>`> 2) the client/mount logs> 3) glustershd logs> -Krutika> > From: "Klearchos Chaloulos (Nokia - GR/Athens)" < > > klearchos.chaloulos at nokia.com > > > > To: gluster-users at gluster.org > > > Sent: Tuesday, January 26, 2016 9:57:38 PM > > > Subject: [Gluster-users] Different file in two bricks, no split-brain > > detected >> > Description of problem: >> > My setup has 5 gluster volumes, and each of them has 2 bricks as backend. >> > When I copy a large file (100MB) in a gluster volume, 9/10 times it works > > OK. > > But about 1 in 10 times the resulting md5 is wrong. After checking I found > > that the file in one brick has the correct md5sum, while the file in the > > other brick has a wrong md5sum. The size of the two files is the same. >> > By running "cmp -l <correct_file> <wrong_file>" >> > I found that the difference was in 49 bytes. So the files in the two bricks > > had the same size, but 49 files were different. Interestingly enough I saw > > the same number of 49 bytes being different at every check that I made. >> > Do you know what might cause this behavior, has anyone seen something like > > this before? Is this a bug in glusterfs? >> > Version-Release number of selected component (if applicable): >> > glusterfs 3.7.5 built on Nov 19 2015 16:29:59 >> > Repository revision: git://git.gluster.com/glusterfs.git >> > Copyright (c) 2006-2011 Gluster Inc. < http://www.gluster.com > >> > GlusterFS comes with ABSOLUTELY NO WARRANTY. >> > You may redistribute copies of GlusterFS under the terms of the GNU General > > Public License. >> > How reproducible: >> > Not easy to reproduce, about 1 in 10 times in some environments, not > > reproducible at all in other environments. >> > Steps to Reproduce: >> > 1. scp <100MB file> <path in gluster volume> >> > Actual results: >> > 1. md5sum of destination should be the same as the source >> > 2. If checksum of files is different between the two bricks, the command > > "gluster volume heal <vol-name> info split-brain" should return that the > > two > > bricks are in split-brain. >> > Expected results: >> > 1. 1 in 10 times the destination file has incorrect checksum. Size is the > > same, but 49 bytes are altered. >> > 2. "gluster volume heal <vol-name> info split-brain" does not return that > > the > > bricks are in split-brain, even though the checksum of the file in the two > > bricks is different. The size of the file is the same in the two bricks. > > But > > 49 bytes are altered. >> > Additional info: >> > _______________________________________________ > > > Gluster-users mailing list > > > Gluster-users at gluster.org > > > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160208/1aa4447f/attachment.html>