Kotresh Hiremath Ravishankar
2016-Jul-21 05:45 UTC
[Gluster-users] bitrot error correction in gluster 3.8 distributed dispersed setup
Hi, Please follow below steps to restore: 1. Determine the path of each corrupted object from gfid shown from bitrot status command # find /bricks/*/.glusterfs -name GFID /bricks/brick1/.glusterfs/path/to/GFID # find /bricks -samefile /bricks/brick1/.glusterfs/path/to/GFID /bricks/brick1/.glusterfs/path/to/GFID /bricks/brick1/path/to/corrupted_file 2. Delete the corrupted files including .glusterfs GFID link file Delete the corrupted files from the path output by the find command. 3. Heal the file If you have client self-heal enabled, the file is healed the next time that you access it. If you do not have client self-heal enabled, you must manually heal the volume with the following command. # gluster volume heal VOLNAME The next time that the bitrot scrubber runs, this GFID is no longer listed (unless it has become corrupted again). Thanks and Regards, Kotresh H R ----- Original Message -----> From: "Ashish Pandey" <aspandey at redhat.com> > To: "itlinux_team" <itlinux at imppc.org> > Cc: gluster-users at gluster.org, "Kotresh Hiremath Ravishankar" <khiremat at redhat.com> > Sent: Thursday, July 21, 2016 10:57:51 AM > Subject: Re: [Gluster-users] bitrot error correction in gluster 3.8 distributed dispersed setup > > > Hi, > > "gluster v heal " command does not heal corrupt files detected by bitrot > feature. > You have modified a file from back end and corrupted it. > > I think to correct your corrupted files (in fact the part of file in case of > disperse volume) you have to delete that part and run gluster v heal > <volname> full > to correct the file. > > Keeping Kotresh in cc for better input of bitrot. > > Ashish > > > ----- Original Message ----- > > From: "itlinux_team" <itlinux at imppc.org> > To: gluster-users at gluster.org > Sent: Wednesday, July 20, 2016 7:43:56 PM > Subject: [Gluster-users] bitrot error correction in gluster 3.8 distributed > dispersed setup > > Hi all, > > I'm testing the bitrot feature on a distributed dispersed setup: 3 x (8 > + 4). > I simulated a bit-flip on the part of a couple of files in a brick, > having bitrot feature enabled before. The output from ''gluster volume > bitrot vol_1 scrub status'' shows that two files are corrupted as shown > below: > > > --- > Volume name : vol_1 > > State of scrub: Active > > Scrub impact: normal > > Scrub frequency: daily > > Bitrot error log location: /var/log/glusterfs/bitd.log > > Scrubber error log location: /var/log/glusterfs/scrub.log > > > ========================================================> > Node: localhost > > Number of Scrubbed files: 564 > > Number of Skipped files: 0 > > Last completed scrub time: 2016-07-20 13:37:16 > > Duration of last scrub (D:M:H:M:S): 0:0:0:50 > > Error count: 0 > > > ========================================================> > Node: glusterserver1 > > Number of Scrubbed files: 562 > > Number of Skipped files: 0 > > Last completed scrub time: 2016-07-20 13:37:16 > > Duration of last scrub (D:M:H:M:S): 0:0:0:51 > > Error count: 2 > > Corrupted object's [GFID]: > > e3cbffbe-12bd-4e1c-8dcf-2ef87ac99990 > > 2550475e-cf9e-4781-9a05-9e11c1dd6b7a > > > ========================================================> > Node: glusterserver3 > > Number of Scrubbed files: 564 > > Number of Skipped files: 0 > > Last completed scrub time: 2016-07-20 13:37:16 > > Duration of last scrub (D:M:H:M:S): 0:0:0:50 > --- > > > I have executed ''gluster volume heal vol_1 full'' but it does not seem > to fix the files, however the file that the client reads is the correct > one. > > How should I handle this kind of corruption within gluster? > > Thanks! > I?aki. > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > >
itlinux_team
2016-Jul-21 08:29 UTC
[Gluster-users] bitrot error correction in gluster 3.8 distributed dispersed setup
Hi, the procedure worked (had to execute a heal manually to heal both files as I don't have self heal enabled). Both files where also in .glusterfs/quarantine , I assume it is safe to delete them too or it will affect the next scrub? Thanks for the info! Regards, I?aki. On 07/21/2016 07:45 AM, Kotresh Hiremath Ravishankar wrote:> Hi, > > Please follow below steps to restore: > > > 1. Determine the path of each corrupted object from gfid shown from bitrot status command > > # find /bricks/*/.glusterfs -name GFID > /bricks/brick1/.glusterfs/path/to/GFID > > # find /bricks -samefile /bricks/brick1/.glusterfs/path/to/GFID > /bricks/brick1/.glusterfs/path/to/GFID > /bricks/brick1/path/to/corrupted_file > > 2. Delete the corrupted files including .glusterfs GFID link file > Delete the corrupted files from the path output by the find command. > > 3. Heal the file > If you have client self-heal enabled, the file is healed the next time that you access it. > If you do not have client self-heal enabled, you must manually heal the volume with the following command. > > # gluster volume heal VOLNAME > > The next time that the bitrot scrubber runs, this GFID is no longer listed (unless it has become corrupted again). > > Thanks and Regards, > Kotresh H R > > ----- Original Message ----- >> From: "Ashish Pandey" <aspandey at redhat.com> >> To: "itlinux_team" <itlinux at imppc.org> >> Cc: gluster-users at gluster.org, "Kotresh Hiremath Ravishankar" <khiremat at redhat.com> >> Sent: Thursday, July 21, 2016 10:57:51 AM >> Subject: Re: [Gluster-users] bitrot error correction in gluster 3.8 distributed dispersed setup >> >> >> Hi, >> >> "gluster v heal " command does not heal corrupt files detected by bitrot >> feature. >> You have modified a file from back end and corrupted it. >> >> I think to correct your corrupted files (in fact the part of file in case of >> disperse volume) you have to delete that part and run gluster v heal >> <volname> full >> to correct the file. >> >> Keeping Kotresh in cc for better input of bitrot. >> >> Ashish >> >> >> ----- Original Message ----- >> >> From: "itlinux_team" <itlinux at imppc.org> >> To: gluster-users at gluster.org >> Sent: Wednesday, July 20, 2016 7:43:56 PM >> Subject: [Gluster-users] bitrot error correction in gluster 3.8 distributed >> dispersed setup >> >> Hi all, >> >> I'm testing the bitrot feature on a distributed dispersed setup: 3 x (8 >> + 4). >> I simulated a bit-flip on the part of a couple of files in a brick, >> having bitrot feature enabled before. The output from ''gluster volume >> bitrot vol_1 scrub status'' shows that two files are corrupted as shown >> below: >> >> >> --- >> Volume name : vol_1 >> >> State of scrub: Active >> >> Scrub impact: normal >> >> Scrub frequency: daily >> >> Bitrot error log location: /var/log/glusterfs/bitd.log >> >> Scrubber error log location: /var/log/glusterfs/scrub.log >> >> >> ========================================================>> >> Node: localhost >> >> Number of Scrubbed files: 564 >> >> Number of Skipped files: 0 >> >> Last completed scrub time: 2016-07-20 13:37:16 >> >> Duration of last scrub (D:M:H:M:S): 0:0:0:50 >> >> Error count: 0 >> >> >> ========================================================>> >> Node: glusterserver1 >> >> Number of Scrubbed files: 562 >> >> Number of Skipped files: 0 >> >> Last completed scrub time: 2016-07-20 13:37:16 >> >> Duration of last scrub (D:M:H:M:S): 0:0:0:51 >> >> Error count: 2 >> >> Corrupted object's [GFID]: >> >> e3cbffbe-12bd-4e1c-8dcf-2ef87ac99990 >> >> 2550475e-cf9e-4781-9a05-9e11c1dd6b7a >> >> >> ========================================================>> >> Node: glusterserver3 >> >> Number of Scrubbed files: 564 >> >> Number of Skipped files: 0 >> >> Last completed scrub time: 2016-07-20 13:37:16 >> >> Duration of last scrub (D:M:H:M:S): 0:0:0:50 >> --- >> >> >> I have executed ''gluster volume heal vol_1 full'' but it does not seem >> to fix the files, however the file that the client reads is the correct >> one. >> >> How should I handle this kind of corruption within gluster? >> >> Thanks! >> I?aki. >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> >>