Hi everyone, As some of you may know, the presence of files/directories which have different GFID''s on different backends can cause the GlusterFS client to throw up errors, or even hang. Among others, we''ve had users of Enomaly affected by this issue. A little background on GFIDs: Each file/directory on a Gluster volume has a unique 128-bit number associated with it called the GFID. This is true regardless of Gluster configuration (distribute or distribute/replicate). One inode, one GFID. The GFID is stored on the backend as the value of the extended attribute "trusted.gfid". Under normal circumstances, the value of this attribute is the same on all the backend bricks. However, certain conditions can cause the value on one or more of the bricks to differ from that on the other bricks. This causes the GlusterFS client to become confused and throw up errors. This applies to both the 3.1.5 and 3.2.1 versions of the filesystem, and previous versions in those series. In a future bugfix release GlusterFS will fix this issue automatically when it detects it. Until then, if you encounter this problem, please use the following set of tools to manually fix it on the backends: https://github.com/vikasgorur/gfid The repository contains the tools as well as a README that explains how to use them. Your questions and comments are welcome. -- Vikas Gorur Engineer - Gluster -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gluster.org/pipermail/gluster-users/attachments/20110708/849f82a7/attachment.htm>
----------------------------------------------------------------------> Message: 1 > Date: Fri, 8 Jul 2011 15:54:19 -0700 > From: Vikas Gorur<vikas at gluster.com> > Subject: [Gluster-users] GFID mismatches and tools to fix them > To: gluster-users at gluster.org > Cc: gluster-devel at gluster.com > Message-ID: > <CAHVddbPm1YARBzMevK6hyN8U9yt-ayETg65i__SFMMNp0NvmMQ at mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Hi everyone, > > As some of you may know, the presence of files/directories which have > different GFID''s on different backends can cause the GlusterFS client to > throw up errors, or even hang. Among others, we''ve had users of Enomaly > affected by this issue. > > A little background on GFIDs: > > Each file/directory on a Gluster volume has a unique 128-bit number > associated with it called the GFID. This is true regardless of Gluster > configuration (distribute or distribute/replicate). One inode, one GFID. > > The GFID is stored on the backend as the value of the extended attribute > "trusted.gfid". Under normal circumstances, the value of this attribute is > the same on all the backend bricks. However, certain conditions can cause > the value on one or more of the bricks to differ from that on the other > bricks. This causes the GlusterFS client to become confused and throw up > errors. This applies to both the 3.1.5 and 3.2.1 versions of the filesystem, > and previous versions in those series. > > In a future bugfix release GlusterFS will fix this issue automatically when > it detects it. Until then, if you encounter this problem, please use the > following set of tools to manually fix it on the backends: > > https://github.com/vikasgorur/gfid > > The repository contains the tools as well as a README that explains how to > use them. > > Your questions and comments are welcome. >Dear Vikas- Thanks for provding these tools. Unfortunately I think I have found a problem with the procedure outlined in the README - I don''t think it works for files with names containing the colon character. I still have a lot of gfid errors in my logs after running the gfid tools on one volume, and all the filenames have one or more '':'' characters. There are 1677 files still affected with "gfid different" so I don''t think it can be a coincidence. Regards -Dan.
On 18/07/11 02:05, Dan Bretherton wrote:> ---------------------------------------------------------------------- >> Message: 1 >> Date: Fri, 8 Jul 2011 15:54:19 -0700 >> From: Vikas Gorur<vikas at gluster.com> >> Subject: [Gluster-users] GFID mismatches and tools to fix them >> To: gluster-users at gluster.org >> Cc: gluster-devel at gluster.com >> Message-ID: >> <CAHVddbPm1YARBzMevK6hyN8U9yt-ayETg65i__SFMMNp0NvmMQ at mail.gmail.com> >> Content-Type: text/plain; charset="iso-8859-1" >> >> Hi everyone, >> >> As some of you may know, the presence of files/directories which have >> different GFID''s on different backends can cause the GlusterFS client to >> throw up errors, or even hang. Among others, we''ve had users of Enomaly >> affected by this issue. >> >> A little background on GFIDs: >> >> Each file/directory on a Gluster volume has a unique 128-bit number >> associated with it called the GFID. This is true regardless of Gluster >> configuration (distribute or distribute/replicate). One inode, one GFID. >> >> The GFID is stored on the backend as the value of the extended attribute >> "trusted.gfid". Under normal circumstances, the value of this >> attribute is >> the same on all the backend bricks. However, certain conditions can >> cause >> the value on one or more of the bricks to differ from that on the other >> bricks. This causes the GlusterFS client to become confused and throw up >> errors. This applies to both the 3.1.5 and 3.2.1 versions of the >> filesystem, >> and previous versions in those series. >> >> In a future bugfix release GlusterFS will fix this issue >> automatically when >> it detects it. Until then, if you encounter this problem, please use the >> following set of tools to manually fix it on the backends: >> >> https://github.com/vikasgorur/gfid >> >> The repository contains the tools as well as a README that explains >> how to >> use them. >> >> Your questions and comments are welcome. >> > Dear Vikas- > Thanks for provding these tools. Unfortunately I think I have found a > problem with the procedure outlined in the README - I don''t think it > works for files with names containing the colon character. I still > have a lot of gfid errors in my logs after running the gfid tools on > one volume, and all the filenames have one or more '':'' characters. > There are 1677 files still affected with "gfid different" so I don''t > think it can be a coincidence. > > Regards > -Dan.I had a closer look at this. It is the output of gfid-mismatch causing the problem; paths are shown with a trailing colon as in GlusterFS log files. The "cut -f1 -d:" to extract the paths obviously removes all the colons. I''m sure there is an easy way to remove the trailing '':'' from filenames but I can''t think of one off hand (and it is 3:30AM). -Dan.
On Mon, Jul 18, 2011 at 03:48:11AM +0100, Dan Bretherton wrote:> I had a closer look at this. It is the output of gfid-mismatch > causing the problem; paths are shown with a trailing colon as in > GlusterFS log files. The "cut -f1 -d:" to extract the paths > obviously removes all the colons. I''m sure there is an easy way to > remove the trailing '':'' from filenames but I can''t think of one off > hand (and it is 3:30AM).Something along the lines of "sed ''s/.$//", as in: dog="doggy:"; echo $dog | sed ''s/.$//'' That would remove any last character. To just get ":": dog="doggy:"; echo $dog | sed ''s/:$//'' (No, I didn''t know that. I googled.) Whit
Or you can also use tr ":" "" to remove ":" On Sun, Jul 17, 2011 at 8:17 PM, Whit Blauvelt <whit.gluster at transpect.com> wrote:> On Mon, Jul 18, 2011 at 03:48:11AM +0100, Dan Bretherton wrote: > >> I had a closer look at this. ?It is the output of gfid-mismatch >> causing the problem; paths are shown with a trailing colon as in >> GlusterFS log files. ?The "cut -f1 -d:" to extract the paths >> obviously removes all the colons. ?I''m sure there is an easy way to >> remove the trailing '':'' from filenames but I can''t think of one off >> hand (and it is 3:30AM). > > Something along the lines of "sed ''s/.$//", as in: > > ? ? ? ?dog="doggy:"; echo $dog | sed ''s/.$//'' > > That would remove any last character. To just get ":": > > ? ? ? ?dog="doggy:"; echo $dog | sed ''s/:$//'' > > (No, I didn''t know that. I googled.) > > Whit > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >
On 17 July 2011 18:05, Dan Bretherton <d.a.bretherton at reading.ac.uk> wrote: Dear Vikas->> Thanks for provding these tools. Unfortunately I think I have found a > problem with the procedure outlined in the README - I don''t think it > works for files with names containing the colon character. I still have > a lot of gfid errors in my logs after running the gfid tools on one > volume, and all the filenames have one or more '':'' characters. There > are 1677 files still affected with "gfid different" so I don''t think it > can be a coincidence. >Thanks for pointing this out, Dan. There was some urgency in writing the tool and I forgot to document that it wouldn''t handle files with a ":" in them. It''ll be fixed soon. -- Vikas Gorur Engineer - Gluster -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gluster.org/pipermail/gluster-users/attachments/20110718/b4c0b8ee/attachment.htm>
공용준(yongjoon kong)/Cloud Computing 기술담당/SKCC
2011-Jul-19 09:03 UTC
[Gluster-users] how to rotate log of client side
Hi, I?m using gluster 3.1.4 Is there any way to rotate client side?s log It?s getting bigger every day. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gluster.org/pipermail/gluster-users/attachments/20110719/d8bf6e79/attachment.htm>
> > Is there any way to rotate client side?s log **** > > ** ** > > It?s getting bigger every day. **** > > ** >get the pid of client process> ** > > ** ># ps aux | grep glusterfs <----- to get the pid of the glusterfs client process move the existing log file to some other file. # mv <current-log-file>.log <some-name-you-choose>.log.1 send SIGHUP to the pid of client process. # kill -HUP $PID> >Your log rotation is complete. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gluster.org/pipermail/gluster-users/attachments/20110719/7d2cfc30/attachment.htm>
> Is there any way to rotate client side?s log > > It?s getting bigger every day.You can also configure logrotate - which is on most Linux systems - to handle this. "man logrotate". It''ll handle compression and deletion of older files, too. Whit
Amar (ಅಮರ್ ತುಂಬಳ್ಳಿ)
2011-Jul-19 12:12 UTC
[Gluster-users] how to rotate log of client side
FYI.. Check inline. 2011/7/19 Amar Tumballi <amar at gluster.com>> Is there any way to rotate client side?s log **** >> >> ** ** >> >> It?s getting bigger every day. **** >> >> ** >> > > get the pid of client process > >> ** >> >> ** >> > # ps aux | grep glusterfs <----- to get the pid of the glusterfs client > process > > move the existing log file to some other file. > > # mv <current-log-file>.log <some-name-you-choose>.log.1 > > send SIGHUP to the pid of client process. > > # kill -HUP $PID > >> >> > > Your log rotation is complete. > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gluster.org/pipermail/gluster-users/attachments/20110719/3c14935e/attachment-0001.htm>
Amar (ಅಮರ್ ತುಂಬಳ್ಳಿ)
2011-Jul-19 12:14 UTC
[Gluster-users] how to rotate log of client side
https://github.com/gluster/glusterfs/blob/master/extras/glusterfs-logrotate <https://github.com/gluster/glusterfs/blob/master/extras/glusterfs-logrotate>This is part of release tarball as of now in 3.2.2 (and future). Its not yet included in RPM files. Need more testing of these before getting this to mainstream. 2011/7/19 Amar (???? ????????) <amarts at gmail.com>> FYI.. Check inline. > > > 2011/7/19 Amar Tumballi <amar at gluster.com> > >> Is there any way to rotate client side?s log **** >>> >>> ** ** >>> >>> It?s getting bigger every day. **** >>> >>> ** >>> >> >> get the pid of client process >> >>> ** >>> >>> ** >>> >> # ps aux | grep glusterfs <----- to get the pid of the glusterfs client >> process >> >> move the existing log file to some other file. >> >> # mv <current-log-file>.log <some-name-you-choose>.log.1 >> >> send SIGHUP to the pid of client process. >> >> # kill -HUP $PID >> >>> >>> >> >> Your log rotation is complete. >> >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gluster.org/pipermail/gluster-users/attachments/20110719/bc6511a3/attachment.htm>
why is it getting too big? what is in it? On Jul 19, 2011, at 4:03 AM, ???(yongjoon kong)/Cloud Computing ????/SKCC wrote:> Hi, I?m using gluster 3.1.4 > > Is there any way to rotate client side?s log > > It?s getting bigger every day. > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-usersLuis E. Cerezo http://www.luiscerezo.org http://twitter.com/luiscerezo http://flickr.com/photos/luiscerezo photos for sale: http://photos.luiscerezo.org Voice: 412 223 7396 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gluster.org/pipermail/gluster-users/attachments/20110719/3a0937ed/attachment.htm>
공용준(yongjoon kong)/Cloud Computing 기술담당/SKCC
2011-Jul-20 00:13 UTC
[Gluster-users] how to rotate log of client side
Well, I adjust log level to TRACE it may be more than 3GB From: Luis Cerezo [mailto:lec at luiscerezo.org] Sent: Wednesday, July 20, 2011 12:27 AM To: ???(yongjoon kong)/Cloud Computing ????/SKCC Cc: gluster-users at gluster.org Subject: Re: [Gluster-users] how to rotate log of client side why is it getting too big? what is in it? On Jul 19, 2011, at 4:03 AM, ???(yongjoon kong)/Cloud Computing ????/SKCC wrote: Hi, I?m using gluster 3.1.4 Is there any way to rotate client side?s log It?s getting bigger every day. _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org<mailto:Gluster-users at gluster.org> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users Luis E. Cerezo http://www.luiscerezo.org http://twitter.com/luiscerezo http://flickr.com/photos/luiscerezo photos for sale: http://photos.luiscerezo.org Voice: 412 223 7396 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gluster.org/pipermail/gluster-users/attachments/20110720/53a49645/attachment.htm>
ah, that''s not so bad. On Jul 19, 2011, at 7:13 PM, ???(yongjoon kong)/Cloud Computing ????/SKCC wrote:> Well, I adjust log level to TRACE it may be more than 3GB > > From: Luis Cerezo [mailto:lec at luiscerezo.org] > Sent: Wednesday, July 20, 2011 12:27 AM > To: ???(yongjoon kong)/Cloud Computing ????/SKCC > Cc: gluster-users at gluster.org > Subject: Re: [Gluster-users] how to rotate log of client side > > why is it getting too big? what is in it? > > > On Jul 19, 2011, at 4:03 AM, ???(yongjoon kong)/Cloud Computing ????/SKCC wrote: > > > Hi, I?m using gluster 3.1.4 > > Is there any way to rotate client side?s log > > It?s getting bigger every day. > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > > > Luis E. Cerezo > > http://www.luiscerezo.org > http://twitter.com/luiscerezo > http://flickr.com/photos/luiscerezo > photos for sale: > http://photos.luiscerezo.org > Voice: 412 223 7396 >Luis E. Cerezo http://www.luiscerezo.org http://twitter.com/luiscerezo http://flickr.com/photos/luiscerezo photos for sale: http://photos.luiscerezo.org Voice: 412 223 7396 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gluster.org/pipermail/gluster-users/attachments/20110719/3203e9f5/attachment.htm>
> > I had a closer look at this. It is the output of gfid-mismatch causing > the problem; paths are shown with a trailing colon as in GlusterFS log > files. The "cut -f1 -d:" to extract the paths obviously removes all the > colons. I''m sure there is an easy way to remove the trailing '':'' from > filenames but I can''t think of one off hand (and it is 3:30AM).Dan, This has been fixed in the repository now. gfid-mismatch will only output the filename, which you can directly pass to gfid-delete. You can still get the verbose output by passing a ''-v'' flag. Please let me know if it works for you now. -- Vikas Gorur Engineer - Gluster -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gluster.org/pipermail/gluster-users/attachments/20110720/257df126/attachment.htm>
On 20/07/11 19:46, Vikas Gorur wrote:> > I had a closer look at this. It is the output of gfid-mismatch > causing > the problem; paths are shown with a trailing colon as in GlusterFS log > files. The "cut -f1 -d:" to extract the paths obviously removes > all the > colons. I''m sure there is an easy way to remove the trailing '':'' from > filenames but I can''t think of one off hand (and it is 3:30AM). > > > Dan, > > This has been fixed in the repository now. gfid-mismatch will only > output the filename, which you can directly pass to gfid-delete. You > can still get the verbose output by passing a ''-v'' flag. > > Please let me know if it works for you now. > > -- > Vikas Gorur > Engineer - Gluster > >Hello Vikas, Yes it does work now for all files - thanks. Regards Dan. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gluster.org/pipermail/gluster-users/attachments/20110723/afaac27e/attachment.htm>