Hi everyone, As some of you may know, the presence of files/directories which have different GFID's on different backends can cause the GlusterFS client to throw up errors, or even hang. Among others, we've had users of Enomaly affected by this issue. A little background on GFIDs: Each file/directory on a Gluster volume has a unique 128-bit number associated with it called the GFID. This is true regardless of Gluster configuration (distribute or distribute/replicate). One inode, one GFID. The GFID is stored on the backend as the value of the extended attribute "trusted.gfid". Under normal circumstances, the value of this attribute is the same on all the backend bricks. However, certain conditions can cause the value on one or more of the bricks to differ from that on the other bricks. This causes the GlusterFS client to become confused and throw up errors. This applies to both the 3.1.5 and 3.2.1 versions of the filesystem, and previous versions in those series. In a future bugfix release GlusterFS will fix this issue automatically when it detects it. Until then, if you encounter this problem, please use the following set of tools to manually fix it on the backends: https://github.com/vikasgorur/gfid The repository contains the tools as well as a README that explains how to use them. Your questions and comments are welcome. -- Vikas Gorur Engineer - Gluster -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20110708/849f82a7/attachment.html>
----------------------------------------------------------------------> Message: 1 > Date: Fri, 8 Jul 2011 15:54:19 -0700 > From: Vikas Gorur<vikas at gluster.com> > Subject: [Gluster-users] GFID mismatches and tools to fix them > To: gluster-users at gluster.org > Cc: gluster-devel at gluster.com > Message-ID: > <CAHVddbPm1YARBzMevK6hyN8U9yt-ayETg65i__SFMMNp0NvmMQ at mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Hi everyone, > > As some of you may know, the presence of files/directories which have > different GFID's on different backends can cause the GlusterFS client to > throw up errors, or even hang. Among others, we've had users of Enomaly > affected by this issue. > > A little background on GFIDs: > > Each file/directory on a Gluster volume has a unique 128-bit number > associated with it called the GFID. This is true regardless of Gluster > configuration (distribute or distribute/replicate). One inode, one GFID. > > The GFID is stored on the backend as the value of the extended attribute > "trusted.gfid". Under normal circumstances, the value of this attribute is > the same on all the backend bricks. However, certain conditions can cause > the value on one or more of the bricks to differ from that on the other > bricks. This causes the GlusterFS client to become confused and throw up > errors. This applies to both the 3.1.5 and 3.2.1 versions of the filesystem, > and previous versions in those series. > > In a future bugfix release GlusterFS will fix this issue automatically when > it detects it. Until then, if you encounter this problem, please use the > following set of tools to manually fix it on the backends: > > https://github.com/vikasgorur/gfid > > The repository contains the tools as well as a README that explains how to > use them. > > Your questions and comments are welcome. >Dear Vikas- Thanks for provding these tools. Unfortunately I think I have found a problem with the procedure outlined in the README - I don't think it works for files with names containing the colon character. I still have a lot of gfid errors in my logs after running the gfid tools on one volume, and all the filenames have one or more ':' characters. There are 1677 files still affected with "gfid different" so I don't think it can be a coincidence. Regards -Dan.
On 17 July 2011 18:05, Dan Bretherton <d.a.bretherton at reading.ac.uk> wrote: Dear Vikas->> Thanks for provding these tools. Unfortunately I think I have found a > problem with the procedure outlined in the README - I don't think it > works for files with names containing the colon character. I still have > a lot of gfid errors in my logs after running the gfid tools on one > volume, and all the filenames have one or more ':' characters. There > are 1677 files still affected with "gfid different" so I don't think it > can be a coincidence. >Thanks for pointing this out, Dan. There was some urgency in writing the tool and I forgot to document that it wouldn't handle files with a ":" in them. It'll be fixed soon. -- Vikas Gorur Engineer - Gluster -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20110718/b4c0b8ee/attachment.html>
> > I had a closer look at this. It is the output of gfid-mismatch causing > the problem; paths are shown with a trailing colon as in GlusterFS log > files. The "cut -f1 -d:" to extract the paths obviously removes all the > colons. I'm sure there is an easy way to remove the trailing ':' from > filenames but I can't think of one off hand (and it is 3:30AM).Dan, This has been fixed in the repository now. gfid-mismatch will only output the filename, which you can directly pass to gfid-delete. You can still get the verbose output by passing a '-v' flag. Please let me know if it works for you now. -- Vikas Gorur Engineer - Gluster -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20110720/257df126/attachment.html>