Gluster users - Feb 2012 - "Structure needs cleaning" error

Hi,

We have recently upgraded our gluster to 3.2.5 and have
encountered the following error.  Gluster seems somehow
confused about one of the files it should be serving up,
specifically
/projects/philex/PE/2010/Oct18/arch07/BalbacFull_250_200_03Mar_3.png

If I go to that directory and simply do an ls *.png I get

ls: BalbacFull_250_200_03Mar_3.png: Structure needs cleaning

(along with a listing of the remaining *.png files)

I know which brick that file resides and if I go to
that brick and look at that file via the underlying nfs,
it is fine.  I had thought this might be a case of
bad meta-pointers, but there are no such pointers on the
other bricks.  In our gluster log file I see the
following error repeated

[2012-02-24 13:58:15.507390] E [client3_1-fops.c:2228:client3_1_lookup_cbk]
0-gdata-client-2: remote operation failed: Structure needs cleaning

[2012-02-24 13:58:15.507501] W [fuse-bridge.c:184:fuse_entry_cbk]
0-glusterfs-fuse: 176567027: LOOKUP()
/projects/philex/PE/2010/Oct18/arch07/BalbacFull_250_200_03Mar_3.png => -1
(Structure needs cleaning)

How do we fix this? Especially in general, since I don't
know if there are other files in with this error, this is
just the only one we have found so far.  Currently this
error is causing our back-up to fail.

Thanks.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-222B                   http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301

On 02/26/2012 12:54 PM, Patrick Haley wrote:
>
> Hi,
>
> We have recently upgraded our gluster to 3.2.5 and have
> encountered the following error.  Gluster seems somehow
> confused about one of the files it should be serving up,
> specifically
> /projects/philex/PE/2010/Oct18/arch07/BalbacFull_250_200_03Mar_3.png
>
> If I go to that directory and simply do an ls *.png I get
>
> ls: BalbacFull_250_200_03Mar_3.png: Structure needs cleaning
This is usually what happens when you have an underlying xfs file system 
as your backing store, and have had a failure which has shut the xfs 
file system down.

Have a look in dmesg for the servers, and see if xfs has shut down.

Reasons for xfs shutdown include a) bugs in xfs/kernel, b) storage 
device/RAID failure.

Our experience is that "b" happens far more often than "a",
though "a"
does happen (especially on some kernels).

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615