Gluster users - May 2010 - Bizarre GlusterFS "coherency" issue

Hi,

I've been chasing down a GlusterFS issue for a while to no avail. We use
2.0.9, restarting client glusterfs does not fix the problem, restarting
server glusterfs does fix the problem. I wonder if this issue is a bug; if
it is a bug, is it fixed in 3.0.4; if it is not fixed, is there a
work-around other than restarting glusterfs server.

*Setup*: 2-node cluster: one glusterfs server, the other as client.
*What I did*: on the server node, remove a tree of files then add the same
tree back through untar a tarball
*Problem*: server can see the tree fine, but sometimes client cannot.

Relevant server log messages:
[2010-05-11 12:51:00] D [server-dentry.c:242:__do_path_resolve_cbk] brick:
looked up for /web-apps/testphp/temp-production/info/control
(329772/control)
[2010-05-11 12:51:00] D [server-helpers.c:96:server_loc_fill] brick: paths
differ for inode(329938): client path
/web-apps/testphp/temp-production/info/control. dentry path
/logging/staged/scraper.1273607423.2.dat/temp-production/info/control
[2010-05-11 12:51:00] E [posix.c:277:setgid_override] [storage/posix]: lstat
on parent directory
(/gluster-srv/logging/staged/scraper.1273607423.2.dat/temp-production/info)
failed: Not a directory
[2010-05-11 12:51:00] D [server-protocol.c:2016:server_open_cbk] server:
2968: OPEN
/logging/staged/scraper.1273607423.2.dat/temp-production/info/control
(329938) ==> -20 (Success)

Relevant client log messages:
[2010-05-11 12:51:00] D [client-protocol.c:4913:client_lookup_cbk] remote:
LOOKUP 327048/testphp (/web-apps/testphp): inode number changed from 329508
to 329535
[2010-05-11 12:51:00] D [fuse-bridge.c:302:need_fresh_lookup] fuse-bridge:
revalidate of /web-apps/testphp failed *(Stale NFS file handle)*
[2010-05-11 12:51:00] W [fuse-bridge.c:645:fuse_fd_cbk] glusterfs-fuse:
7201: OPEN() /web-apps/testphp/temp-production/info/control => -1 (Success)

Appreciate any insight.
Lei

Attached are our glusterfs configuration. This seems a timing issue, I still
do not have a simple script to show the problem, it is reproducible through
our pylons based webapp - not sure if that is relevant.

Appreciate ideas...

On May 11, 2010, at 1:51 PM, Lei Zhang wrote:
> Hi,
> 
> I've been chasing down a GlusterFS issue for a while to no avail. We
use
> 2.0.9, restarting client glusterfs does not fix the problem, restarting
> server glusterfs does fix the problem. I wonder if this issue is a bug; if
> it is a bug, is it fixed in 3.0.4; if it is not fixed, is there a
> work-around other than restarting glusterfs server.
This looks like it is related to inode numbers being reused. Inode number
handling was vastly improved in the 3.x releases and it fixes many issues like
this one. Please try 3.0.4 and let us know how it goes.

--
Vikas Gorur
Engineer - Gluster

"It is a capital mistake to theorize before one has data"
	-- Sherlock Holmes