Gluster users - Jul 2013 - Weird Glusterfs problem

Hello Gluster gurus,

I'm experimenting with Gluster and I've run across a problem that I
don't understand.

I have Ubuntu 12.04 with Gluster 3.2.5.  I created a volume and filled
it with data.  Then I left an rm -fr going to wipe it out over night. 
When I came in this morning, I found a lot of stuff like this:

rm: cannot remove `old/old.5/test-host/20130707/etc/init.d/apport':
Stale NFS file handle

The log file data-testvol.log shows:

[2013-07-09 10:13:37.229796] W [stat-prefetch.c:2632:sp_unlink_helper]
0-testvol-stat-prefetch: lookup-behind has failed for path
(/old/old.5/test-host/20130707/etc/init.d/apport)(Stale NFS file
handle), unwinding unlink call waiting on it
[2013-07-09 10:13:37.229811] W [fuse-bridge.c:911:fuse_unlink_cbk]
0-glusterfs-fuse: 105965: UNLINK()
/old/old.5/test-host/20130707/etc/init.d/apport => -1 (Stale NFS file
handle)

The interesting thing is that if I delete
old/old.5/test-host/20130707/etc/init.d/apport by hand, it works:

root at gluster-server-a:/data/testvol# cd
old/old.5/test-host/20130707/etc/init.d
root at gluster-server-a:/data/testvol/old/old.5/test-host/20130707/etc/init.d#
rm apport

This seems to have worked for files, and for some directories.  But for
other directories there's still a problem.  Gluster seems to think that
some directories are not empty, even though there is nothing inside:

root at gluster-server-a:/data/testvol/old/old.5/[...stuff...]/Maildir# ls -al
total 72
drwxr-xrwx 3 root root 16384 Jul  8 23:46 .
drwxr-xrwx 3 root root 16384 Jul  9 00:10 ..
drwxr-xr-x 2 root root 16384 Jul  9 10:27 .Old Mail.Mailbox12092004_end
root at gluster-server-a:/data/bkupc1/old/old.5/[...stuff...]/Maildir# cd
.Old\ Mail.Mailbox12092004_end/
root at gluster-server-a:/data/bkupc1/old/old.5/[...stuff...]/Maildir/.Old
Mail.Mailbox12092004_end# ls -al
total 48
drwxr-xr-x 3 root root 16384 Jul  9 10:27 .
drwxr-xrwx 3 root root 16384 Jul  8 23:46 ..
root at gluster-server-a:/data/bkupc1/old/old.5/[...stuff...]/Maildir/.Old
Mail.Mailbox12092004_end# cd ..
root at gluster-server-a:/data/bkupc1/old/old.5/[...stuff...]/Maildir#
rmdir .Old\ Mail.Mailbox12092004_end/
rmdir: failed to remove `.Old Mail.Mailbox12092004_end/': Directory not
empty
root at gluster-server-a:/data/bkupc1/old/old.5/[...stuff...]/Maildir# rm
-fr .Old\ Mail.Mailbox12092004_end/
rm: cannot remove `.Old Mail.Mailbox12092004_end': Directory not empty

Next I rebooted every node in the cluster.  The problem persists.

Anyone have an idea what's going on?  Or how to fix it?

Michael

It seems that if I manually delete the problem file/directory from each
brick directly, the problem goes away.

I'm still wondering what caused this, and how to avoid it in production.

Michael

On 07/09/2013 10:35 AM, Michael Peek wrote:
> Hello Gluster gurus,
>
> I'm experimenting with Gluster and I've run across a problem that I
> don't understand.
>
> I have Ubuntu 12.04 with Gluster 3.2.5.  I created a volume and filled
> it with data.  Then I left an rm -fr going to wipe it out over night. 
> When I came in this morning, I found a lot of stuff like this:
>
> rm: cannot remove `old/old.5/test-host/20130707/etc/init.d/apport':
> Stale NFS file handle
>
> The log file data-testvol.log shows:
>
> [2013-07-09 10:13:37.229796] W [stat-prefetch.c:2632:sp_unlink_helper]
> 0-testvol-stat-prefetch: lookup-behind has failed for path
> (/old/old.5/test-host/20130707/etc/init.d/apport)(Stale NFS file
> handle), unwinding unlink call waiting on it
> [2013-07-09 10:13:37.229811] W [fuse-bridge.c:911:fuse_unlink_cbk]
> 0-glusterfs-fuse: 105965: UNLINK()
> /old/old.5/test-host/20130707/etc/init.d/apport => -1 (Stale NFS file
> handle)
>