Gluster users - Apr 2013 - Odd error with older Gluster/NFS3 tail swallowing

Had some data loss with an older - 3.1.4 - Gluster share last night. Now
trying to see what the best lessons are to learn from it. Obviously it's too
old a version for a bug report to matter. Wondering if anyone recognizes
this particular sort of error condition though. 

It's a 300 G replicated share, mounted by Gluster's NFS3 to several
systems.
It was getting fuller than we like it to be, at over 80%, so I copied a
directory containg 26 G off of it. Checked the copy and it was good. Then I
went and "rm -r"'d that directory. After a few minutes it
complained "Cannot
delete directory, directory not empty," citing a subdirectory. Strange.

So I stopped the process and looked in that subdirectory. The subdirectory
had within it ... the whole of the Gluster share. Damn. Yes, the "rm
-r"
had, due to this illusion, managed to wipe out over half of the share
because it had descended into other directories at its root level via this
illusion. 

And it's an illusion. It wasn't some bizarre tail-eating self-mounting.
The
copy of the directory shows it populated by three subdirectories with a few
files in them. The version of the directory on the Gluster NFS share has
none of those three. Instead it has the whole of the root-level directies of
the share visible there. Both of the backing shares also have those three
directories properly populated - no tail-eating mount. Another system with
the same NFS3 mount of this share shows those three directories properly. So
it's an illusion based in some error in how Gluster and/or the NFS client is
presenting the directory structure on the one client. The backing stores are
ext4; the kernel is old enough to be fully compatible with Gluster in ext4.
Anyway, no apparent error at that level.

I'm a generalist as a sysadmin, in a smallish shop, half-ignorant on
filesystems. Aside from putting more weight on the "pursuing the cutting
edge may sometimes be safer than staying with the apparent time-proven
stability of an older version" side of the perennial debate between
progressive and conservative approaches, what should I learn from this? The
NFS client is running a 2.6.24 kernel - yeah, that's old, but it's been
very
reliable up until this. Is this some known NFS client problem since fixed?
Or a really-bizarre one-off?

If it is something that can still "just happen," what's safe
procedure?
Looking through every subdirectory of a tree about to be deleted to make
sure it hasn't in a virtual way and without anything that shows to
"mount"
mounted the whole filesystem its within to itself seems much. I did have,
fortunately, an rsnapshot backup of the whole thing, so have been able to
restore. But I'd like to avoid the whole experience next time. What's
the
wisest way to go?

Thanks
Whit

On Tue, Apr 09, 2013 at 09:44:26AM -0400, Whit Blauvelt
wrote:
> Had some data loss with an older - 3.1.4 - Gluster share last night. Now
> trying to see what the best lessons are to learn from it. Obviously
it's too
> old a version for a bug report to matter. Wondering if anyone recognizes
> this particular sort of error condition though. 
> 
> It's a 300 G replicated share, mounted by Gluster's NFS3 to several
systems.
> It was getting fuller than we like it to be, at over 80%, so I copied a
> directory containg 26 G off of it. Checked the copy and it was good. Then I
> went and "rm -r"'d that directory. After a few minutes it
complained "Cannot
> delete directory, directory not empty," citing a subdirectory.
Strange.
> 
> So I stopped the process and looked in that subdirectory. The subdirectory
> had within it ... the whole of the Gluster share. Damn. Yes, the "rm
-r"
> had, due to this illusion, managed to wipe out over half of the share
> because it had descended into other directories at its root level via this
> illusion. 
Would the likely culprit in this sort of error in the appearance of the
filesystem from the NFS client likely be failing RAM on the client? Is the
scheme something along the lines of there being a base address for the root
of the NFS mount, with addresses within the mount being at an offset from
that, so that loss of the offset for the address of a subdirectory could
result in the subdirectory seeming itself to contain the whole NFS mount?

Obviously I have no knowledge on this level. Any of your filesystem gurus
seen this before, or have a hypothesis?

Thanks,
Whit