Gluster users - Jun 2012 - unable to delete 'empty' dirs with 3.3qa42

Just before updrading to 3.3final, we had an rsync collision on our 
gluster filesystem which left us with undeletable dirs.

The transport is IPoIB over 4 bricks as shown below.

$ gluster volume info

Volume Name: gli
Type: Distribute
Volume ID: 76cc5e88-0ac4-42ac-a4a3-31bf2ba611d4
Status: Started
Number of Bricks: 4
Transport-type: tcp,rdma
Bricks:
Brick1: pbs1ib:/bducgl
Brick2: pbs2ib:/bducgl
Brick3: pbs3ib:/bducgl
Brick4: pbs4ib:/bducgl
Options Reconfigured:
performance.io-cache: on
performance.quick-read: on
performance.io-thread-count: 64

The last 100 line of the gluster log is here: <http://pastie.org/4036243>

The problem started when I was running an rsync with --delete while 
moving a large /home to the gluster dir to prune the target of older 
cruft in a /home move..  A user directed a job to start reading from a 
directory that was simultaneously being pruned and all hell broke loose.

Now we're left with a number of dirs and files that can't be deleted.  
'du' says there are files in an apparently empty dir:

  root at bduc-login:/gl/tvanerp/adni
561 $ du -sh *
40K     dicom

altho 'ls' can't see them from the client;

root at bduc-login:/gl/tvanerp/adni
562 $ ls -lat dicom
total 40
drwxrwxr-x 3 tvanerp tvanerp 94214 Jun  5 23:40 ./
drwxrwxr-x 3 tvanerp tvanerp    72 Apr 23 13:11 ../


 From the bricks you can see that there are still files there:
Tue Jun 05 23:18:02 [3.06 2.42 1.65][457.48/606]  root at bduc-login:~
556 $ ssh pbs2 'ls -lat /bducgl/tvanerp/adni/dicom'
total 24
drwxrwxr-x 3 7335 7335 28672 2012-06-05 22:59 .
drwxr-xr-x 2 7335 7335  8192 2012-06-05 14:50 128_S_0947_20080422_MPRAGE
drwxrwxr-x 3 7335 7335    18 2012-04-23 13:11 ..

Tue Jun 05 23:18:26 [2.82 2.42 1.67][457.62/606]  root at bduc-login:~
557 $ ssh pbs3 'ls -lat /bducgl/tvanerp/adni/dicom'
total 28
drwxrwxr-x 3 7335 7335 28672 2012-06-05 23:00 .
drwxr-xr-x 2 7335 7335 12288 2012-06-05 14:52 128_S_0947_20080422_MPRAGE
drwxrwxr-x 3 7335 7335    18 2012-04-23 13:11 ..

Tue Jun 05 23:18:51 [2.73 2.43 1.69][457.7/606]  root at bduc-login:~
558 $ ssh pbs4 'ls -lat /bducgl/tvanerp/adni/dicom'
total 32
drwxrwxr-x 3 7335 7335 36864 2012-06-05 22:59 .
drwxr-xr-x 2 7335 7335 12288 2012-06-05 14:50 128_S_0947_20080422_MPRAGE
---------T 2 root root     0 2012-05-20 18:35 adni_pib_subjects.txt
drwxrwxr-x 3 7335 7335    18 2012-04-23 13:11 ..

but the client is not able to see/delete them, even as root.

Suggestions?

harry




-- 
Harry Mangalam,Research Computing, OIT,
Rm 225 MSTB, UC Irvine [mailcode 2225]
Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
Lat/Long: (33.642025,-117.844414) [paste into Google Maps]
--

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120605/8d9fd407/attachment.html>