Lustre discuss - Sep 2009 - odd issue with deleting files and folders

Hi,

We have 50 nodes each of which has a lustre partition mounted on 
/scratch. Prior to using lustre we used NFS so the head node actually 
passes lustre traffic through to the lustre servers.

We run a particular job on these nodes where they are each running a job 
which populates a directory with files and directories, in total 270 
files and directories will be created over a very short period of time, 
each node produces a different number of files and directories. At the 
end we remove the directory.

In the listings below we see some really odd output

nitrogen:/scratch/007$ ls
data.input  data.target
nitrogen:/scratch/007$ ls -l
total 245916
-rw-rw-rw-  1 megan  users  250596016 Sep 11 15:39 data.input
-rw-rw-rw-  1 megan  users    1199024 Sep 11 15:39 data.target
nitrogen:/scratch/007$ rm -rf *
nitrogen:/scratch/007$ ls -l
total 16
nitrogen:/scratch/007$ ls -l
total 16
-rw-rw-r--  1 zomojo zomojo     0 Sep 11  2009 tree_000171.csv
nitrogen:/scratch/007$ ls -l
total 16
-rw-rw-r--  1 zomojo zomojo     0 Sep 11  2009 tree_000171.csv
nitrogen:/scratch/007$ ls
tree_000152.csv  tree_000171.csv
nitrogen:/scratch/007$ ls -l
total 16
-rw-rw-r--  1 zomojo zomojo     0 Sep 11 15:52 tree_000152.csv
-rw-rw-r--  1 zomojo zomojo     0 Sep 11  2009 tree_000171.csv
nitrogen:/scratch/007$ ls -l
total 16
-rw-rw-r--  1 zomojo zomojo     0 Sep 11 15:52 tree_000152.csv
-rw-rw-r--  1 zomojo zomojo     0 Sep 11  2009 tree_000171.csv
nitrogen:/scratch/007$

later, on another client the output looks like

 ls -l /scratch/007
total 0
-rw-rw-r-- 1 zomojo zomojo 0 Sep 11  2009 tree_000138.csv
-rw-rw-r-- 1 zomojo zomojo 0 Sep 11 16:08 tree_000150.csv
-rw-rw-r-- 1 zomojo zomojo 0 Sep 11 15:52 tree_000152.csv
-rw-rw-r-- 1 zomojo zomojo 0 Sep 11 15:59 tree_000165.csv
-rw-rw-r-- 1 zomojo zomojo 0 Sep 11  2009 tree_000171.csv
-rw-rw-r-- 1 zomojo zomojo 0 Sep 11  2009 tree_000175.csv
-rw-rw-r-- 1 zomojo zomojo 0 Sep 11 16:08 tree_000177.csv
-rw-rw-r-- 1 zomojo zomojo 0 Sep 11  2009 tree_000186.csv
-rw-rw-r-- 1 zomojo zomojo 0 Sep 11 16:08 tree_000187.csv

So there are a couple of issues here, one is how come the rm -rf did not 
complete, why are we seeing such differences in output between clients 
and I know the times are not shown here but why does it take so long to 
update ?

We are running lustre 1.6.7.2 on the clients and 1.6.7.1 on the 
servers.  All servers and clients are running CentOS5.3

Thanks

Phil.