Lustre discuss - May 2011 - df and du difference on lustre fs

Greetings,
I''ve got such strange situation:

We''ve installed lustre-client(2.0.0) and xen(4.0) on Debian squeeze. 
Then we loaded img files with virtual machine images on lustre (images 
were striped on 2 OSTs). VM started but then crashed. After that I tried 
to remove img files from lustre. Client hangs and xen server hangs too. 
After that I removed img files from another lustre client without any 
problems and reboot xen server. When I make df -h it seems that img 
files are on the lustre, but ls command doesn''t show deleted files and 
du -sm shows correct used space on lustre.

df -h /lustre/
Filesystem            Size  Used Avail Use% Mounted on
10.165.175.201 at tcp0:/lihep
                       5.4T   62G  5.1T   2% /lustre

du -sm /lustre/
41630    /lustre/

On OSTs :

# df -h
Filesystem            Size  Used Avail Use% Mounted on
...
/dev/md10             2.7T   32G  2.6T   2% /ost1

# df -h
Filesystem            Size  Used Avail Use% Mounted on
...
/dev/md13             2.7T   30G  2.6T   2% /ost2

How can I fix it ?

With regards,
Ekaterina Popova

Hello!

On May 26, 2011, at 8:19 AM, Ekaterina Popova wrote:
> We''ve installed lustre-client(2.0.0) and xen(4.0) on Debian
squeeze.
> Then we loaded img files with virtual machine images on lustre (images 
> were striped on 2 OSTs). VM started but then crashed. After that I tried 
> to remove img files from lustre. Client hangs and xen server hangs too. 
kernel messages from the hung instance would be interesting to see.
> After that I removed img files from another lustre client without any 
> problems and reboot xen server. When I make df -h it seems that img 
> files are on the lustre, but ls command doesn''t show deleted files
and
> du -sm shows correct used space on lustre.
I assume that the "hung" client was holding the file open, then when
you
unlinked the files from another client, only name got removed on MDS
and objects on OSTs remained.
> # df -h
> Filesystem            Size  Used Avail Use% Mounted on
> ...
> /dev/md13             2.7T   30G  2.6T   2% /ost2
> How can I fix it ?
Since you already killed the client holding the file open
once MDS registers the death of that client due to ping timeout,
it would close the descriptor and would proceed with destroying the
now orphan objects.

Alternatively next time your MDS restarts such orphaned objects should
also be destroyed.

Bye,
    Oleg
--
Oleg Drokin
Senior Software Engineer
Whamcloud, Inc.