Lustre discuss - May 2008 - free space of Lustre fs: unknown quantity?

Hi all,

on our test cluster I observe a rather large difference between "df"
and
"lfs df":

 >  lfs df  /lustre
 >  UUID                                 1K-blocks                Used 
                    Available    Use%   Mounted on
 >  ...
 >  filesystem summary:  137454163496     7056126648     130398036848 
  5%      /lustre

vs.
 > df  /lustre
 >  Filesystem                        1K-blocks                   Used 
            Available     Use%   Mounted on
 > mds at tcp0:/lust1         137454163496         73850360   130398036848 
   1%     /lustre

Using the human readable form, df -h, shows what this means: lfs 
believes that 6.6TB are in use, while the system thinks it''s only 71GB.
And of course, total and available space are equal.
I know that these tools for checking free and used disk space are 
notoriously imprecise. Not surprisingly then, "du -hs /lustre" gives
me
just 47 GB. I think I''d like to stick to "du" - gives best
looking
results ;-)

So, any ideas on what goes wrong here? Does something go wrong at all?

Oh, btw,  it is Lustre version 1.6.4.3 under Debian Etch 64 on this 
system. And I can observe this behavior on several clients, also after 
unmounting and remounting.

Regards,
Thomas

On May 07, 2008  17:57 +0200, Thomas Roth wrote:
> on our test cluster I observe a rather large difference between
"df" and
> "lfs df":
> 
>  >  lfs df  /lustre
>  >  UUID                                 1K-blocks                Used 
>                     Available    Use%   Mounted on
>  >  ...
>  >  filesystem summary:  137454163496     7056126648     130398036848 
>   5%      /lustre
> 
> vs.
>  > df  /lustre
>  >  Filesystem                        1K-blocks                   Used 
>             Available     Use%   Mounted on
>  > mds at tcp0:/lust1         137454163496         73850360  
130398036848
>    1%     /lustre
> 
> Using the human readable form, df -h, shows what this means: lfs 
> believes that 6.6TB are in use, while the system thinks it''s only
71GB.
> And of course, total and available space are equal.
> I know that these tools for checking free and used disk space are 
> notoriously imprecise. Not surprisingly then, "du -hs /lustre"
gives me
> just 47 GB. I think I''d like to stick to "du" - gives
best looking
> results ;-)
> 
> So, any ideas on what goes wrong here? Does something go wrong at all?
This is bug 14283, and is fixed in 1.6.5.  The problem is described best at:

https://bugzilla.lustre.org/show_bug.cgi?id=14283#c2

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.