Lustre discuss - May 2006 - df reports wrong sizes with lustre file system greater 8Tb

On Mar 01, 2005  16:24 -0700, Monk, Stephen wrote:
> When I create a lustre file system greater than ~8TB df reports back
> strange info.
> 
> For example with a ~15 TB file system (15 OST''s)
> 
> Filesystem            Size  Used Avail Use% Mounted on
> config                    64Z  8.2G -1.1T   0% /scratch1
> 
> ~8 TB shows:
> config                 64Z  3.2G  8.0T   1% /scratch1
> 
> ~7 TB shows:
> config                7.4T  2.8G  7.0T   1% /scratch1
> 
> This is with the stock  2.4.21-27.0.2.EL_lustre.1.4.0.10 kernel rpms
> from the CFS download site.
> 
> I don''t see this with a system that is running lustre 1.4.0 and
the
> 2.4.21-20.0.1.EL kernel.
There was a change in the RHEL kernel to support statfs64, and this
apparently has caused problems because of signed->unsigned conversion.

Lustre scales the large statfs results to fit into a 32-bit value, since
most kernels do not support a full 64-bit result and also the filesystem
doesn''t know if it is being called with a 32-bit or 64-bit interface.
This is preferential to just returning "EOVERFLOW" for the majority of
Lustre filesystems that are so large...

Could you try:

--- lustre/llite/llite_lib.c    1 Mar 2005 09:17:30 -0000       1.20.2.15.2.16
+++ lustre/llite/llite_lib.c    2 Mar 2005 06:16:58 -0000
@@ -1143,7 +1143,7 @@ int ll_statfs(struct super_block *sb, st
         statfs_unpack(sfs, &osfs);
  
         if (sizeof(sfs->f_blocks) == 4) {
-                while (osfs.os_blocks > ~0UL) {
+                while (osfs.os_blocks > ~0UL >> 1) {
                         sfs->f_bsize <<= 1;
  
                         osfs.os_blocks >>= 1;


Cheers, Andreas
--
Andreas Dilger

A bit more info. on this...

-Andreas the patch below did not change the behavior of df....I could
have done something
wrong but the patch did take and lustre binaries built ok...:(

	Could you try:

	--- lustre/llite/llite_lib.c    1 Mar 2005 09:17:30 -0000
1.20.2.15.2.16
	+++ lustre/llite/llite_lib.c    2 Mar 2005 06:16:58 -0000
	@@ -1143,7 +1143,7 @@ int ll_statfs(struct super_block *sb, st
         statfs_unpack(sfs, &osfs);
 =20
         if (sizeof(sfs->f_blocks) =3D=3D 4) {
	-                while (osfs.os_blocks > ~0UL) {
	+                while (osfs.os_blocks > ~0UL >> 1) {
                         sfs->f_bsize <<=3D 1;
 =20
   	                      osfs.os_blocks >>=3D 1;

-Jeffery Baker (jwbaker@acm.org) said:

	"64Z" is just libc and printf flipping out with large values on
32-bit systems.  	This is unrelated to lustre, I think.

This may be the case as I installed this same RHEL-3 u4 kernel on a
RHEL-3 u3 system with a slightly older version of libc and df reports as
it should...see below.

-u4 (update 4 system) with u4 lustre kernel:

[smonk@ga623 ~]$ cat /etc/redhat-release=20
Red Hat Enterprise Linux WS release 3 (Taroon Update 4)
[smonk@ga623 ~]$ uname -a
Linux ga623 2.4.21-27.0.2.EL_lustre.1.4.0.10smp #1 SMP Tue Feb 15
03:09:59 EST 2005 i686 i686 i386 GNU/Linux
[smonk@ga623 ~]$ ldd /bin/df
        libc.so.6 =3D> /lib/tls/libc.so.6 (0x00e6a000)
        /lib/ld-linux.so.2 =3D> /lib/ld-linux.so.2 (0x00b95000)
[smonk@ga623 ~]$ rpm -qf /lib/tls/libc.so.6=20
glibc-2.3.2-95.30
[smonk@ga623 ~]$ rpm -qf /lib/ld-linux.so.2=20
glibc-2.3.2-95.30
[smonk@ga623 ~]$ df -h /scratch1
Filesystem            Size  Used Avail Use% Mounted on
config                 64Z  8.2G -1.1T   0% /scratch1


-U3 system with u4 lustre kernel:

[smonk@ca890 ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux WS release 3 (Taroon Update 3)
[smonk@ca890 ~]$ uname -a
Linux ca890 2.4.21-27.0.2.EL_lustre.1.4.0.10smp #1 SMP Tue Feb 15
03:09:59 EST 2005 i686 i686 i386 GNU/Linux
[smonk@ca890 ~]$ ldd /bin/df
        libc.so.6 =3D> /lib/tls/libc.so.6 (0x00584000)
        /lib/ld-linux.so.2 =3D> /lib/ld-linux.so.2 (0x00df8000)
[smonk@ca890 ~]$ rpm -qf /lib/tls/libc.so.6
glibc-2.3.2-95.27
[smonk@ca890 ~]$ rpm -qf /lib/ld-linux.so.2
glibc-2.3.2-95.27=20
[smonk@ca890 ~]$ df -h /scratch1
Filesystem            Size  Used Avail Use% Mounted on
config                 16T  8.2G   15T   1% /scratch1

So it appears to be an issue with the libc from RHEL-3 U4...I''ll see
what RedHat has to say.

Thanks to all for your help,
Steve

When I create a lustre file system greater than ~8TB df reports back
strange info.

For example with a ~15 TB file system (15 OST''s)

Filesystem            Size  Used Avail Use% Mounted on
config                    64Z  8.2G -1.1T   0% /scratch1

~8 TB shows:
config                 64Z  3.2G  8.0T   1% /scratch1

~7 TB shows:
config                7.4T  2.8G  7.0T   1% /scratch1

This is with the stock  2.4.21-27.0.2.EL_lustre.1.4.0.10 kernel rpms
from the CFS download site.

I don''t see this with a system that is running lustre 1.4.0 and the
2.4.21-20.0.1.EL kernel.

Just curious if anyone else has seen this...I''m not sure if this is a
lustre issue vs. a kernel issue.

Steve