Lustre discuss - Nov 2010 - mkfs.lustre fails, ldiskfs: ext4 or ext3 ?

Hi all,

my attempt to format a new OST failed (mkfs.lustre: Unable to mount
/dev/sdd: Invalid argument), obviously because sdd has "device size
15253504MB", and the log tells me ''LDISKFS-fs does not support
filesystems greater than 8TB and can cause data corruption.''

However, this is a Lustre 1.8.4 installation, I have compiled the kernel
and modules myself and I have checked the ''modules/lustre/ldiskfs -
config.log'' to start with the following:

It was created by Lustre ldiskfs configure 3.1.3, which was
generated by GNU Autoconf 2.61.  Invocation command line was

  $ ./configure --prefix=/usr --with-linux=/usr/src/linux-2.6.27.39-0.3
--disable-quilt --disable-dependency-tracking --disable-doc
--disable-utils --disable-liblustre --enable-snmp --enable-ldiskfs
--disable-zerocopy --disable-tests --enable-quota CC=gcc-4.3
--enable-ext4 --with-lustre-hack --with-sockets --cache-file=/dev/null
--srcdir=.

All subsequent references to ext4 in this config.log seem to indicate
that my ldiskfs should be based on ext4 and thus be able to handle a
large device.

Is there a way to check ask the module itself about its capabilities?
Or is there perhaps a magic switch for mkfs.lustre?

Cheers,
Thomas

-- 
--------------------------------------------------------------------
Thomas Roth
Department: IT/HPC
GSI Darmstadt

On 2010-11-03, at 13:25, Thomas Roth wrote:
> my attempt to format a new OST failed (mkfs.lustre: Unable to mount
> /dev/sdd: Invalid argument), obviously because sdd has "device size
> 15253504MB", and the log tells me ''LDISKFS-fs does not
support
> filesystems greater than 8TB and can cause data corruption.''
> 
> However, this is a Lustre 1.8.4 installation, I have compiled the kernel
> and modules myself and I have checked the ''modules/lustre/ldiskfs
-
> config.log'' to start with the following:
> 
> It was created by Lustre ldiskfs configure 3.1.3, which was
> generated by GNU Autoconf 2.61.  Invocation command line was
> 
>  $ ./configure --prefix=/usr --with-linux=/usr/src/linux-2.6.27.39-0.3
> --disable-quilt --disable-dependency-tracking --disable-doc
> --disable-utils --disable-liblustre --enable-snmp --enable-ldiskfs
> --disable-zerocopy --disable-tests --enable-quota CC=gcc-4.3
> --enable-ext4 --with-lustre-hack --with-sockets --cache-file=/dev/null
> --srcdir=.
> 
> All subsequent references to ext4 in this config.log seem to indicate
> that my ldiskfs should be based on ext4 and thus be able to handle a
> large device.
The ext4 that is included in 2.6.27 is (AFAIR) missing patches that would allow
it to run with 16TB filesystems, or at least that was the case when we were
testing 16TB filesystems.  As a result, we only allow 16TB filesystems with
RHEL5.4 ext4-based ldiskfs, which was tested extensively.
> Is there a way to check ask the module itself about its capabilities?
> Or is there perhaps a magic switch for mkfs.lustre?
The code that limits the filesystem size to 8TB on untested platforms is in the
ext4-force_over_8tb-sles11.patch patch.  Either you can test with the mount
option "-O force_over_8tb", or rebuild the kernel without that patch.

There are tools included with Lustre called llverdev and llverfs that can be
used to do partial or full data integrity testing of large filesystems (and the
underlying block/SCSI/ATA drivers and h/w or s/w RAID).

Cheers, Andreas
--
Andreas Dilger
Lustre Technical Lead
Oracle Corporation Canada Inc.