Ext3 users - Mar 2004 - impossible to create a ext3 filesystem on a LVM2 Logical Volume

Dears,

I'm faced with the above mentionned problem.

System is a HP Proliant DL380 with HP SmartArray 5304-256 Controller and 
a HP StorageWorks 4300 disk shelf.
OS is Debian/testing with kernel 2.6.3 and LVM2.

I've created a new LV and when I try to create the FS on it, it fails 
when saying "Writing superblocks and filesystem accounting
information:"
sar output shows that CPU is in 100% wio.

Only way to solve it is to perform a hard reboot (killing the process 
does nothing).

I've performed a strace of the 'mkfs.ext3' and last system call is a
call to fsync (see hereafter as small part of the trace file)

Any help appreciated !

Paul

write(1, "done\n", 5done
)                   = 5
write(1, "Writing superblocks and filesyst"..., 59Writing superblocks 
and filesystem accounting information: ) = 59
time(NULL)                              = 1080538375
lseek(3, 24576, SEEK_SET)               = 24576
write(3, "\0\0\0\0\0\0\0\0\6\265g@\6\265g@\6\265g@\0\0\0\0\0\0\0"..., 
4096) = 4096
lseek(3, 35725312, SEEK_SET)            = 35725312
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
lseek(3, 6385664, SEEK_SET)             = 6385664
write(3, "\30\6\0\0\31\n\0\0\32\16\0\0\33\22\0\0\34\26\0\0\35\32"..., 
4096) = 4096
lseek(3, 35713024, SEEK_SET)            = 35713024
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
lseek(3, 35717120, SEEK_SET)            = 35717120
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
lseek(3, 35721216, SEEK_SET)            = 35721216
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
lseek(3, 35729408, SEEK_SET)            = 35729408
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
lseek(3, 31580160, SEEK_SET)            = 31580160
write(3, "\37\36\0\0
\36\0\0!\36\0\0\"\36\0\0#\36\0\0$\36\0\0%\36"...,
4096) = 4096
lseek(3, 1024, SEEK_SET)                = 1024
........
_llseek(3, 47915732992, [47915732992], SEEK_SET) = 0
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
_llseek(3, 48049950720, [48049950720], SEEK_SET) = 0
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
_llseek(3, 48184168448, [48184168448], SEEK_SET) = 0
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
_llseek(3, 48318386176, [48318386176], SEEK_SET) = 0
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
_llseek(3, 48452603904, [48452603904], SEEK_SET) = 0
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
_llseek(3, 48586821632, [48586821632], SEEK_SET) = 0
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
_llseek(3, 48721039360, [48721039360], SEEK_SET) = 0
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
_llseek(3, 48855257088, [48855257088], SEEK_SET) = 0
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
_llseek(3, 49392128000, [49392128000], SEEK_SET) = 0
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
_llseek(3, 49526345728, [49526345728], SEEK_SET) = 0
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
_llseek(3, 49660563456, [49660563456], SEEK_SET) = 0
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
_llseek(3, 49794781184, [49794781184], SEEK_SET) = 0
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
_llseek(3, 49928998912, [49928998912], SEEK_SET) = 0
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
_llseek(3, 48989474816, [48989474816], SEEK_SET) = 0
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
_llseek(3, 49123692544, [49123692544], SEEK_SET) = 0
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
_llseek(3, 49257910272, [49257910272], SEEK_SET) = 0
write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
4096) = 4096
fsync(3

Hi,

On Mon, 2004-03-29 at 10:56, Paul Libert wrote:
> I'm faced with the above mentionned problem.
Well, I know for a fact it's not impossible: I've got a box here running
Fedora Core 2 test release, and it's using lvm2 for all of its
filesystems except for /boot.  I wonder what's different about your
setup?
> I've created a new LV and when I try to create the FS on it, it fails 
> when saying "Writing superblocks and filesystem accounting
information:"
> sar output shows that CPU is in 100% wio.
If it's in wio, chances are the fs is waiting for an lvm2 IO to
complete.  That could be stuck in the device-mapper layer, or in the
underlying device driver.  We'd need to see a stack trace of the stuck
process to investigate.  (alt-sysrq-t should give you that.)
> I've performed a strace of the 'mkfs.ext3' and last system call
is a
> call to fsync (see hereafter as small part of the trace file)
Yep, that's consistent with the device being stuck.

In any case, if "mkfs" is hanging it's not an ext3 fault, because
the
filesystem isn't running by that stage.

Cheers,
 Stephen