Ext3 users - Nov 2005 - EXT3-fs error (device md2): ext3_journal_start_sb: Detected aborted journal...

Hi,

I'm running a production server (Debian Sarge install) whose root
filesystem (a software raid 1 array of 2 partitions of IDE drive)
exhibited the following problem:

Oct 28 06:00:06 server2 kernel: attempt to access beyond end of device
Oct 28 06:00:06 server2 kernel: md2: rw=1, want=3050401328, limit=16353920
[...] a few of the above line snipped, want is different each time

Oct 28 06:00:06 server2 kernel: md2: rw=1, want=2323778952, limit=16353920
Oct 28 06:00:06 server2 kernel: printk: 2 messages suppressed.
Oct 28 06:00:06 server2 kernel: Buffer I/O error on device md2, logical block
3511697840
Oct 28 06:00:06 server2 kernel: lost page write due to I/O error on md2
Oct 28 06:00:06 server2 kernel: Aborting journal on device md2.
Oct 28 06:05:01 server2 kernel: ext3_abort called.
Oct 28 06:05:01 server2 kernel: EXT3-fs error (device md2):
ext3_journal_start_sb: Detected aborted journal
Oct 28 06:05:01 server2 kernel: Remounting filesystem read-only

Now the root filesystem is remounted read-only.
Running fsck on it produces the following:

server2:~# e2fsck /dev/md2
e2fsck 1.37 (21-Mar-2005)
Pass 1: Checking inodes, blocks, and sizes
Inode 8 has illegal block(s).  Clear<y>? yes

Illegal block #2371 (3939553560) in inode 8.  CLEARED.
Illegal block #2372 (2534662274) in inode 8.  CLEARED.
Illegal block #2373 (860109200) in inode 8.  CLEARED.
Illegal block #2374 (3289467369) in inode 8.  CLEARED.
Illegal block #2375 (3883044785) in inode 8.  CLEARED.
Illegal block #2376 (819724782) in inode 8.  CLEARED.
Illegal block #2377 (2957378758) in inode 8.  CLEARED.
Illegal block #2378 (1131441392) in inode 8.  CLEARED.
Illegal block #2379 (1473257247) in inode 8.  CLEARED.
Illegal block #2380 (2359314433) in inode 8.  CLEARED.
Illegal block #2381 (448867375) in inode 8.  CLEARED.
Too many illegal blocks in inode 8.
Clear inode<y>? yes

Restarting e2fsck from the beginning...
Pass 1: Checking inodes, blocks, and sizes
Inode 8 has illegal block(s).  Clear<y>? 

and loops forever.

I know inode 8 is the journal inode.

I didn't try to reboot the server as I fear the recovery process would
not work and would need a human presence to force the fsck (see question
#1)

What can I do to remotely repair this root filesystem (as the server is
in a datacenter from which I'm far at the moment), and remount it rw ?

Thank you,
-- 
Brice Figureau

On Nov 08, 2005  16:14 +0100, Brice Figureau wrote:
> Restarting e2fsck from the beginning...
> Pass 1: Checking inodes, blocks, and sizes
> Inode 8 has illegal block(s).  Clear<y>? 
> 
> and loops forever.
> 
> I know inode 8 is the journal inode.
> 
> What can I do to remotely repair this root filesystem (as the server is
> in a datacenter from which I'm far at the moment), and remount it rw ?
Running 'debugfs -w -R "feature ^has_journal,^needs_recovery"
/dev/md2'
should remove the journal from the filesystem, and then your e2fsck
may work.  Don't forget to add it back afterward "tune2fs -j
/dev/md2".

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.