Ext3 users - Dec 2002 - Assertion failure in do_get_write_access() at transaction.c:737: "(((jh2bh(jh)) on a 2.4.18-14 (RH8.0) kernel while experiencing SCSI errors

Hi,

I use ext3 over RAID1+0 LUN.  When I disconnect my fiber cable to the
RAID (in order to force a migration to a failover node) I expect to see
I/O errors and then a failure to write/read to the filesystem, but in
addition I get an Assertion failure. 

The end result is of course an unusable system which needs to be
rebooted. (I reboot the system via machine_restart which works okay but
I guess a regular reboot will get stuck)

My kernel is basically 2.4.18-14 with a couple of patches applied mainly
in networking area - nothing within miles of the ext3/filesystems area.

Is this a known issue ? 

Dec 24 13:48:34 10.17.0.2 kernel:  I/O error: dev 08:33, sector 110024 
Dec 24 13:48:34 10.17.0.2 kernel: SCSI disk error : host 3 channel 0 id
0 lun 1 return code = 10000 
Dec 24 13:48:34 10.17.0.2 kernel:  I/O error: dev 08:33, sector 847008 
Dec 24 13:48:34 10.17.0.2 kernel: SCSI disk error : host 3 channel 0 id
0 lun 1 return code = 10000 
Dec 24 13:48:34 10.17.0.2 kernel:  I/O error: dev 08:33, sector 846984 
Dec 24 13:48:34 10.17.0.2 kernel: EXT3-fs error (device sd(8,51)):
ext3_get_inode_loc: unable to read inode block
Dec 24 13:48:35 10.17.0.2 kernel: SCSI disk error : host 3 channel 0 id
0 lun 1 return code = 10000 
Dec 24 13:48:35 10.17.0.2 kernel:  I/O error: dev 08:33, sector 148376 
Dec 24 13:48:35 10.17.0.2 kernel: SCSI disk error : host 3 channel 0 id
0 lun 1 return code = 10000 
Dec 24 13:48:35 10.17.0.2 kernel:  I/O error: dev 08:33, sector 110032 
Dec 24 13:48:35 10.17.0.2 kernel: SCSI disk error : host 3 channel 0 id
0 lun 1 return code = 10000 
Dec 24 13:48:35 10.17.0.2 kernel:  I/O error: dev 08:33, sector 0 
Dec 24 13:48:35 10.17.0.2 kernel: EXT3-fs error (device sd(8,51)) in
ext3_reserve_inode_write: IO failure 
Dec 24 13:48:36 10.17.0.2 kernel: SCSI disk error : host 3 channel 0 id
0 lun 1 return code = 10000 
Dec 24 13:48:36 10.17.0.2 kernel:  I/O error: dev 08:33, sector 0 
Dec 24 13:48:36 10.17.0.2 kernel: EXT3-fs error (device sd(8,51)) in
ext3_new_inode: IO failure 
Dec 24 13:48:37 10.17.0.2 kernel: SCSI disk error : host 3 channel 0 id
0 lun 1 return code = 10000 
Dec 24 13:48:37 10.17.0.2 kernel:  I/O error: dev 08:33, sector 0 
Dec 24 13:48:37 10.17.0.2 kernel: EXT3-fs error (device sd(8,51)) in
ext3_setattr: IO failure 
Dec 24 13:48:38 10.17.0.2 kernel: SCSI disk error : host 3 channel 0 id
0 lun 1 return code = 10000 
Dec 24 13:48:38 10.17.0.2 kernel:  I/O error: dev 08:33, sector 0 

Dec 24 13:48:38 10.17.0.2 kernel: Assertion failure in
do_get_write_access() at transaction.c:737: "(((jh2bh(jh))
Dec 24 13:48:38 10.17.0.2 kernel: ------------[ cut here ]------------ 
Dec 24 13:48:38 10.17.0.2 kernel: kernel BUG at transaction.c:737! 
Dec 24 13:48:38 10.17.0.2 kernel: invalid operand: 0000 
Dec 24 13:48:38 10.17.0.2 kernel: CPU:    0 
Dec 24 13:48:38 10.17.0.2 kernel: EIP:
0010:[do_get_write_access+1125/1360]    Not tainted 
Dec 24 13:48:38 10.17.0.2 kernel: EIP:    0010:[<c016e845>]    Not
tainted 
Dec 24 13:48:38 10.17.0.2 kernel: EFLAGS: 00010286 
Dec 24 13:48:38 10.17.0.2 kernel:  
Dec 24 13:48:38 10.17.0.2 kernel: EIP is at  (2.4.18-14exa) 
Dec 24 13:48:38 10.17.0.2 kernel: eax: 0000007b   ebx: e62f7000   ecx:
00000000   edx: f5650000 
Dec 24 13:48:38 10.17.0.2 agent[25459]: panic: fswait: child 25496
exited: status 11  
Dec 24 13:48:38 10.17.0.2 agent[25459]: traceback:  
Dec 24 13:48:38 10.17.0.2 agent[25459]: ^IFrame [0] 0x8056c7e  
Dec 24 13:48:38 10.17.0.2 agent[25459]: ^IFrame [1] 0x8056d2d  
Dec 24 13:48:38 10.17.0.2 agent[25459]: ^IFrame [2] 0x80552f9  
Dec 24 13:48:38 10.17.0.2 agent[25459]: ^IFrame [3] 0x8053d81  
Dec 24 13:48:38 10.17.0.2 agent[25459]: ^IFrame [4] 0x8070bd2  
Dec 24 13:48:38 10.17.0.2 agent[25459]: ^IFrame [5] 0x8048101  
Dec 24 13:48:38 10.17.0.2 agent[25459]: in main process getpid 25459
getpgrp 25459  
Dec 24 13:48:38 10.17.0.2 kernel: esi: ced3af80   edi: d2eb8c00   ebp:
00000001   esp: ec1edb90 
Dec 24 13:48:38 10.17.0.2 kernel: ds: 0018   es: 0018   ss: 0018 
Dec 24 13:48:38 10.17.0.2 kernel: Process agent (pid: 25496,
stackpage=ec1ed000) 
Dec 24 13:48:38 10.17.0.2 kernel: Stack: c02a95e0 c02a4c4d c02a4b29
000002e1 c02abc60 00000001 00000000 00000000 
Dec 24 13:48:38 10.17.0.2 kernel:        eebc73c0 00000000 c0168677
cae6cc60 d2eb8c94 d2eb8c00 cae6cc60 c69a5880 
Dec 24 13:48:38 10.17.0.2 kernel:        c016e967 cae6cc60 c69a5880
00000000 00000000 f3452000 00000296 000000aa 
Dec 24 13:48:39 10.17.0.2 kernel: Call Trace: [ext3_dirty_inode+215/304]
(0xec1edbb8)) 
Dec 24 13:48:39 10.17.0.2 kernel: Call Trace: [<c0168677>]
(0xec1edbb8)) 
Dec 24 13:48:39 10.17.0.2 kernel: [journal_get_write_access+55/96]
(0xec1edbd0)) 
Dec 24 13:48:39 10.17.0.2 kernel: [<c016e967>]  (0xec1edbd0)) 
Dec 24 13:48:39 10.17.0.2 kernel: [ext3_new_block+1076/2272]
(0xec1edbf0)) 
Dec 24 13:48:39 10.17.0.2 kernel: [<c0163274>]  (0xec1edbf0)) 
Dec 24 13:48:39 10.17.0.2 kernel: [ext3_do_update_inode+710/848]
(0xec1edc00)) 
Dec 24 13:48:39 10.17.0.2 kernel: [<c0168146>]  (0xec1edc00)) 
Dec 24 13:48:39 10.17.0.2 kernel: [ext3_reserve_inode_write+49/176]
(0xec1edc2c)) 
Dec 24 13:48:39 10.17.0.2 kernel: [<c01684e1>]  (0xec1edc2c)) 
Dec 24 13:48:39 10.17.0.2 kernel: [__wait_on_buffer+142/160]
(0xec1edc44)) 
Dec 24 13:48:39 10.17.0.2 kernel: [<c014104e>]  (0xec1edc44)) 


Thanks, and merry christmas to all.

Yuval 

PS - please CC me for any replies as I'm not subscribed to the list.
thanks.

--
Yuval Yeret
Exanet
yuval@exanet.com
http://www.exanet.com
Tel.  972-9-9717782
Fax. 972-9-9717778


 <<Yuval Yeret (E-mail).vcf>>

> I use ext3 over RAID1+0 LUN.  When I disconnect my fiber cable to the
> RAID (in order to force a migration to a failover node) I expect to see
> I/O errors and then a failure to write/read to the filesystem, but in
> addition I get an Assertion failure. 
Steven has fixed this in his latest patch sets.  I don't know if it's
going to make it to 2.4.21.  The two patches in question are at:

http://people.redhat.com/sct/patches/ext3-2.4/dev/03-updates-sct/3004-expect-debug.patch
http://people.redhat.com/sct/patches/ext3-2.4/dev/03-updates-sct/3005-expect-iofail.patch

Regards,
 Robert.

-- 
Robert Walsh
Amalgamated Durables, Inc.  -  "We don't make the things you buy."
Email: rjwalsh@durables.org