Ext3 users - Nov 2002 - 2.4.18 ext3 problems

Hi,

Gettings lots of nasty ext3 errors, as follows:

attempt to access beyond end of device
03:06: rw=0, want=1989391900, limit=55167178
EXT3-fs error (device ide0(3,6)): ext3_free_branches: Read failure,
inode=10092558, block=-1113264762

- several thousand of these in the course of a few seconds

And lots of these:

EXT3-fs error (device ide0(3,6)): ext3_free_blocks: Freeing blocks not
in datazone - block = 3602131006, count = 1


And then finally (few seconds after the above):

attempt to access beyond end of device
03:06: rw=0, want=274736132, limit=55167178
EXT3-fs error (device ide0(3,6)): ext3_free_branches: Read failure,
inode=10092558, block=-1541928704
Assertion failure in do_get_write_access() at transaction.c:708:
"handle->h_buffer_credits > 0"
invalid operand: 0000
CPU:    0
EIP:    0010:[<c0156fb7>]    Not tainted
EFLAGS: 00010286
eax: 00000063   ebx: c57dd430   ecx: ccf36780   edx: ffffffff
esi: 00000000   edi: c57dd430   ebp: 00000000   esp: c5c6bcd0
ds: 0018   es: 0018   ss: 0018
Process videoexe (pid: 984, stackpage=c5c6b000)
Stack: c0232720 c02328e6 c0232700 000002c4 c0232921 cff47800 cfe7ee00
c57dd430
       cff47894 cff47800 00000000 00000000 cff47800 cce1d760 c01570d8
cfe7ee00
       c57dd430 00000000 00000000 c69799c0 cfe7ee00 c9d68820 c0150798
cfe7ee00
Call Trace: [<c01570d8>] [<c0150798>] [<c01570e0>]
[<c01508fc>]
[<c0150b98>]
   [<c0150a68>] [<c0150a68>] [<c0150a68>] [<c0150c79>]
[<c0150f0b>]
[<c01567b9>]
   [<c014ebad>] [<c014ec4c>] [<c014ecf3>] [<c014ec4c>]
[<c0140287>]
[<c013ea38>]
   [<c01382c6>] [<c01383a2>] [<c0106b87>]

Code: 0f 0b 83 c4 14 8b 54 24 28 8b 42 04 48 8b 4c 24 28 89 41 04

That finally caused the thread that was running to lock solid and the
machine had to be power-cycled.

Sorry I can't put the oops through ksymoops - opportunity lost.

The system is running stock 2.4.18 build patched for lm-sensors

Hardware is:

Abit ST6 Motherboard + 256Meg + 1.2Gig Celeron
2x80 gig harddisks (make unknown !)
2xbttv capture cards
1xnetgear ethernet card

The machine normally runs at around 40% idle, but is doing lot of disk
writes, essentially its capturing video
and writing it to disk (into 2 large partitions, 50 gig each) and its
possible that one of the partitions was fullish (> 95%) - but not
totally full.

Strangely the same sort of things has happened on another customers site
- but that one is now not rebooting (!!!) and so I cannot retrieve any
logs (as I have to do it via modem) - it has a very similar spec.

I've trawled thought linux.kernel and this mailing list and I've read
lots of 2.4.18 + ext3 problem reports, I've also
seen a few things about 2.4.19 and so was thinking about trying a site
with 2.4.20-pre11. Obviously its possible
that a hardware failure has occurred, but same problem on two sites?.

I'd appreciate it if anyone could give me some idea of what might have
happened and what would be a good
strategy to stop this from happening in the future.

I've got a system on test in the office at the moment and we'll be
dragging a failed one back from site to do more testing on so I might be
able to give more details later.

Any help would be gratefully appreciated.

Cheers,

Glen

On Fri, Nov 08, 2002 at 12:16:38PM +0000, glen wrote:
> Hi,
> 
> Gettings lots of nasty ext3 errors, as follows:
> 
> attempt to access beyond end of device
> 03:06: rw=0, want=1989391900, limit=55167178
> EXT3-fs error (device ide0(3,6)): ext3_free_branches: Read failure,
> inode=10092558, block=-1113264762
> 
> - several thousand of these in the course of a few seconds
Maschin kaput?
I think your drive needs to be replaced -- at least I noticed similar
errors when my drive reached EOL.

-- 
Ralf Hildebrandt (Im Auftrag des Referat V a)   Ralf.Hildebrandt@charite.de
Charite Campus Mitte                            Tel.  +49 (0)30-450 570-155
Referat V a - Kommunikationsnetze -             Fax.  +49 (0)30-450 570-916
I wish you'd tell me what kind of systems they're using instead,
because HP can't be doing much worse than Sun "would you like the
compiler or internet options with that" Microsystems, or Silicon "hey
be glad the support-contract number isn't a 1-900" Graphics. Then
there's Digital "It sucks in 64 bits, you can't suck in 64 bits
anywhere else" Equipment Corp (Did we mention it's 64 bits?).

Hi,

On Fri, Nov 08, 2002 at 12:16:38PM +0000, glen wrote:
 
> Gettings lots of nasty ext3 errors, as follows:
> 
> attempt to access beyond end of device
> 03:06: rw=0, want=1989391900, limit=55167178
That's a data corruption.  It usually turns out to be hardware
problems in my experience, but you could also try a current Marcelo or
-ac kernel, as there are some fixes in there which might be relevant.

Basically, there's a metadata block on disk which tells ext3 where other
blocks are, and that metadata has been corrupted for reasons unknown.
It usually turns out to be something like IDE write caching problems
or bad memory.  But because there's important metadata which is
garbled, ext3 is being told to look in All The Wrong Places for other
parts of the file, resulting in all the other IO failures caused by
illegal blocks getting referenced.
> attempt to access beyond end of device
> 03:06: rw=0, want=274736132, limit=55167178
> EXT3-fs error (device ide0(3,6)): ext3_free_branches: Read failure,
> inode=10092558, block=-1541928704
> Assertion failure in do_get_write_access() at transaction.c:708:
> "handle->h_buffer_credits > 0"
I'll look into that --- even in the presence of disk corruption, ext3
shouldn't panic.

Cheers,
 Stephen