Just wondering if someone could help me debug this:
I was moving data from an ATARAID (via Promise FastTrack100 Controller)
into the LVM device below: 58,1:
# cat /proc/lvm/VGs/foo3/LVs/bar
name: /dev/foo3/bar
size: 476315648
access: 3
status: 1
number: 0
open: 1
allocation: 0
device: 58:01
# /sbin/pvscan
pvscan -- reading all physical volumes (this may take a while...)
pvscan -- ACTIVE PV "/dev/hdm" of VG "foo1" [76.31 GB / 0
free]
pvscan -- ACTIVE PV "/dev/hdn" of VG "foo3" [76.31 GB / 0
free]
pvscan -- ACTIVE PV "/dev/hdk" of VG "foo3" [74.50 GB / 0
free]
pvscan -- ACTIVE PV "/dev/hdl" of VG "foo2" [74.50 GB / 0
free]
pvscan -- ACTIVE PV "/dev/hdi" of VG "foo1" [76.31 GB / 0
free]
pvscan -- ACTIVE PV "/dev/hdj" of VG "foo2" [74.50 GB / 0
free]
pvscan -- ACTIVE PV "/dev/hdg" of VG "foo3" [76.31 GB / 0
free]
pvscan -- ACTIVE PV "/dev/hde" of VG "foo1" [74.52 GB / 0
free]
pvscan -- total: 8 [603.47 GB] / in use: 8 [603.47 GB] / in no VG: 0 [0]
#
All of a sudden, the load averate on the box started going crazy, and
all this was scrolling in my messges file. From remove, I could not
reboot the box. It hung and totally froze up on anything that needed to
access the below filesystem. A hardboot was the only way to get the
system back up and running.
<much more of this above>
Oct 19 16:04:39 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_new_block: Allocating block in system zone - block = 20251124
Oct 19 16:04:39 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_new_block: Allocating block in system zone - block = 20251126
Oct 19 16:04:39 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_new_block: Allocating block in system zone - block = 20251127
Oct 19 16:04:39 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_new_block: Allocating block in system zone - block = 20251128
Oct 19 16:04:39 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_new_block: Allocating block in system zone - block = 20251131
Oct 19 16:04:39 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_new_block: Allocating block in system zone - block = 20251132
Oct 19 16:04:39 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_new_block: Allocating block in system zone - block = 20251135
Oct 19 16:04:39 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_new_block: Allocating block in system zone - block = 20251136
Oct 19 16:04:39 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_new_block: Allocating block in system zone - block = 20251137
Oct 19 16:04:39 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_new_block: Allocating block in system zone - block = 20251140
Oct 19 16:04:39 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_new_block: Allocating block in system zone - block = 20251143
Oct 19 16:04:39 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_new_block: Allocating block in system zone - block = 20251145
Oct 19 16:04:39 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_new_block: Allocating block in system zone - block = 20251146
Oct 19 16:04:39 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_new_block: Allocating block in system zone - block = 20251148
Oct 19 16:04:39 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_new_block: Allocating block in system zone - block = 20251151
Oct 19 16:04:39 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_new_block: Allocating block in system zone - block = 20251152
Oct 19 16:04:43 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_add_entry: bad entry in directory #10125314: rec_len %% 4 != 0
- offset=0, inode=4288315399, rec_len=65535, name_len=255
Oct 19 16:04:44 noname last message repeated 53 times
Oct 19 16:05:04 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_add_entry: bad entry in directory #10125314: rec_len %% 4 != 0
- offset=0, inode=4288315399, rec_len=65535, name_len=255
Oct 19 16:05:04 noname last message repeated 3 times
Oct 19 16:20:50 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_readdir: bad entry in directory #10125314: rec_len %% 4 != 0 -
offset=0, inode=4288315399, rec_len=65535, name_len=255
Oct 19 16:20:58 noname last message repeated 3 times
Oct 19 16:44:16 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_readdir: bad entry in directory #10125314: rec_len %% 4 != 0 -
offset=0, inode=4288315399, rec_len=65535, name_len=255
Oct 19 17:21:17 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_readdir: bad entry in directory #10125314: rec_len %% 4 != 0 -
offset=0, inode=4288315399, rec_len=65535, name_len=255
Oct 19 17:22:04 noname last message repeated 4 times
Oct 19 17:22:27 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_add_entry: bad entry in directory #10125314: rec_len %% 4 != 0
- offset=0, inode=4288315399, rec_len=65535, name_len=255
Oct 19 17:22:40 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_readdir: bad entry in directory #10125314: rec_len %% 4 != 0 -
offset=0, inode=4288315399, rec_len=65535, name_len=255
Oct 19 17:22:46 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_readdir: bad entry in directory #10125314: rec_len %% 4 != 0 -
offset=0, inode=4288315399, rec_len=65535, name_len=255
Oct 19 17:22:46 noname kernel: EXT3-fs warning (device lvm(58,1)):
empty_dir: bad directory (dir #10125314) - no `.' or `..'
Oct 19 17:22:46 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_free_blocks: Freeing blocks in system zones - Block = 20250624
, count = 1
Oct 19 17:23:52 noname kernel: EXT3-fs error (device lvm(58,1)):
ext3_new_block: Allocating block in system zone - block = 20250624
Oct 19 17:23:52 noname kernel: Assertion failure in
unmap_underlying_metadata() at buffer.c:1519: "!buffer_jlist_eq(old_bh,
3)"
Oct 19 17:23:52 noname kernel: invalid operand: 0000
Oct 19 17:23:52 noname kernel: CPU: 0
Oct 19 17:23:52 noname kernel: EIP: 0010:[<c012df12>] Not tainted
Oct 19 17:23:52 noname kernel: EFLAGS: 00010292
Oct 19 17:23:52 noname kernel: eax: 00000064 ebx: c83e0260 ecx:
00000009 edx: df16df64
Oct 19 17:23:52 noname kernel: esi: c03e9a80 edi: c5a80780 ebp:
00001000 esp: ca49fe1c
Oct 19 17:23:52 noname kernel: ds: 0018 es: 0018 ss: 0018
Oct 19 17:23:52 noname kernel: Process mv (pid: 3104,
stackpage=ca49f000)
Oct 19 17:23:52 noname kernel: Stack: c0213d00 c0213e0e c0213cdd
000005ef c0213e28 c5a80780 dee63c80 00000299
Oct 19 17:23:52 noname kernel: c83e0260 00000001 00001000
ca49fe78 c012e23b 00000000 ca49fe78 c012e25c
Oct 19 17:23:52 noname kernel: c83e0260 c83e0260 c37ce000
00001000 00000299 00000000 c83e0260 00000001
Oct 19 17:23:52 noname kernel: Call Trace: [<c012e23b>] [<c012e25c>]
[<c015960b>] [<c0153164>] [<c012e9d1>]
Oct 19 17:23:52 noname kernel: [<c014cb10>] [<c014cead>]
[<c014cb10>]
[<c01261ef>] [<c012136c>] [<c014b1b2>]
Oct 19 17:23:52 noname kernel: [<c012bb76>] [<c0106ceb>]
Oct 19 17:23:52 noname kernel:
Oct 19 17:23:52 noname kernel: Code: 0f 0b 83 c4 14 56 e8 63 12 01 00 59
b8 01 00 00 00 0f b3 46
Anyone have any idea what could have caused this?
SYSTEM:
Kernel: v2.4.10-ac11 (stock LVM and EXT3 patches included with main
kernel patch)
E2fsprogs: v1.25
Utils: v2.11k
Also, is there any way to throttle the access (r/w) to a filesystem?
Say to 2MB/sec or something? Just on the off chance that it's a problem
with the way either the Promise Ultra100 cards or the LVM or the
filesystem handle load?
-JL
