Hi,
Running MySQL Cluster datanode (a process of large memory which checkpoints
data to disk periodically) on a 2xXeon Dell PowerEdge SC1425
(2.6.9-5.ELsmp#1 SMP x86_64), and recieved this notice in
/var/log/messages. The datanode
process crashed, and using the partition after this seemed to hang
processes, eg ls /data/ would hang. Reboot fixed this, boot.log states that
the partition was clean,
Dec 19 08:33:51 data3 fsck: /data: recovering journal
Dec 19 08:33:54 data3 fsck: /data: clean, 9907/6311872 files,
3946561/12619049 blocks
I posted this info to the cluster list initially, they seem to think it
might be a memory > disk problem or perhaps a bug? And said you guys might
know more about it.
Dec 18 03:18:09 data3 kernel: Unable to handle kernel NULL pointer
dereference at 0000000000000020 RIP:
Dec 18 03:18:09 data3 kernel:
<ffffffffa003cfc8>{:jbd:journal_dirty_metadata+71}
Dec 18 03:18:09 data3 kernel: PML4 f4cb067 PGD 2d383067 PMD 0
Dec 18 03:18:09 data3 kernel: Oops: 0000 [1] SMP
Dec 18 03:18:09 data3 kernel: CPU 2
Dec 18 03:18:09 data3 kernel: Modules linked in: md5 ipv6 i2c_dev i2c_core
sunrpc dm_mod button battery ac uhci_hcd ehci_hcd e1000 ext3 jbd ata_piix
libata sd_mod scsi_mod
Dec 18 03:18:09 data3 kernel: Pid: 11186, comm: ndbd Not tainted
2.6.9-5.ELsmp
Dec 18 03:18:09 data3 kernel: RIP: 0010:[<ffffffffa003cfc8>]
<ffffffffa003cfc8>{:jbd:journal_dirty_metadata+71}
Dec 18 03:18:09 data3 kernel: RSP: 0018:000001007c18fbb8 EFLAGS: 00010246
Dec 18 03:18:09 data3 kernel: RAX: 0000000000000000 RBX: 00000101ad1405c0
RCX: 00000000ffffffff
Dec 18 03:18:09 data3 kernel: RDX: 000000000000000f RSI: 00000100447c2ce8
RDI: 0000010024e32820
Dec 18 03:18:09 data3 kernel: RBP: 00000100447c2ce8 R08: 0000000000000000
R09: 0000000000000000
Dec 18 03:18:09 data3 kernel: R10: 0000000000000001 R11: 0000000000000001
R12: 0000000000000000
Dec 18 03:18:09 data3 kernel: R13: 00000100cfe15c00 R14: 0000010024e32820
R15: 000001011cd75130
Dec 18 03:18:09 data3 kernel: FS: 0000000040076960(005b)
GS:ffffffff804bf400(0000) knlGS:0000000000000000
Dec 18 03:18:09 data3 kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
000000008005003b
Dec 18 03:18:09 data3 kernel: CR2: 0000000000000020 CR3: 00000000cff0e000
CR4: 00000000000006e0
Dec 18 03:18:09 data3 kernel: Process ndbd (pid: 11186, threadinfo
000001007c18e000, task 00000101ae70c030)
Dec 18 03:18:09 data3 kernel: Stack: 000001012431cc80 0000000000000000
000001011cd75240 00000100cfe43c00
Dec 18 03:18:09 data3 kernel: 0000000000000001 ffffffffa00528c2
00000100447c2ce8 000001007c18fc38
Dec 18 03:18:09 data3 kernel: 0000010024e32820 000001011cd75240
Dec 18 03:18:09 data3 kernel: Call
Trace:<ffffffffa00528c2>{:ext3:ext3_mark_iloc_dirty+740}
Dec 18 03:18:09 data3 kernel:
<ffffffffa0052a14>{:ext3:ext3_mark_inode_dirty+65}
Dec 18 03:18:10 data3 kernel:
<ffffffffa003c4c6>{:jbd:journal_start+223}
<ffffffffa0054726>{:ext3:ext3_dirty_inode+99}
Dec 18 03:18:10 data3 kernel:
<ffffffff8018fef0>{__mark_inode_dirty+40}
<ffffffff80189748>{inode_update_time+182}
Dec 18 03:18:10 data3 kernel:
<ffffffff80155b7b>{generic_file_aio_write_nolock+620}
Dec 18 03:18:10 data3 kernel:
<ffffffff80155c9c>{generic_file_aio_write+126}
<ffffffffa004fed0>{:ext3:ext3_file_write+22}
Dec 18 03:18:10 data3 kernel: <ffffffff80171f8d>{do_sync_write+173}
<ffffffff801493ba>{do_futex+590}
Dec 18 03:18:10 data3 kernel:
<ffffffff80132ff0>{autoremove_wake_function+0}
<ffffffff80172088>{vfs_write+207}
Dec 18 03:18:10 data3 kernel: <ffffffff80172170>{sys_write+69}
<ffffffff8010ffd2>{system_call+126}
Dec 18 03:18:10 data3 kernel:
Dec 18 03:18:10 data3 kernel:
Dec 18 03:18:10 data3 kernel: Code: 49 39 5c 24 20 75 4b 41 83 7c 24 0c 02
75 43 49 3b 5d 50 0f
Dec 18 03:18:10 data3 kernel: RIP
<ffffffffa003cfc8>{:jbd:journal_dirty_metadata+71} RSP
<000001007c18fbb8>
Dec 18 03:18:10 data3 kernel: CR2: 0000000000000020
Any clarification someone can provide would be great,
Thanks,
Adam
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/ext3-users/attachments/20051220/296f9c2b/attachment.htm>