Hi all - I'm having a strange issue and I'm trying to figure out what the problem is. We have a 3 node RHEL 4 AS Oracle 10gR2 RAC cluster which utilizes an iscsi SAN. We have 2 disks mounted through OCFS2 (most current release) for the ORACLE_HOME and for the CRS Voting Disk. The rest of the disks are mounted via ASM (also the most current release). It seems that for some reason, one node keeps crashing (Watson) and when it goes, one of the other nodes is soon to follow. I have attached the rather lengthy messages from /var/log/ messages. Has anyone seen any behavior like this? The weird thing to note is that when nodes 1 and 2 go down, the third one keeps chugging without any apparent issues. Here are the logs - first from Node 2 (watson) which goes down first I think - then Node 1 (sherlock). If anyone has any ideas as to where to look for the source of these troubles, I'd greatly appreciate it. Thanks Adam Mar 12 05:26:15 watson kernel: Unable to handle kernel paging request at virtual address 003b0222 Mar 12 05:26:15 watson kernel: printing eip: Mar 12 05:26:15 watson kernel: c01c074e Mar 12 05:26:15 watson kernel: *pde = 359b1001 Mar 12 05:26:15 watson kernel: Oops: 0002 [#1] Mar 12 05:26:15 watson kernel: SMP Mar 12 05:26:15 watson kernel: Modules linked in: ocfs2(U) debugfs(U) ipv6 parpo rt_pc lp parport crc32c libcrc32c md5 iscsi_sfnet scsi_transport_iscsi oracleasm (U) autofs4 i2c_dev i2c_core nfs lockd nfs_acl ocfs2_dlmfs(U) ocfs2_dlm(U) ocfs2 _nodemanager(U) configfs(U) sunrpc dm_mirror dm_mod button battery ac uhci_hcd e hci_hcd hw_random shpchp e1000 bond2(U) bond1(U) bond0(U) floppy sg ext3 jbd meg araid_mbox megaraid_mm sd_mod scsi_mod Mar 12 05:26:15 watson kernel: CPU: 1 Mar 12 05:26:15 watson kernel: EIP: 0060:[<c01c074e>] Not tainted VLI Mar 12 05:26:15 watson kernel: EFLAGS: 00010206 (2.6.9-34.ELsmp) Mar 12 05:26:15 watson kernel: EIP is at __rb_rotate_left+0x12/0x36 Mar 12 05:26:15 watson kernel: eax: 003b0222 ebx: eef832c8 ecx: dd29fe00 e dx: d200ae08 Mar 12 05:26:15 watson kernel: esi: eef832c8 edi: f6429e5c ebp: dd29fdc0 e sp: f6429e28 Mar 12 05:26:15 watson kernel: ds: 007b es: 007b ss: 0068 Mar 12 05:26:15 watson kernel: Process ocfs2vote-1 (pid: 3978, threadinfo=f64290 00 task=c3a8eeb0) Mar 12 05:26:15 watson kernel: Stack: dd29fdc0 c01c0989 dd29fd80 00000000 f91b30 22 eef832c4 00000000 00000000 Mar 12 05:26:15 watson kernel: eef83324 eef832ac eef83100 f91b30a3 f6429e 60 dd29fd40 f7c43480 f7c43480 Mar 12 05:26:15 watson kernel: eef83100 00000001 f91b9d14 f5faf800 eef833 24 c3a8eeb0 c38a2200 00000002 Mar 12 05:26:15 watson kernel: Call Trace: Mar 12 05:26:15 watson kernel: [<c01c0989>] __rb_erase_color +0x120/0x16b Mar 12 05:26:15 watson kernel: [<f91b3022>] __ocfs2_extent_map_drop +0x4b/0x86 [ ocfs2] Mar 12 05:26:15 watson kernel: [<f91b30a3>] ocfs2_extent_map_drop +0x2c/0x69 [oc fs2] Mar 12 05:26:15 watson kernel: [<f91b9d14>] ocfs2_clear_inode +0x404/0xc0a [ocfs 2] Mar 12 05:26:15 watson kernel: [<c01766b4>] __sync_single_inode +0x1b7/0x1c1 Mar 12 05:26:15 watson kernel: [<c0170008>] clear_inode+0xcc/0x102 Mar 12 05:26:15 watson kernel: [<f91b98c9>] ocfs2_delete_inode +0x404/0x44b [ocf s2] Mar 12 05:26:15 watson kernel: [<f91b94c5>] ocfs2_delete_inode +0x0/0x44b [ocfs2 ] Mar 12 05:26:15 watson kernel: [<c0170be4>] generic_delete_inode +0xa2/0x104 Mar 12 05:26:15 watson kernel: [<f91ba600>] ocfs2_drop_inode +0xe6/0x12a [ocfs2] Mar 12 05:26:15 watson kernel: [<c0170dc0>] iput+0x5f/0x61 Mar 12 05:26:15 watson kernel: [<f91d5a47>] ocfs2_process_vote+0x71f/ 0x727 [ocf s2] Mar 12 05:26:15 watson kernel: [<c02d0589>] schedule+0x83d/0x8d3 Mar 12 05:26:15 watson kernel: [<f91d5c46>] ocfs2_vote_thread +0x0/0x121 [ocfs2] Mar 12 05:26:15 watson kernel: [<f91d5b68>] ocfs2_vote_thread_do_work +0x119/0x1 7a [ocfs2] Mar 12 05:26:15 watson kernel: [<f91d5c46>] ocfs2_vote_thread +0x0/0x121 [ocfs2] Mar 12 05:26:15 watson kernel: [<f91d5d39>] ocfs2_vote_thread +0xf3/0x121 [ocfs2 ] Mar 12 05:26:15 watson kernel: [<c0120291>] autoremove_wake_function +0x0/0x2d Mar 12 05:26:15 watson kernel: [<c0120291>] autoremove_wake_function +0x0/0x2d Mar 12 05:26:15 watson kernel: [<c0133ecd>] kthread+0x73/0x9b Mar 12 05:26:15 watson kernel: [<c0133e5a>] kthread+0x0/0x9b Mar 12 05:26:15 watson kernel: [<c01041f5>] kernel_thread_helper +0x5/0xb Mar 12 05:26:15 watson kernel: Code: f9 01 76 ed 31 c0 5b c3 57 b9 45 00 00 00 8 9 c7 31 c0 f3 ab 5f c3 90 90 90 53 89 d3 8b 50 08 89 c1 8b 42 0c 85 c0 89 41 08 74 02 <89> 08 89 4a 0c 8b 01 85 c0 89 02 74 11 8b 01 3b 48 0c 75 05 89 Mar 12 05:26:15 watson kernel: <0>Fatal exception: panic in 5 seconds Mar 12 05:25:09 sherlock kernel: (0,1):o2net_idle_timer:1310 connection to node watson (num 1) at 10.3.3.11:7777 has been idle for 10 seconds, shutting it down. Mar 12 05:25:09 sherlock kernel: (0,1):o2net_idle_timer:1321 here are some times that might help debug the situation: (tmr 1142159099.775825 now 1142159109.7738 47 dr 1142159099.775812 adv 1142159099.775829:1142159099.775831 func (1167074b:5 05) 1142159099.198517:1142159099.198535) Mar 12 05:25:09 sherlock kernel: (2903,1):o2net_set_nn_state:411 no longer conne cted to node watson (num 1) at 10.3.3.11:7777 Mar 12 05:25:09 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -112 Mar 12 05:25:09 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:09 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:09 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:09 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:09 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:13 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:13 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:13 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:13 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:13 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:13 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:13 sherlock kernel: (7982,0):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:13 sherlock kernel: (7982,0):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:13 sherlock kernel: (7982,1):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:13 sherlock kernel: (7982,1):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:13 sherlock kernel: (7982,1):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:13 sherlock kernel: (7982,1):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:13 sherlock kernel: (7982,1):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:13 sherlock kernel: (7982,1):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:13 sherlock kernel: (7982,1):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:13 sherlock kernel: (7982,1):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:13 sherlock kernel: (7982,1):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:13 sherlock kernel: (7982,1):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:13 sherlock kernel: (7982,1):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:13 sherlock kernel: (7982,1):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:14 sherlock kernel: (2903,1):ocfs2_dlm_eviction_cb:118 device (8,49 ): dlm has evicted node 1 Mar 12 05:25:14 sherlock kernel: (8109,1):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:14 sherlock kernel: (8109,1):dlm_wait_for_node_death:285 BD08C3AD58 994411981AAA8C5938F5E0: waiting 5000ms for notification of death of node 1 Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_send_remote_convert_request:393 ER ROR: status = -107 Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_wait_for_node_death:285 9ED43B6CE8 DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of node 1 Mar 12 05:25:14 sherlock kernel: eip: f8ceeba2 Mar 12 05:25:14 sherlock kernel: ------------[ cut here ]------------ Mar 12 05:25:14 sherlock kernel: kernel BUG at include/asm/spinlock.h: 133! Mar 12 05:25:14 sherlock kernel: invalid operand: 0000 [#1] Mar 12 05:25:14 sherlock kernel: SMP Mar 12 05:25:14 sherlock kernel: Modules linked in: ocfs2(U) debugfs (U) ipv6 par port_pc lp parport crc32c libcrc32c md5 iscsi_sfnet scsi_transport_iscsi oraclea sm(U) autofs4 i2c_dev i2c_core nfs lockd nfs_acl ocfs2_dlmfs(U) ocfs2_dlm(U) ocf s2_nodemanager(U) configfs(U) sunrpc dm_mirror dm_mod button battery ac uhci_hcd ehci_hcd hw_random shpchp e1000 bond2(U) bond1(U) bond0(U) floppy sg ext3 jbd m egaraid_mbox megaraid_mm sd_mod scsi_mod Mar 12 05:25:14 sherlock kernel: CPU: 2 Mar 12 05:25:14 sherlock kernel: EIP: 0060:[<c02d11e8>] Not tainted VLI Mar 12 05:25:14 sherlock kernel: EFLAGS: 00010216 (2.6.9-34.ELsmp) Mar 12 05:25:14 sherlock kernel: EIP is at _spin_lock+0x1c/0x34 Mar 12 05:25:14 sherlock kernel: eax: c02e4ca6 ebx: e7726c94 ecx: f648de50 edx: f8ceeba2 Mar 12 05:25:14 sherlock kernel: esi: e7726c80 edi: 00000001 ebp: 00000000 esp: f648de54 Mar 12 05:25:14 sherlock kernel: ds: 007b es: 007b ss: 0068 Mar 12 05:25:14 sherlock kernel: Process o2hb-BD08C3AD58 (pid: 3862, threadinfof648d000 task=f5c800b0) Mar 12 05:25:14 sherlock kernel: Stack: 00000001 f8ceeba2 e7726c88 f5992000 f8ce eb88 00000001 00000001 f5992000 Mar 12 05:25:14 sherlock kernel: 00000001 00000001 f8cfe684 f5992030 f599 2000 f8cfe76a f599215c f5992158 Mar 12 05:25:14 sherlock kernel: f8dd0920 f8dba8f7 c3897c80 00000000 f648 dedc f648dedc f8dce8a0 f8dbaa27 Mar 12 05:25:14 sherlock kernel: Call Trace: Mar 12 05:25:14 sherlock kernel: [<f8ceeba2>] dlm_mle_node_down +0x10/0x73 [ocfs 2_dlm] Mar 12 05:25:14 sherlock kernel: [<f8ceeb88>] dlm_hb_event_notify_attached+0x6e /0x78 [ocfs2_dlm] Mar 12 05:25:14 sherlock kernel: [<f8cfe684>] __dlm_hb_node_down +0x1a6/0x267 [o cfs2_dlm] Mar 12 05:25:14 sherlock kernel: [<f8cfe76a>] dlm_hb_node_down_cb +0x25/0x3a [oc fs2_dlm] Mar 12 05:25:14 sherlock kernel: [<f8dba8f7>] o2hb_fire_callbacks +0x62/0x6c [oc fs2_nodemanager] Mar 12 05:25:14 sherlock kernel: [<f8dbaa27>] o2hb_run_event_list +0x126/0x162 [ ocfs2_nodemanager] Mar 12 05:25:14 sherlock kernel: [<f8dbb0f9>] o2hb_check_slot +0x4d2/0x4e7 [ocfs 2_nodemanager] Mar 12 05:25:14 sherlock kernel: [<c02243c2>] submit_bio+0xca/0xd2 Mar 12 05:25:14 sherlock kernel: [<f8dbb3ed>] o2hb_do_disk_heartbeat +0x2b4/0x32 5 [ocfs2_nodemanager] Mar 12 05:25:14 sherlock kernel: [<f8dbb4e2>] o2hb_thread+0x0/0x291 [ocfs2_node manager] Mar 12 05:25:14 sherlock kernel: [<f8dbb56b>] o2hb_thread+0x89/0x291 [ocfs2_nod emanager] Mar 12 05:25:14 sherlock kernel: [<f8dbb4e2>] o2hb_thread+0x0/0x291 [ocfs2_node manager] Mar 12 05:25:14 sherlock kernel: [<c0133ecd>] kthread+0x73/0x9b Mar 12 05:25:14 sherlock kernel: [<c0133e5a>] kthread+0x0/0x9b Mar 12 05:25:14 sherlock kernel: [<c01041f5>] kernel_thread_helper +0x5/0xb Mar 12 05:25:14 sherlock kernel: Code: 00 75 09 f0 81 02 00 00 00 01 30 c9 89 c8 c3 53 89 c3 81 78 04 ad 4e ad de 74 18 ff 74 24 04 68 a6 4c 2e c0 e8 54 14 e5 f f 58 5a <0f> 0b 85 00 60 3d 2e c0 f0 fe 0b 79 09 f3 90 80 3b 00 7e f9 eb Mar 12 05:25:14 sherlock kernel: <0>Fatal exception: panic in 5 seconds