Hello On a reboot of one of our nodes, the others have crashed with kernel panic. This is what netconsole captured. Anyone have any idea what the problem is? Sep 14 15:22:23 172.27.100.2 (6280,2): o2net_idle_timer:1476 here are some times that might help debug the situation: (tmr 1252930915.974412 now 1252930945.966783 dr 1252930915.974385 adv 1252930915.974412:1252930915.974413 func (a423e7e1:505) 1252930904.543344:1252930904.543352) Sep 14 15:22:23 172.27.100.2 (30417,0): dlm_send_remote_convert_request:395 ERROR: status = -112 Sep 14 15:22:23 172.27.100.2 (30417,0): dlm_wait_for_node_death:370 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of death of node 1 Sep 14 15:22:24 172.27.100.2 (30129,1): dlm_send_remote_convert_request:395 ERROR: status = -112 Sep 14 15:22:24 172.27.100.2 (30129,1): dlm_wait_for_node_death:370 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of death of node 1 Sep 14 15:22:53 172.27.100.2 (2312,2): o2net_connect_expired:1637 ERROR: no connection established with node 1 after 30.0 seconds, giving up and returning errors. Sep 14 15:22:53 172.27.100.2 (1653,0): dlm_do_master_request:1335 ERROR: link to 1 went down! Sep 14 15:22:53 172.27.100.2 (1653,0): dlm_get_lock_resource:912 ERROR: status = -107 Sep 14 15:22:53 172.27.100.2 (30417,0): dlm_send_remote_convert_request:395 ERROR: status = -107 Sep 14 15:22:53 172.27.100.2 (30417,0): dlm_wait_for_node_death:370 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of death of node 1 Sep 14 15:22:53 172.27.100.2 (30129,0): dlm_send_remote_convert_request:395 ERROR: status = -107 Sep 14 15:22:53 172.27.100.2 (30129,0): dlm_wait_for_node_death:370 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of death of node 1 Sep 14 15:23:23 172.27.100.2 (2312,3): o2net_connect_expired:1637 ERROR: no connection established with node 1 after 30.0 seconds, giving up and returning errors. Sep 14 15:23:23 172.27.100.2 (1653,0): dlm_do_master_request:1335 ERROR: link to 1 went down! Sep 14 15:23:23 172.27.100.2 (1653,0): dlm_get_lock_resource:912 ERROR: status = -107 Sep 14 15:23:23 172.27.100.2 (30129,0): dlm_send_remote_convert_request:395 ERROR: status = -107 Sep 14 15:23:23 172.27.100.2 (30129,0): dlm_wait_for_node_death:370 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of death of node 1 Sep 14 15:23:23 172.27.100.2 (30417,0): dlm_send_remote_convert_request:395 ERROR: status = -107 Sep 14 15:23:23 172.27.100.2 (30417,0): dlm_wait_for_node_death:370 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of death of node 1 Sep 14 15:23:23 172.27.100.2 (23657,1): dlm_do_master_request:1335 ERROR: link to 1 went down! Sep 14 15:23:23 172.27.100.2 (23657,1): dlm_get_lock_resource:912 ERROR: status = -107 Sep 14 15:23:23 172.27.100.2 (5251,1): dlm_do_master_request:1335 ERROR: link to 1 went down! Sep 14 15:23:23 172.27.100.2 (5251,1): dlm_get_lock_resource:912 ERROR: status = -107 Sep 14 15:23:23 172.27.100.2 (5251,1): dlm_do_master_request:1335 ERROR: link to 1 went down! Sep 14 15:23:23 172.27.100.2 (5251,1): dlm_get_lock_resource:912 ERROR: status = -107 Sep 14 15:23:24 172.27.100.2 (1653,0): dlm_do_master_request:1335 ERROR: link to 1 went down! Sep 14 15:23:24 172.27.100.2 (1653,0): dlm_get_lock_resource:912 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (2312,2): o2net_connect_expired:1637 ERROR: no connection established with node 1 after 30.0 seconds, giving up and returning errors. Sep 14 15:23:54 172.27.100.2 (9967,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (9967,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (9908,3): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (10756,0): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (10883,0): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (9903,0): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (11183,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (10880,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (10894,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (10907,1): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (10879,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (10906,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (9841,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (10852,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (11227,3): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (10852,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (10904,3): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (4107,0): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (10907,1): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (5750,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (30417,0): dlm_send_remote_convert_request:395 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (30417,0): dlm_wait_for_node_death:370 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of death of node 1 Sep 14 15:23:54 172.27.100.2 (30129,0): dlm_send_remote_convert_request:395 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (30129,0): dlm_wait_for_node_death:370 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of death of node 1 Sep 14 15:23:54 172.27.100.2 (10773,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (11003,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (9539,3): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (11009,3): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (10554,3): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (10907,1): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (11235,1): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (11235,1): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (11039,1): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (9836,1): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (10553,1): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (10553,1): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (10587,1): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (10754,1): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (11306,3): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (10907,1): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (9778,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (9778,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (9778,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (9778,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (11111,3): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (9836,1): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:54 172.27.100.2 (4107,0): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:55 172.27.100.2 (10885,3): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:55 172.27.100.2 (11004,3): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:55 172.27.100.2 (9403,3): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:55 172.27.100.2 (10980,3): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:55 172.27.100.2 (10980,3): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:55 172.27.100.2 (10848,0): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:55 172.27.100.2 (10537,0): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:55 172.27.100.2 (10537,0): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:55 172.27.100.2 (8817,0): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:55 172.27.100.2 (9909,0): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:55 172.27.100.2 (9839,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:55 172.27.100.2 (9779,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:55 172.27.100.2 (9911,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:55 172.27.100.2 (9908,0): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:55 172.27.100.2 (10961,1): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:55 172.27.100.2 (1653,0): dlm_do_master_request:1335 ERROR: link to 1 went down! Sep 14 15:23:55 172.27.100.2 (1653,0): dlm_get_lock_resource:912 ERROR: status = -107 Sep 14 15:23:55 172.27.100.2 (9901,0): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:59 172.27.100.2 (11012,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:59 172.27.100.2 (11040,0): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:59 172.27.100.2 (2334,0): ocfs2_dlm_eviction_cb:98 device (8,1): dlm has evicted node 1 Sep 14 15:23:59 172.27.100.2 (10895,2): dlm_send_remote_unlock_request:359 ERROR: status = -107 Sep 14 15:23:59 172.27.100.2 ------------: cut here ]------------ Sep 14 15:23:59 172.27.100.2 kernel: BUG at /usr/src/ocfs2-1.4.1/fs/ocfs2/dlm/dlmrecovery.c:2197! Sep 14 15:23:59 172.27.100.2 invalid: opcode: 0000 [#1] Sep 14 15:23:59 172.27.100.2 SMP: Sep 14 15:23:59 172.27.100.2 Sep 14 15:23:59 172.27.100.2 last: sysfs file: /devices/pci0000:00/0000:00:00.0/irq Sep 14 15:23:59 172.27.100.2 Modules: linked in: Sep 14 15:23:59 172.27.100.2 netconsole: Sep 14 15:23:59 172.27.100.2 ocfs2(U): Sep 14 15:23:59 172.27.100.2 ocfs2_dlmfs(U): Sep 14 15:23:59 172.27.100.2 ocfs2_dlm(U): Sep 14 15:23:59 172.27.100.2 ocfs2_nodemanager(U): Sep 14 15:23:59 172.27.100.2 configfs: Sep 14 15:23:59 172.27.100.2 ipv6: Sep 14 15:23:59 172.27.100.2 xfrm_nalgo: Sep 14 15:23:59 172.27.100.2 crypto_api: Sep 14 15:23:59 172.27.100.2 dm_mirror: Sep 14 15:23:59 172.27.100.2 dm_multipath: Sep 14 15:23:59 172.27.100.2 dm_mod: Sep 14 15:23:59 172.27.100.2 video: Sep 14 15:23:59 172.27.100.2 sbs: Sep 14 15:23:59 172.27.100.2 backlight: Sep 14 15:23:59 172.27.100.2 i2c_ec: Sep 14 15:23:59 172.27.100.2 i2c_core: Sep 14 15:23:59 172.27.100.2 button: Sep 14 15:23:59 172.27.100.2 battery: Sep 14 15:23:59 172.27.100.2 asus_acpi: Sep 14 15:23:59 172.27.100.2 ac: Sep 14 15:23:59 172.27.100.2 parport_pc: Sep 14 15:23:59 172.27.100.2 lp: Sep 14 15:23:59 172.27.100.2 parport: Sep 14 15:23:59 172.27.100.2 ide_cd: Sep 14 15:23:59 172.27.100.2 cdrom: Sep 14 15:23:59 172.27.100.2 pcspkr: Sep 14 15:23:59 172.27.100.2 serio_raw: Sep 14 15:23:59 172.27.100.2 i6300esb: Sep 14 15:23:59 172.27.100.2 e752x_edac: Sep 14 15:23:59 172.27.100.2 edac_mc: Sep 14 15:23:59 172.27.100.2 floppy: Sep 14 15:23:59 172.27.100.2 tg3: Sep 14 15:23:59 172.27.100.2 e1000: Sep 14 15:23:59 172.27.100.2 mppVhba(U): Sep 14 15:23:59 172.27.100.2 usb_storage: Sep 14 15:23:59 172.27.100.2 qla2xxx: Sep 14 15:23:59 172.27.100.2 scsi_transport_fc: Sep 14 15:23:59 172.27.100.2 ata_piix: Sep 14 15:23:59 172.27.100.2 libata: Sep 14 15:23:59 172.27.100.2 cciss: Sep 14 15:23:59 172.27.100.2 mppUpper(U): Sep 14 15:23:59 172.27.100.2 sg: Sep 14 15:23:59 172.27.100.2 sd_mod: Sep 14 15:23:59 172.27.100.2 scsi_mod: Sep 14 15:23:59 172.27.100.2 ext3: Sep 14 15:23:59 172.27.100.2 jbd: Sep 14 15:23:59 172.27.100.2 uhci_hcd: Sep 14 15:23:59 172.27.100.2 ohci_hcd: Sep 14 15:23:59 172.27.100.2 ehci_hcd: Sep 14 15:23:59 172.27.100.2 Sep 14 15:23:59 172.27.100.2 CPU: 0 Sep 14 15:23:59 172.27.100.2 EIP: 0060:[<f8f68296>] Tainted: G VLI Sep 14 15:23:59 172.27.100.2 EFLAGS: 00010246 (2.6.18-92.1.22.el5PAE #1) Sep 14 15:23:59 172.27.100.2 EIP: is at __dlm_hb_node_down+0x6f1/0x8ae [ocfs2_dlm] Sep 14 15:23:59 172.27.100.2 eax: 00000000 ebx: e90ee480 ecx: e90ee4a8 edx: 00000001 Sep 14 15:23:59 172.27.100.2 esi: f4c33000 edi: e90ee498 ebp: e90ee498 esp: f186ae84 Sep 14 15:23:59 172.27.100.2 ds: 007b es: 007b ss: 0068 Sep 14 15:23:59 172.27.100.2 Process: o2hb-2ACADF1C94 (pid: 2334, ti=f186a000 task=f7fc5aa0 task.ti=f186a000) Sep 14 15:23:59 172.27.100.2 Sep 14 15:23:59 172.27.100.2 Stack: Sep 14 15:23:59 172.27.100.2 00000001: Sep 14 15:23:59 172.27.100.2 0000604d: Sep 14 15:23:59 172.27.100.2 00000001: Sep 14 15:23:59 172.27.100.2 01000292: Sep 14 15:23:59 172.27.100.2 00000000: Sep 14 15:23:59 172.27.100.2 01000000: Sep 14 15:23:59 172.27.100.2 00000002: Sep 14 15:23:59 172.27.100.2 00000001: Sep 14 15:23:59 172.27.100.2 Sep 14 15:23:59 172.27.100.2 Sep 14 15:23:59 172.27.100.2 01000001: Sep 14 15:23:59 172.27.100.2 f4c33000: Sep 14 15:23:59 172.27.100.2 00000001: Sep 14 15:23:59 172.27.100.2 f8d0a2a0: Sep 14 15:23:59 172.27.100.2 f186af4c: Sep 14 15:23:59 172.27.100.2 f8f6ae23: Sep 14 15:23:59 172.27.100.2 f4c33158: Sep 14 15:23:59 172.27.100.2 f4c33154: Sep 14 15:23:59 172.27.100.2 Sep 14 15:23:59 172.27.100.2 Sep 14 15:23:59 172.27.100.2 f8cf5203: Sep 14 15:23:59 172.27.100.2 00000001: Sep 14 15:23:59 172.27.100.2 f1b23580: Sep 14 15:23:59 172.27.100.2 c73535be: Sep 14 15:23:59 172.27.100.2 e69511ff: Sep 14 15:23:59 172.27.100.2 00000001: Sep 14 15:23:59 172.27.100.2 f116c024: Sep 14 15:23:59 172.27.100.2 f8cf5ff8: Sep 14 15:23:59 172.27.100.2 Sep 14 15:23:59 172.27.100.2 Call: Trace: Sep 14 15:23:59 172.27.100.2 Sep 14 15:23:59 172.27.100.2 dlm_hb_node_down_cb+0x35/0x46: [ocfs2_dlm] Sep 14 15:23:59 172.27.100.2 Sep 14 15:23:59 172.27.100.2 o2hb_run_event_list+0x112/0x15e: [ocfs2_nodemanager] Sep 14 15:23:59 172.27.100.2 Sep 14 15:23:59 172.27.100.2 o2hb_do_disk_heartbeat+0x85a/0x9bf: [ocfs2_nodemanager] Sep 14 15:23:59 172.27.100.2 Sep 14 15:23:59 172.27.100.2 o2hb_thread+0x8e/0x3e1: [ocfs2_nodemanager] Sep 14 15:23:59 172.27.100.2 Sep 14 15:23:59 172.27.100.2 o2hb_thread+0x0/0x3e1: [ocfs2_nodemanager] Sep 14 15:23:59 172.27.100.2 Sep 14 15:23:59 172.27.100.2 kthread+0xc0/0xeb: Sep 14 15:23:59 172.27.100.2 Sep 14 15:23:59 172.27.100.2 kthread+0x0/0xeb: Sep 14 15:23:59 172.27.100.2 Sep 14 15:23:59 172.27.100.2 kernel_thread_helper+0x7/0x10: Sep 14 15:23:59 172.27.100.2 =======================: Sep 14 15:23:59 172.27.100.2 Code: Sep 14 15:23:59 172.27.100.2 17: Sep 14 15:23:59 172.27.100.2 f7: Sep 14 15:23:59 172.27.100.2 f8: Sep 14 15:23:59 172.27.100.2 ff: Sep 14 15:23:59 172.27.100.2 70: Sep 14 15:23:59 172.27.100.2 10: Sep 14 15:23:59 172.27.100.2 ff: Sep 14 15:23:59 172.27.100.2 b1: Sep 14 15:23:59 172.27.100.2 a8: Sep 14 15:23:59 172.27.100.2 00: Sep 14 15:23:59 172.27.100.2 00: Sep 14 15:23:59 172.27.100.2 00: Sep 14 15:23:59 172.27.100.2 68: Sep 14 15:23:59 172.27.100.2 40: Sep 14 15:23:59 172.27.100.2 78: Sep 14 15:23:59 172.27.100.2 f7: Sep 14 15:23:59 172.27.100.2 f8: Sep 14 15:23:59 172.27.100.2 e8: Sep 14 15:23:59 172.27.100.2 9d: Sep 14 15:23:59 172.27.100.2 e7: Sep 14 15:23:59 172.27.100.2 4b: Sep 14 15:23:59 172.27.100.2 c7: Sep 14 15:23:59 172.27.100.2 83: Sep 14 15:23:59 172.27.100.2 c4: Sep 14 15:23:59 172.27.100.2 28: Sep 14 15:23:59 172.27.100.2 0f: Sep 14 15:23:59 172.27.100.2 b6: Sep 14 15:23:59 172.27.100.2 54: Sep 14 15:23:59 172.27.100.2 24: Sep 14 15:23:59 172.27.100.2 23: Sep 14 15:23:59 172.27.100.2 0f: Sep 14 15:23:59 172.27.100.2 a3: Sep 14 15:23:59 172.27.100.2 93: Sep 14 15:23:59 172.27.100.2 b4: Sep 14 15:23:59 172.27.100.2 00: Sep 14 15:23:59 172.27.100.2 00: Sep 14 15:23:59 172.27.100.2 00: Sep 14 15:23:59 172.27.100.2 19: Sep 14 15:23:59 172.27.100.2 c0: Sep 14 15:23:59 172.27.100.2 85: Sep 14 15:23:59 172.27.100.2 c0: Sep 14 15:23:59 172.27.100.2 75: Sep 14 15:23:59 172.27.100.2 08: Sep 14 15:23:59 172.27.100.2 syslog-ng[25714]: Error processing log message: <0f> Sep 14 15:23:59 172.27.100.2 0b: Sep 14 15:23:59 172.27.100.2 95: Sep 14 15:23:59 172.27.100.2 08: Sep 14 15:23:59 172.27.100.2 40: Sep 14 15:23:59 172.27.100.2 72: Sep 14 15:23:59 172.27.100.2 f7: Sep 14 15:23:59 172.27.100.2 f8: Sep 14 15:23:59 172.27.100.2 f0: Sep 14 15:23:59 172.27.100.2 0f: Sep 14 15:23:59 172.27.100.2 b3: Sep 14 15:23:59 172.27.100.2 93: Sep 14 15:23:59 172.27.100.2 b4: Sep 14 15:23:59 172.27.100.2 00: Sep 14 15:23:59 172.27.100.2 00: Sep 14 15:23:59 172.27.100.2 00: Sep 14 15:23:59 172.27.100.2 eb: Sep 14 15:23:59 172.27.100.2 65: Sep 14 15:23:59 172.27.100.2 0f: Sep 14 15:23:59 172.27.100.2 b6: Sep 14 15:23:59 172.27.100.2 7c: Sep 14 15:23:59 172.27.100.2 Sep 14 15:23:59 172.27.100.2 EIP: [<f8f68296>] Sep 14 15:23:59 172.27.100.2 __dlm_hb_node_down+0x6f1/0x8ae: [ocfs2_dlm] Sep 14 15:23:59 172.27.100.2 SS: ESP 0068:f186ae84 Sep 14 15:23:59 172.27.100.2 Sep 14 15:23:59 172.27.100.2 kernel: Kernel panic - not syncing: Fatal exception Sep 14 15:23:59 172.27.100.2 Thank you -- Cristian Gae cristian.gae at netbridge.ro
Sunil Mushran
2009-Sep-14 19:03 UTC
[Ocfs2-users] kernel panic - BUG at dlmrecovery.c:2197
Please can you file a bugzilla. http://oss.oracle.com/bugzilla Attach the netconsole logs from all nodes. Also mention some info about your cluster... num nodes, arch, mem, etc. The oops is due to an over zealous BUG_ON. Not really required. But it does hint at a possible race in o2dlm. File the bugzilla so that we remember to fix it. Cristian Gae wrote:> Hello > > On a reboot of one of our nodes, the others have crashed with kernel > panic. This is what netconsole captured. > > Anyone have any idea what the problem is? > > > Sep 14 15:22:23 172.27.100.2 (6280,2): o2net_idle_timer:1476 here are > some times that might help debug the situation: (tmr 1252930915.974412 > now 1252930945.966783 dr 1252930915.974385 adv > 1252930915.974412:1252930915.974413 func (a423e7e1:505) > 1252930904.543344:1252930904.543352) > > > Sep 14 15:22:23 172.27.100.2 (30417,0): > dlm_send_remote_convert_request:395 ERROR: status = -112 > > > Sep 14 15:22:23 172.27.100.2 (30417,0): dlm_wait_for_node_death:370 > 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of > death of node 1 > Sep 14 15:22:24 172.27.100.2 (30129,1): > dlm_send_remote_convert_request:395 ERROR: status = -112 > > > Sep 14 15:22:24 172.27.100.2 (30129,1): dlm_wait_for_node_death:370 > 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of > death of node 1 > Sep 14 15:22:53 172.27.100.2 (2312,2): o2net_connect_expired:1637 ERROR: > no connection established with node 1 after 30.0 seconds, giving up and > returning errors. > Sep 14 15:22:53 172.27.100.2 (1653,0): dlm_do_master_request:1335 ERROR: > link to 1 went down! > > Sep 14 15:22:53 172.27.100.2 (1653,0): dlm_get_lock_resource:912 ERROR: > status = -107 > > Sep 14 15:22:53 172.27.100.2 (30417,0): > dlm_send_remote_convert_request:395 ERROR: status = -107 > > > Sep 14 15:22:53 172.27.100.2 (30417,0): dlm_wait_for_node_death:370 > 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of > death of node 1 > Sep 14 15:22:53 172.27.100.2 (30129,0): > dlm_send_remote_convert_request:395 ERROR: status = -107 > > > Sep 14 15:22:53 172.27.100.2 (30129,0): dlm_wait_for_node_death:370 > 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of > death of node 1 > Sep 14 15:23:23 172.27.100.2 (2312,3): o2net_connect_expired:1637 ERROR: > no connection established with node 1 after 30.0 seconds, giving up and > returning errors. > Sep 14 15:23:23 172.27.100.2 (1653,0): dlm_do_master_request:1335 ERROR: > link to 1 went down! > > Sep 14 15:23:23 172.27.100.2 (1653,0): dlm_get_lock_resource:912 ERROR: > status = -107 > > Sep 14 15:23:23 172.27.100.2 (30129,0): > dlm_send_remote_convert_request:395 ERROR: status = -107 > > > Sep 14 15:23:23 172.27.100.2 (30129,0): dlm_wait_for_node_death:370 > 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of > death of node 1 > Sep 14 15:23:23 172.27.100.2 (30417,0): > dlm_send_remote_convert_request:395 ERROR: status = -107 > > > Sep 14 15:23:23 172.27.100.2 (30417,0): dlm_wait_for_node_death:370 > 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of > death of node 1 > Sep 14 15:23:23 172.27.100.2 (23657,1): dlm_do_master_request:1335 > ERROR: link to 1 went down! > > Sep 14 15:23:23 172.27.100.2 (23657,1): dlm_get_lock_resource:912 ERROR: > status = -107 > > Sep 14 15:23:23 172.27.100.2 (5251,1): dlm_do_master_request:1335 ERROR: > link to 1 went down! > > Sep 14 15:23:23 172.27.100.2 (5251,1): dlm_get_lock_resource:912 ERROR: > status = -107 > > Sep 14 15:23:23 172.27.100.2 (5251,1): dlm_do_master_request:1335 ERROR: > link to 1 went down! > > Sep 14 15:23:23 172.27.100.2 (5251,1): dlm_get_lock_resource:912 ERROR: > status = -107 > > Sep 14 15:23:24 172.27.100.2 (1653,0): dlm_do_master_request:1335 ERROR: > link to 1 went down! > > Sep 14 15:23:24 172.27.100.2 (1653,0): dlm_get_lock_resource:912 ERROR: > status = -107 > > Sep 14 15:23:54 172.27.100.2 (2312,2): o2net_connect_expired:1637 ERROR: > no connection established with node 1 after 30.0 seconds, giving up and > returning errors. > Sep 14 15:23:54 172.27.100.2 (9967,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (9967,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (9908,3): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (10756,0): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (10883,0): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (9903,0): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (11183,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (10880,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (10894,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (10907,1): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (10879,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (10906,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (9841,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (10852,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (11227,3): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (10852,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (10904,3): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (4107,0): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (10907,1): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (5750,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (30417,0): > dlm_send_remote_convert_request:395 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (30417,0): dlm_wait_for_node_death:370 > 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of > death of node 1 > Sep 14 15:23:54 172.27.100.2 (30129,0): > dlm_send_remote_convert_request:395 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (30129,0): dlm_wait_for_node_death:370 > 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of > death of node 1 > Sep 14 15:23:54 172.27.100.2 (10773,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (11003,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (9539,3): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (11009,3): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (10554,3): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (10907,1): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (11235,1): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (11235,1): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (11039,1): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (9836,1): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (10553,1): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (10553,1): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (10587,1): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (10754,1): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (11306,3): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (10907,1): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (9778,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (9778,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (9778,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (9778,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (11111,3): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (9836,1): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:54 172.27.100.2 (4107,0): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:55 172.27.100.2 (10885,3): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:55 172.27.100.2 (11004,3): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:55 172.27.100.2 (9403,3): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:55 172.27.100.2 (10980,3): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:55 172.27.100.2 (10980,3): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:55 172.27.100.2 (10848,0): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:55 172.27.100.2 (10537,0): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:55 172.27.100.2 (10537,0): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:55 172.27.100.2 (8817,0): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:55 172.27.100.2 (9909,0): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:55 172.27.100.2 (9839,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:55 172.27.100.2 (9779,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:55 172.27.100.2 (9911,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:55 172.27.100.2 (9908,0): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:55 172.27.100.2 (10961,1): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:55 172.27.100.2 (1653,0): dlm_do_master_request:1335 ERROR: > link to 1 went down! > > Sep 14 15:23:55 172.27.100.2 (1653,0): dlm_get_lock_resource:912 ERROR: > status = -107 > > Sep 14 15:23:55 172.27.100.2 (9901,0): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:59 172.27.100.2 (11012,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:59 172.27.100.2 (11040,0): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:59 172.27.100.2 (2334,0): ocfs2_dlm_eviction_cb:98 device > (8,1): dlm has evicted node 1 > > Sep 14 15:23:59 172.27.100.2 (10895,2): > dlm_send_remote_unlock_request:359 ERROR: status = -107 > > > Sep 14 15:23:59 172.27.100.2 ------------: cut here ]------------ > > > Sep 14 15:23:59 172.27.100.2 kernel: BUG at > /usr/src/ocfs2-1.4.1/fs/ocfs2/dlm/dlmrecovery.c:2197! > > > Sep 14 15:23:59 172.27.100.2 invalid: opcode: 0000 [#1] > > > Sep 14 15:23:59 172.27.100.2 SMP: > > > Sep 14 15:23:59 172.27.100.2 > > > Sep 14 15:23:59 172.27.100.2 last: sysfs file: > /devices/pci0000:00/0000:00:00.0/irq > > > Sep 14 15:23:59 172.27.100.2 Modules: linked in: > > > Sep 14 15:23:59 172.27.100.2 netconsole: > > > Sep 14 15:23:59 172.27.100.2 ocfs2(U): > > > Sep 14 15:23:59 172.27.100.2 ocfs2_dlmfs(U): > > > Sep 14 15:23:59 172.27.100.2 ocfs2_dlm(U): > > > Sep 14 15:23:59 172.27.100.2 ocfs2_nodemanager(U): > > > Sep 14 15:23:59 172.27.100.2 configfs: > > > Sep 14 15:23:59 172.27.100.2 ipv6: > > > Sep 14 15:23:59 172.27.100.2 xfrm_nalgo: > > > Sep 14 15:23:59 172.27.100.2 crypto_api: > > > Sep 14 15:23:59 172.27.100.2 dm_mirror: > > > Sep 14 15:23:59 172.27.100.2 dm_multipath: > > > Sep 14 15:23:59 172.27.100.2 dm_mod: > > > Sep 14 15:23:59 172.27.100.2 video: > > > Sep 14 15:23:59 172.27.100.2 sbs: > > > Sep 14 15:23:59 172.27.100.2 backlight: > > > Sep 14 15:23:59 172.27.100.2 i2c_ec: > > > Sep 14 15:23:59 172.27.100.2 i2c_core: > > > Sep 14 15:23:59 172.27.100.2 button: > > > Sep 14 15:23:59 172.27.100.2 battery: > > > Sep 14 15:23:59 172.27.100.2 asus_acpi: > > > Sep 14 15:23:59 172.27.100.2 ac: > > > Sep 14 15:23:59 172.27.100.2 parport_pc: > > > Sep 14 15:23:59 172.27.100.2 lp: > > > Sep 14 15:23:59 172.27.100.2 parport: > > > Sep 14 15:23:59 172.27.100.2 ide_cd: > > > Sep 14 15:23:59 172.27.100.2 cdrom: > > > Sep 14 15:23:59 172.27.100.2 pcspkr: > > > Sep 14 15:23:59 172.27.100.2 serio_raw: > > > Sep 14 15:23:59 172.27.100.2 i6300esb: > > > Sep 14 15:23:59 172.27.100.2 e752x_edac: > > > Sep 14 15:23:59 172.27.100.2 edac_mc: > > > Sep 14 15:23:59 172.27.100.2 floppy: > > > Sep 14 15:23:59 172.27.100.2 tg3: > > > Sep 14 15:23:59 172.27.100.2 e1000: > > > Sep 14 15:23:59 172.27.100.2 mppVhba(U): > > > Sep 14 15:23:59 172.27.100.2 usb_storage: > > > Sep 14 15:23:59 172.27.100.2 qla2xxx: > > > Sep 14 15:23:59 172.27.100.2 scsi_transport_fc: > > > Sep 14 15:23:59 172.27.100.2 ata_piix: > > > Sep 14 15:23:59 172.27.100.2 libata: > > > Sep 14 15:23:59 172.27.100.2 cciss: > > > Sep 14 15:23:59 172.27.100.2 mppUpper(U): > > > Sep 14 15:23:59 172.27.100.2 sg: > > > Sep 14 15:23:59 172.27.100.2 sd_mod: > > > Sep 14 15:23:59 172.27.100.2 scsi_mod: > > > Sep 14 15:23:59 172.27.100.2 ext3: > > > Sep 14 15:23:59 172.27.100.2 jbd: > > > Sep 14 15:23:59 172.27.100.2 uhci_hcd: > > > Sep 14 15:23:59 172.27.100.2 ohci_hcd: > > > Sep 14 15:23:59 172.27.100.2 ehci_hcd: > > > Sep 14 15:23:59 172.27.100.2 > > > Sep 14 15:23:59 172.27.100.2 CPU: 0 > > > Sep 14 15:23:59 172.27.100.2 EIP: 0060:[<f8f68296>] Tainted: G > VLI > > Sep 14 15:23:59 172.27.100.2 EFLAGS: 00010246 (2.6.18-92.1.22.el5PAE > #1) > > Sep 14 15:23:59 172.27.100.2 EIP: is at __dlm_hb_node_down+0x6f1/0x8ae > [ocfs2_dlm] > > Sep 14 15:23:59 172.27.100.2 eax: 00000000 ebx: e90ee480 ecx: > e90ee4a8 edx: 00000001 > > Sep 14 15:23:59 172.27.100.2 esi: f4c33000 edi: e90ee498 ebp: > e90ee498 esp: f186ae84 > > Sep 14 15:23:59 172.27.100.2 ds: 007b es: 007b ss: 0068 > > > Sep 14 15:23:59 172.27.100.2 Process: o2hb-2ACADF1C94 (pid: 2334, > ti=f186a000 task=f7fc5aa0 task.ti=f186a000) > > Sep 14 15:23:59 172.27.100.2 > > > Sep 14 15:23:59 172.27.100.2 Stack: > > > Sep 14 15:23:59 172.27.100.2 00000001: > > > Sep 14 15:23:59 172.27.100.2 0000604d: > > > Sep 14 15:23:59 172.27.100.2 00000001: > > > Sep 14 15:23:59 172.27.100.2 01000292: > > > Sep 14 15:23:59 172.27.100.2 00000000: > > > Sep 14 15:23:59 172.27.100.2 01000000: > > > Sep 14 15:23:59 172.27.100.2 00000002: > > > Sep 14 15:23:59 172.27.100.2 00000001: > > > Sep 14 15:23:59 172.27.100.2 > > > Sep 14 15:23:59 172.27.100.2 > > > Sep 14 15:23:59 172.27.100.2 01000001: > > > Sep 14 15:23:59 172.27.100.2 f4c33000: > > > Sep 14 15:23:59 172.27.100.2 00000001: > > > Sep 14 15:23:59 172.27.100.2 f8d0a2a0: > > > Sep 14 15:23:59 172.27.100.2 f186af4c: > > > Sep 14 15:23:59 172.27.100.2 f8f6ae23: > > > Sep 14 15:23:59 172.27.100.2 f4c33158: > > > Sep 14 15:23:59 172.27.100.2 f4c33154: > > > Sep 14 15:23:59 172.27.100.2 > > > Sep 14 15:23:59 172.27.100.2 > > > Sep 14 15:23:59 172.27.100.2 f8cf5203: > > > Sep 14 15:23:59 172.27.100.2 00000001: > > > Sep 14 15:23:59 172.27.100.2 f1b23580: > > > Sep 14 15:23:59 172.27.100.2 c73535be: > > > Sep 14 15:23:59 172.27.100.2 e69511ff: > > > Sep 14 15:23:59 172.27.100.2 00000001: > > > Sep 14 15:23:59 172.27.100.2 f116c024: > > > Sep 14 15:23:59 172.27.100.2 f8cf5ff8: > > > Sep 14 15:23:59 172.27.100.2 > > > Sep 14 15:23:59 172.27.100.2 Call: Trace: > > > Sep 14 15:23:59 172.27.100.2 > > > Sep 14 15:23:59 172.27.100.2 dlm_hb_node_down_cb+0x35/0x46: [ocfs2_dlm] > > > Sep 14 15:23:59 172.27.100.2 > > > Sep 14 15:23:59 172.27.100.2 o2hb_run_event_list+0x112/0x15e: > [ocfs2_nodemanager] > > Sep 14 15:23:59 172.27.100.2 > > > Sep 14 15:23:59 172.27.100.2 o2hb_do_disk_heartbeat+0x85a/0x9bf: > [ocfs2_nodemanager] > > Sep 14 15:23:59 172.27.100.2 > > > Sep 14 15:23:59 172.27.100.2 o2hb_thread+0x8e/0x3e1: [ocfs2_nodemanager] > > > Sep 14 15:23:59 172.27.100.2 > > > Sep 14 15:23:59 172.27.100.2 o2hb_thread+0x0/0x3e1: [ocfs2_nodemanager] > > > Sep 14 15:23:59 172.27.100.2 > > > Sep 14 15:23:59 172.27.100.2 kthread+0xc0/0xeb: > > > Sep 14 15:23:59 172.27.100.2 > > > Sep 14 15:23:59 172.27.100.2 kthread+0x0/0xeb: > > > Sep 14 15:23:59 172.27.100.2 > > > Sep 14 15:23:59 172.27.100.2 kernel_thread_helper+0x7/0x10: > > > Sep 14 15:23:59 172.27.100.2 =======================: > > > Sep 14 15:23:59 172.27.100.2 Code: > > > Sep 14 15:23:59 172.27.100.2 17: > > > Sep 14 15:23:59 172.27.100.2 f7: > > > Sep 14 15:23:59 172.27.100.2 f8: > > > Sep 14 15:23:59 172.27.100.2 ff: > > > Sep 14 15:23:59 172.27.100.2 70: > > > Sep 14 15:23:59 172.27.100.2 10: > > > Sep 14 15:23:59 172.27.100.2 ff: > > > Sep 14 15:23:59 172.27.100.2 b1: > > > Sep 14 15:23:59 172.27.100.2 a8: > > > Sep 14 15:23:59 172.27.100.2 00: > > > Sep 14 15:23:59 172.27.100.2 00: > > > Sep 14 15:23:59 172.27.100.2 00: > > > Sep 14 15:23:59 172.27.100.2 68: > > > Sep 14 15:23:59 172.27.100.2 40: > > > Sep 14 15:23:59 172.27.100.2 78: > > > Sep 14 15:23:59 172.27.100.2 f7: > > > Sep 14 15:23:59 172.27.100.2 f8: > > > Sep 14 15:23:59 172.27.100.2 e8: > > > Sep 14 15:23:59 172.27.100.2 9d: > > > Sep 14 15:23:59 172.27.100.2 e7: > > > Sep 14 15:23:59 172.27.100.2 4b: > > > Sep 14 15:23:59 172.27.100.2 c7: > > > Sep 14 15:23:59 172.27.100.2 83: > > > Sep 14 15:23:59 172.27.100.2 c4: > Sep 14 15:23:59 172.27.100.2 28: > Sep 14 15:23:59 172.27.100.2 0f: > Sep 14 15:23:59 172.27.100.2 b6: > Sep 14 15:23:59 172.27.100.2 54: > Sep 14 15:23:59 172.27.100.2 24: > Sep 14 15:23:59 172.27.100.2 23: > Sep 14 15:23:59 172.27.100.2 0f: > Sep 14 15:23:59 172.27.100.2 a3: > Sep 14 15:23:59 172.27.100.2 93: > Sep 14 15:23:59 172.27.100.2 b4: > Sep 14 15:23:59 172.27.100.2 00: > Sep 14 15:23:59 172.27.100.2 00: > Sep 14 15:23:59 172.27.100.2 00: > Sep 14 15:23:59 172.27.100.2 19: > Sep 14 15:23:59 172.27.100.2 c0: > Sep 14 15:23:59 172.27.100.2 85: > Sep 14 15:23:59 172.27.100.2 c0: > Sep 14 15:23:59 172.27.100.2 75: > Sep 14 15:23:59 172.27.100.2 08: > Sep 14 15:23:59 172.27.100.2 syslog-ng[25714]: Error processing log > message: <0f> > Sep 14 15:23:59 172.27.100.2 0b: > Sep 14 15:23:59 172.27.100.2 95: > Sep 14 15:23:59 172.27.100.2 08: > Sep 14 15:23:59 172.27.100.2 40: > Sep 14 15:23:59 172.27.100.2 72: > Sep 14 15:23:59 172.27.100.2 f7: > Sep 14 15:23:59 172.27.100.2 f8: > Sep 14 15:23:59 172.27.100.2 f0: > Sep 14 15:23:59 172.27.100.2 0f: > Sep 14 15:23:59 172.27.100.2 b3: > Sep 14 15:23:59 172.27.100.2 93: > Sep 14 15:23:59 172.27.100.2 b4: > Sep 14 15:23:59 172.27.100.2 00: > Sep 14 15:23:59 172.27.100.2 00: > Sep 14 15:23:59 172.27.100.2 00: > Sep 14 15:23:59 172.27.100.2 eb: > Sep 14 15:23:59 172.27.100.2 65: > Sep 14 15:23:59 172.27.100.2 0f: > Sep 14 15:23:59 172.27.100.2 b6: > Sep 14 15:23:59 172.27.100.2 7c: > Sep 14 15:23:59 172.27.100.2 > Sep 14 15:23:59 172.27.100.2 EIP: [<f8f68296>] > Sep 14 15:23:59 172.27.100.2 __dlm_hb_node_down+0x6f1/0x8ae: [ocfs2_dlm] > Sep 14 15:23:59 172.27.100.2 SS: ESP 0068:f186ae84 > Sep 14 15:23:59 172.27.100.2 > Sep 14 15:23:59 172.27.100.2 kernel: Kernel panic - not syncing: Fatal > exception > Sep 14 15:23:59 172.27.100.2 > > > > Thank you > > -- > Cristian Gae > cristian.gae at netbridge.ro > > _______________________________________________ > Ocfs2-users mailing list > Ocfs2-users at oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs2-users >
Filed as bug 1175. http://oss.oracle.com/bugzilla/show_bug.cgi?id=1175 Sunil Mushran wrote:> Please can you file a bugzilla. http://oss.oracle.com/bugzilla > > Attach the netconsole logs from all nodes. Also mention some info about > your cluster... num nodes, arch, mem, etc. > > The oops is due to an over zealous BUG_ON. Not really required. But it > does hint at a possible race in o2dlm. > > File the bugzilla so that we remember to fix it. > > Cristian Gae wrote: >> Hello >> >> On a reboot of one of our nodes, the others have crashed with kernel >> panic. This is what netconsole captured. >> >> Anyone have any idea what the problem is? >> >> >> Sep 14 15:22:23 172.27.100.2 (6280,2): o2net_idle_timer:1476 here are >> some times that might help debug the situation: (tmr 1252930915.974412 >> now 1252930945.966783 dr 1252930915.974385 adv >> 1252930915.974412:1252930915.974413 func (a423e7e1:505) >> 1252930904.543344:1252930904.543352) >> >> Sep 14 15:22:23 172.27.100.2 (30417,0): >> dlm_send_remote_convert_request:395 ERROR: status = -112 >> >> Sep 14 15:22:23 172.27.100.2 (30417,0): dlm_wait_for_node_death:370 >> 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of >> death of node 1 >> Sep 14 15:22:24 172.27.100.2 (30129,1): >> dlm_send_remote_convert_request:395 ERROR: status = -112 >> >> Sep 14 15:22:24 172.27.100.2 (30129,1): dlm_wait_for_node_death:370 >> 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of >> death of node 1 >> Sep 14 15:22:53 172.27.100.2 (2312,2): o2net_connect_expired:1637 >> ERROR: no connection established with node 1 after 30.0 seconds, >> giving up and returning errors. >> Sep 14 15:22:53 172.27.100.2 (1653,0): dlm_do_master_request:1335 >> ERROR: link to 1 went down! >> Sep 14 15:22:53 172.27.100.2 (1653,0): dlm_get_lock_resource:912 >> ERROR: status = -107 >> Sep 14 15:22:53 172.27.100.2 (30417,0): >> dlm_send_remote_convert_request:395 ERROR: status = -107 >> >> Sep 14 15:22:53 172.27.100.2 (30417,0): dlm_wait_for_node_death:370 >> 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of >> death of node 1 >> Sep 14 15:22:53 172.27.100.2 (30129,0): >> dlm_send_remote_convert_request:395 ERROR: status = -107 >> >> Sep 14 15:22:53 172.27.100.2 (30129,0): dlm_wait_for_node_death:370 >> 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of >> death of node 1 >> Sep 14 15:23:23 172.27.100.2 (2312,3): o2net_connect_expired:1637 >> ERROR: no connection established with node 1 after 30.0 seconds, >> giving up and returning errors. >> Sep 14 15:23:23 172.27.100.2 (1653,0): dlm_do_master_request:1335 >> ERROR: link to 1 went down! >> Sep 14 15:23:23 172.27.100.2 (1653,0): dlm_get_lock_resource:912 >> ERROR: status = -107 >> Sep 14 15:23:23 172.27.100.2 (30129,0): >> dlm_send_remote_convert_request:395 ERROR: status = -107 >> >> Sep 14 15:23:23 172.27.100.2 (30129,0): dlm_wait_for_node_death:370 >> 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of >> death of node 1 >> Sep 14 15:23:23 172.27.100.2 (30417,0): >> dlm_send_remote_convert_request:395 ERROR: status = -107 >> >> Sep 14 15:23:23 172.27.100.2 (30417,0): dlm_wait_for_node_death:370 >> 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of >> death of node 1 >> Sep 14 15:23:23 172.27.100.2 (23657,1): dlm_do_master_request:1335 >> ERROR: link to 1 went down! >> Sep 14 15:23:23 172.27.100.2 (23657,1): dlm_get_lock_resource:912 >> ERROR: status = -107 >> Sep 14 15:23:23 172.27.100.2 (5251,1): dlm_do_master_request:1335 >> ERROR: link to 1 went down! >> Sep 14 15:23:23 172.27.100.2 (5251,1): dlm_get_lock_resource:912 >> ERROR: status = -107 >> Sep 14 15:23:23 172.27.100.2 (5251,1): dlm_do_master_request:1335 >> ERROR: link to 1 went down! >> Sep 14 15:23:23 172.27.100.2 (5251,1): dlm_get_lock_resource:912 >> ERROR: status = -107 >> Sep 14 15:23:24 172.27.100.2 (1653,0): dlm_do_master_request:1335 >> ERROR: link to 1 went down! >> Sep 14 15:23:24 172.27.100.2 (1653,0): dlm_get_lock_resource:912 >> ERROR: status = -107 >> Sep 14 15:23:54 172.27.100.2 (2312,2): o2net_connect_expired:1637 >> ERROR: no connection established with node 1 after 30.0 seconds, >> giving up and returning errors. >> Sep 14 15:23:54 172.27.100.2 (9967,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (9967,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (9908,3): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (10756,0): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (10883,0): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (9903,0): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (11183,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (10880,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (10894,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (10907,1): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (10879,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (10906,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (9841,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (10852,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (11227,3): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (10852,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (10904,3): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (4107,0): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (10907,1): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (5750,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (30417,0): >> dlm_send_remote_convert_request:395 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (30417,0): dlm_wait_for_node_death:370 >> 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of >> death of node 1 >> Sep 14 15:23:54 172.27.100.2 (30129,0): >> dlm_send_remote_convert_request:395 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (30129,0): dlm_wait_for_node_death:370 >> 2ACADF1C940347218F29577838A5F5B6: waiting 5000ms for notification of >> death of node 1 >> Sep 14 15:23:54 172.27.100.2 (10773,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (11003,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (9539,3): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (11009,3): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (10554,3): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (10907,1): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (11235,1): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (11235,1): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (11039,1): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (9836,1): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (10553,1): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (10553,1): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (10587,1): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (10754,1): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (11306,3): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (10907,1): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (9778,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (9778,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (9778,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (9778,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (11111,3): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (9836,1): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:54 172.27.100.2 (4107,0): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:55 172.27.100.2 (10885,3): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:55 172.27.100.2 (11004,3): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:55 172.27.100.2 (9403,3): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:55 172.27.100.2 (10980,3): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:55 172.27.100.2 (10980,3): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:55 172.27.100.2 (10848,0): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:55 172.27.100.2 (10537,0): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:55 172.27.100.2 (10537,0): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:55 172.27.100.2 (8817,0): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:55 172.27.100.2 (9909,0): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:55 172.27.100.2 (9839,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:55 172.27.100.2 (9779,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:55 172.27.100.2 (9911,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:55 172.27.100.2 (9908,0): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:55 172.27.100.2 (10961,1): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:55 172.27.100.2 (1653,0): dlm_do_master_request:1335 >> ERROR: link to 1 went down! >> Sep 14 15:23:55 172.27.100.2 (1653,0): dlm_get_lock_resource:912 >> ERROR: status = -107 >> Sep 14 15:23:55 172.27.100.2 (9901,0): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:59 172.27.100.2 (11012,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:59 172.27.100.2 (11040,0): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:59 172.27.100.2 (2334,0): ocfs2_dlm_eviction_cb:98 device >> (8,1): dlm has evicted node 1 >> Sep 14 15:23:59 172.27.100.2 (10895,2): >> dlm_send_remote_unlock_request:359 ERROR: status = -107 >> >> Sep 14 15:23:59 172.27.100.2 ------------: cut here ]------------ >> >> Sep 14 15:23:59 172.27.100.2 kernel: BUG at >> /usr/src/ocfs2-1.4.1/fs/ocfs2/dlm/dlmrecovery.c:2197! >> >> Sep 14 15:23:59 172.27.100.2 invalid: opcode: 0000 [#1] >> >> Sep 14 15:23:59 172.27.100.2 SMP: >> >> Sep 14 15:23:59 172.27.100.2 >> >> Sep 14 15:23:59 172.27.100.2 last: sysfs file: >> /devices/pci0000:00/0000:00:00.0/irq >> >> Sep 14 15:23:59 172.27.100.2 Modules: linked in: >> >> Sep 14 15:23:59 172.27.100.2 netconsole: >> >> Sep 14 15:23:59 172.27.100.2 ocfs2(U): >> >> Sep 14 15:23:59 172.27.100.2 ocfs2_dlmfs(U): >> >> Sep 14 15:23:59 172.27.100.2 ocfs2_dlm(U): >> >> Sep 14 15:23:59 172.27.100.2 ocfs2_nodemanager(U): >> >> Sep 14 15:23:59 172.27.100.2 configfs: >> >> Sep 14 15:23:59 172.27.100.2 ipv6: >> >> Sep 14 15:23:59 172.27.100.2 xfrm_nalgo: >> >> Sep 14 15:23:59 172.27.100.2 crypto_api: >> >> Sep 14 15:23:59 172.27.100.2 dm_mirror: >> >> Sep 14 15:23:59 172.27.100.2 dm_multipath: >> >> Sep 14 15:23:59 172.27.100.2 dm_mod: >> >> Sep 14 15:23:59 172.27.100.2 video: >> >> Sep 14 15:23:59 172.27.100.2 sbs: >> >> Sep 14 15:23:59 172.27.100.2 backlight: >> >> Sep 14 15:23:59 172.27.100.2 i2c_ec: >> >> Sep 14 15:23:59 172.27.100.2 i2c_core: >> >> Sep 14 15:23:59 172.27.100.2 button: >> >> Sep 14 15:23:59 172.27.100.2 battery: >> >> Sep 14 15:23:59 172.27.100.2 asus_acpi: >> >> Sep 14 15:23:59 172.27.100.2 ac: >> >> Sep 14 15:23:59 172.27.100.2 parport_pc: >> >> Sep 14 15:23:59 172.27.100.2 lp: >> >> Sep 14 15:23:59 172.27.100.2 parport: >> >> Sep 14 15:23:59 172.27.100.2 ide_cd: >> >> Sep 14 15:23:59 172.27.100.2 cdrom: >> >> Sep 14 15:23:59 172.27.100.2 pcspkr: >> >> Sep 14 15:23:59 172.27.100.2 serio_raw: >> >> Sep 14 15:23:59 172.27.100.2 i6300esb: >> >> Sep 14 15:23:59 172.27.100.2 e752x_edac: >> >> Sep 14 15:23:59 172.27.100.2 edac_mc: >> >> Sep 14 15:23:59 172.27.100.2 floppy: >> >> Sep 14 15:23:59 172.27.100.2 tg3: >> >> Sep 14 15:23:59 172.27.100.2 e1000: >> >> Sep 14 15:23:59 172.27.100.2 mppVhba(U): >> >> Sep 14 15:23:59 172.27.100.2 usb_storage: >> >> Sep 14 15:23:59 172.27.100.2 qla2xxx: >> >> Sep 14 15:23:59 172.27.100.2 scsi_transport_fc: >> >> Sep 14 15:23:59 172.27.100.2 ata_piix: >> >> Sep 14 15:23:59 172.27.100.2 libata: >> >> Sep 14 15:23:59 172.27.100.2 cciss: >> >> Sep 14 15:23:59 172.27.100.2 mppUpper(U): >> >> Sep 14 15:23:59 172.27.100.2 sg: >> >> Sep 14 15:23:59 172.27.100.2 sd_mod: >> >> Sep 14 15:23:59 172.27.100.2 scsi_mod: >> >> Sep 14 15:23:59 172.27.100.2 ext3: >> >> Sep 14 15:23:59 172.27.100.2 jbd: >> >> Sep 14 15:23:59 172.27.100.2 uhci_hcd: >> >> Sep 14 15:23:59 172.27.100.2 ohci_hcd: >> >> Sep 14 15:23:59 172.27.100.2 ehci_hcd: >> >> Sep 14 15:23:59 172.27.100.2 >> >> Sep 14 15:23:59 172.27.100.2 CPU: 0 >> >> Sep 14 15:23:59 172.27.100.2 EIP: 0060:[<f8f68296>] Tainted: G >> VLI >> Sep 14 15:23:59 172.27.100.2 EFLAGS: 00010246 (2.6.18-92.1.22.el5PAE >> #1) >> Sep 14 15:23:59 172.27.100.2 EIP: is at __dlm_hb_node_down+0x6f1/0x8ae >> [ocfs2_dlm] >> Sep 14 15:23:59 172.27.100.2 eax: 00000000 ebx: e90ee480 ecx: >> e90ee4a8 edx: 00000001 >> Sep 14 15:23:59 172.27.100.2 esi: f4c33000 edi: e90ee498 ebp: >> e90ee498 esp: f186ae84 >> Sep 14 15:23:59 172.27.100.2 ds: 007b es: 007b ss: 0068 >> >> Sep 14 15:23:59 172.27.100.2 Process: o2hb-2ACADF1C94 (pid: 2334, >> ti=f186a000 task=f7fc5aa0 task.ti=f186a000) >> Sep 14 15:23:59 172.27.100.2 >> >> Sep 14 15:23:59 172.27.100.2 Stack: >> >> Sep 14 15:23:59 172.27.100.2 00000001: >> >> Sep 14 15:23:59 172.27.100.2 0000604d: >> >> Sep 14 15:23:59 172.27.100.2 00000001: >> >> Sep 14 15:23:59 172.27.100.2 01000292: >> >> Sep 14 15:23:59 172.27.100.2 00000000: >> >> Sep 14 15:23:59 172.27.100.2 01000000: >> >> Sep 14 15:23:59 172.27.100.2 00000002: >> >> Sep 14 15:23:59 172.27.100.2 00000001: >> >> Sep 14 15:23:59 172.27.100.2 >> >> Sep 14 15:23:59 172.27.100.2 >> >> Sep 14 15:23:59 172.27.100.2 01000001: >> >> Sep 14 15:23:59 172.27.100.2 f4c33000: >> >> Sep 14 15:23:59 172.27.100.2 00000001: >> >> Sep 14 15:23:59 172.27.100.2 f8d0a2a0: >> >> Sep 14 15:23:59 172.27.100.2 f186af4c: >> >> Sep 14 15:23:59 172.27.100.2 f8f6ae23: >> >> Sep 14 15:23:59 172.27.100.2 f4c33158: >> >> Sep 14 15:23:59 172.27.100.2 f4c33154: >> >> Sep 14 15:23:59 172.27.100.2 >> >> Sep 14 15:23:59 172.27.100.2 >> >> Sep 14 15:23:59 172.27.100.2 f8cf5203: >> >> Sep 14 15:23:59 172.27.100.2 00000001: >> >> Sep 14 15:23:59 172.27.100.2 f1b23580: >> >> Sep 14 15:23:59 172.27.100.2 c73535be: >> >> Sep 14 15:23:59 172.27.100.2 e69511ff: >> >> Sep 14 15:23:59 172.27.100.2 00000001: >> >> Sep 14 15:23:59 172.27.100.2 f116c024: >> >> Sep 14 15:23:59 172.27.100.2 f8cf5ff8: >> >> Sep 14 15:23:59 172.27.100.2 >> >> Sep 14 15:23:59 172.27.100.2 Call: Trace: >> >> Sep 14 15:23:59 172.27.100.2 >> >> Sep 14 15:23:59 172.27.100.2 dlm_hb_node_down_cb+0x35/0x46: [ocfs2_dlm] >> >> Sep 14 15:23:59 172.27.100.2 >> >> Sep 14 15:23:59 172.27.100.2 o2hb_run_event_list+0x112/0x15e: >> [ocfs2_nodemanager] >> Sep 14 15:23:59 172.27.100.2 >> >> Sep 14 15:23:59 172.27.100.2 o2hb_do_disk_heartbeat+0x85a/0x9bf: >> [ocfs2_nodemanager] >> Sep 14 15:23:59 172.27.100.2 >> >> Sep 14 15:23:59 172.27.100.2 o2hb_thread+0x8e/0x3e1: >> [ocfs2_nodemanager] >> >> Sep 14 15:23:59 172.27.100.2 >> >> Sep 14 15:23:59 172.27.100.2 o2hb_thread+0x0/0x3e1: [ocfs2_nodemanager] >> >> Sep 14 15:23:59 172.27.100.2 >> >> Sep 14 15:23:59 172.27.100.2 kthread+0xc0/0xeb: >> >> Sep 14 15:23:59 172.27.100.2 >> >> Sep 14 15:23:59 172.27.100.2 kthread+0x0/0xeb: >> >> Sep 14 15:23:59 172.27.100.2 >> >> Sep 14 15:23:59 172.27.100.2 kernel_thread_helper+0x7/0x10: >> >> Sep 14 15:23:59 172.27.100.2 =======================: >> >> Sep 14 15:23:59 172.27.100.2 Code: >> >> Sep 14 15:23:59 172.27.100.2 17: >> >> Sep 14 15:23:59 172.27.100.2 f7: >> >> Sep 14 15:23:59 172.27.100.2 f8: >> >> Sep 14 15:23:59 172.27.100.2 ff: >> >> Sep 14 15:23:59 172.27.100.2 70: >> >> Sep 14 15:23:59 172.27.100.2 10: >> >> Sep 14 15:23:59 172.27.100.2 ff: >> >> Sep 14 15:23:59 172.27.100.2 b1: >> >> Sep 14 15:23:59 172.27.100.2 a8: >> >> Sep 14 15:23:59 172.27.100.2 00: >> >> Sep 14 15:23:59 172.27.100.2 00: >> >> Sep 14 15:23:59 172.27.100.2 00: >> >> Sep 14 15:23:59 172.27.100.2 68: >> >> Sep 14 15:23:59 172.27.100.2 40: >> >> Sep 14 15:23:59 172.27.100.2 78: >> >> Sep 14 15:23:59 172.27.100.2 f7: >> >> Sep 14 15:23:59 172.27.100.2 f8: >> >> Sep 14 15:23:59 172.27.100.2 e8: >> >> Sep 14 15:23:59 172.27.100.2 9d: >> >> Sep 14 15:23:59 172.27.100.2 e7: >> >> Sep 14 15:23:59 172.27.100.2 4b: >> >> Sep 14 15:23:59 172.27.100.2 c7: >> >> Sep 14 15:23:59 172.27.100.2 83: >> >> Sep 14 15:23:59 172.27.100.2 c4: >> Sep 14 15:23:59 172.27.100.2 28: >> Sep 14 15:23:59 172.27.100.2 0f: >> Sep 14 15:23:59 172.27.100.2 b6: >> Sep 14 15:23:59 172.27.100.2 54: >> Sep 14 15:23:59 172.27.100.2 24: >> Sep 14 15:23:59 172.27.100.2 23: >> Sep 14 15:23:59 172.27.100.2 0f: >> Sep 14 15:23:59 172.27.100.2 a3: >> Sep 14 15:23:59 172.27.100.2 93: >> Sep 14 15:23:59 172.27.100.2 b4: >> Sep 14 15:23:59 172.27.100.2 00: >> Sep 14 15:23:59 172.27.100.2 00: >> Sep 14 15:23:59 172.27.100.2 00: >> Sep 14 15:23:59 172.27.100.2 19: >> Sep 14 15:23:59 172.27.100.2 c0: >> Sep 14 15:23:59 172.27.100.2 85: >> Sep 14 15:23:59 172.27.100.2 c0: >> Sep 14 15:23:59 172.27.100.2 75: >> Sep 14 15:23:59 172.27.100.2 08: >> Sep 14 15:23:59 172.27.100.2 syslog-ng[25714]: Error processing log >> message: <0f> >> Sep 14 15:23:59 172.27.100.2 0b: >> Sep 14 15:23:59 172.27.100.2 95: >> Sep 14 15:23:59 172.27.100.2 08: >> Sep 14 15:23:59 172.27.100.2 40: >> Sep 14 15:23:59 172.27.100.2 72: >> Sep 14 15:23:59 172.27.100.2 f7: >> Sep 14 15:23:59 172.27.100.2 f8: >> Sep 14 15:23:59 172.27.100.2 f0: >> Sep 14 15:23:59 172.27.100.2 0f: >> Sep 14 15:23:59 172.27.100.2 b3: >> Sep 14 15:23:59 172.27.100.2 93: >> Sep 14 15:23:59 172.27.100.2 b4: >> Sep 14 15:23:59 172.27.100.2 00: >> Sep 14 15:23:59 172.27.100.2 00: >> Sep 14 15:23:59 172.27.100.2 00: >> Sep 14 15:23:59 172.27.100.2 eb: >> Sep 14 15:23:59 172.27.100.2 65: >> Sep 14 15:23:59 172.27.100.2 0f: >> Sep 14 15:23:59 172.27.100.2 b6: >> Sep 14 15:23:59 172.27.100.2 7c: >> Sep 14 15:23:59 172.27.100.2 >> Sep 14 15:23:59 172.27.100.2 EIP: [<f8f68296>] >> Sep 14 15:23:59 172.27.100.2 __dlm_hb_node_down+0x6f1/0x8ae: [ocfs2_dlm] >> Sep 14 15:23:59 172.27.100.2 SS: ESP 0068:f186ae84 >> Sep 14 15:23:59 172.27.100.2 >> Sep 14 15:23:59 172.27.100.2 kernel: Kernel panic - not syncing: Fatal >> exception >> Sep 14 15:23:59 172.27.100.2 >> >> >> >> Thank you >> >> -- >> Cristian Gae >> cristian.gae at netbridge.ro >> >> _______________________________________________ >> Ocfs2-users mailing list >> Ocfs2-users at oss.oracle.com >> http://oss.oracle.com/mailman/listinfo/ocfs2-users >> >-- Cristian Gae Director IT Netbridge Services cristian.gae at netbridge.ro 0749 018 817 -- Acest mesaj impreuna cu fisierele transmise constituie o informatie confidentiala si se adreseaza numai persoanei/persoanelor fizice sau juridice mentionata/e ca destinatar. Daca nu sunteti destinatarul acestui mesaj si ati primit e-mailul din greseala, va rugam anuntati administratorul de sistem. Va aducem la cunostinta ca opiniile exprimate in acest e-mail reprezinta punctul de vedere al autorului si nu cel al intregii societati. Primitorul trebuie sa verifice existenta unor virusi in acest e-mail si in continutul fisierele atasate. Societatea Netbridge Services SRL nu este responsabila pentru transmiterea necorespunzatoare a informatiei cauzate de un virus.