Having major OCFS2 blues here... Still having issues maintaining a stable cluster. I've tried isolating the issue by getting an entirely dedicated ethernet switch for the OCFS2 cluster. I've tried shutting off all machines, and slowly bringing them back online. This sort of works. So far I have been able to get 2 nodes back online, but I don't know how long until they mysteriously reboot again for no reason. What I'm currently up against is an Unknown code B o while mounting /dev/sdc on /raid2005. Check 'dmesg' for more information on this error. I check the syslog and I get [(mount.ocfs2,1388,0):dlm_join_domain:1857 Timed out joining dlm domain ] So I'm stuck because this particular server which won't mount is a key player in the cluster. Any suggestions would be greatly appreciated. Tony ==[ 16.390699] Loading iSCSI transport class v2.0-870. [ 16.476589] iscsi: registered transport (tcp) [ 16.534914] OCFS2 Node Manager 1.5.0 [ 16.562053] FS-Cache: Netfs 'nfs' registered for caching [ 16.596508] OCFS2 DLM 1.5.0 [ 16.623089] Installing knfsd (copyright (C) 1996 okir at monad.swb.de). [ 16.654034] ocfs2: Registered cluster interface o2cb [ 16.721418] OCFS2 DLMFS 1.5.0 [ 16.722765] OCFS2 User DLM kernel interface loaded [ 16.743879] iscsi: registered transport (iser) [ 16.753559] iscsid (715): /proc/715/oom_adj is deprecated, please use /proc/715/oom_score_adj instead. [ 17.867235] tg3 0000:04:00.0: eth0: Link is up at 1000 Mbps, full duplex [ 17.889970] tg3 0000:04:00.0: eth0: Flow control is off for TX and off for RX [ 17.912306] console [netcon0] enabled [ 17.913397] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 17.956188] netconsole: network logging started [ 18.610037] floppy0: no floppy controllers found [ 18.658230] tg3 0000:05:00.0: eth1: Link is up at 1000 Mbps, full duplex [ 18.658998] tg3 0000:05:00.0: eth1: Flow control is off for TX and off for RX [ 18.661089] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready [ 28.100007] eth0: no IPv6 routers present [ 28.890006] eth1: no IPv6 routers present [ 51.003765] scsi4 : iSCSI Initiator over TCP/IP [ 51.516361] scsi 4:0:0:0: Direct-Access IFT DS S16E-G2240 386C PQ: 0 ANSI: 5 [ 51.517630] sd 4:0:0:0: Attached scsi generic sg2 type 0 [ 51.518447] sd 4:0:0:0: [sdc] 70315401216 512-byte logical blocks: (36.0 TB/32.7 TiB) [ 51.543439] scsi 4:0:0:1: Enclosure IFT DS S16E-G2240 386C PQ: 0 ANSI: 4 [ 51.565949] scsi 4:0:0:1: Attached scsi generic sg3 type 13 [ 51.566039] sd 4:0:0:0: [sdc] Write Protect is off [ 51.566045] sd 4:0:0:0: [sdc] Mode Sense: 83 00 00 08 [ 51.566566] sd 4:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 51.574645] sdc: unknown partition table [ 51.700297] sd 4:0:0:0: [sdc] Attached SCSI disk [ 51.727783] ses 4:0:0:1: Attached Enclosure device [ 113.902499] o2net: accepted connection from node pedge38 (num 4) at 10.88.0.38:7777 [ 117.812160] OCFS2 1.5.0 [ 127.520176] (mount.ocfs2,1388,0):dlm_join_domain:1857 Timed out joining dlm domain A3AA504BE42E4D3D8A15248D8FCD49BB after 94000 msecs [ 127.543603] ocfs2: Unmounting device (8,32) on (node 0) [ 127.780023] o2net: no longer connected to node pedge38 (num 4) at 10.88.0.38:777 ===
To add to my previous message, After some time waiting, I try to bring the mount point up again, I get these messages, appearing like it is going to work..... [ 127.520176] (mount.ocfs2,1388,0):dlm_join_domain:1857 Timed out joining dlm domain A3AA504BE42E4D3D8A15248D8FCD49BB after 94000 msecs [ 127.543603] ocfs2: Unmounting device (8,32) on (node 0) [ 127.780023] o2net: no longer connected to node pedge38 (num 4) at 10.88.0.38:7777 [ 745.068033] o2dlm: Nodes in domain A3AA504BE42E4D3D8A15248D8FCD49BB: 1 [ 745.119791] ocfs2: Mounting device (8,32) on (node 1, slot 1) with ordered data mode. [ 745.141503] (ocfs2rec,2060,0):ocfs2_replay_journal:1601 Recovering node 3 from slot 0 on device (8,32) [ 757.582921] o2net: accepted connection from node pedge38 (num 4) at 10.88.0.38:7777 [ 758.424804] o2net: accepted connection from node pedge36 (num 3) at 10.88.0.36:7777 Then I issue a df -k to see the magic, it locks up, I wait, and eventually there is a kernel panic. Of course none of this is actually sending over netconsole to the remote logging server. Tony
I managed to get ahold of the kernel panic message because it's happening on any new machines I try to introduce to the cluster: [ 66.276054] OCFS2 1.5.0 [ 66.337531] o2dlm: Nodes in domain A3AA504BE42E4D3D8A15248D8FCD49BB: 3 5 [ 66.380092] ocfs2: Mounting device (8,16) on (node 5, slot 2) with ordered data mode. [ 66.401719] (ocfs2rec,1382,0):ocfs2_replay_journal:1601 Recovering node 1 from slot 1 on device (8,16) [ 118.890439] o2net: connected to node pedge33 (num 1) at 10.88.0.33:7777 [ 118.911765] o2net: connected to node pedge38 (num 4) at 10.88.0.38:7777 [ 240.440024] INFO: task kworker/u:3:46 blocked for more than 120 seconds. [ 240.460495] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 240.481125] kworker/u:3 D 0000000000000001 0 46 2 0x00000000 [ 240.501761] ffff88020338dbd0 0000000000000046 ffff88020338dfd8 ffff88020338c000 [ 240.522547] 0000000000013d00 ffff88020333c858 ffff88020338dfd8 0000000000013d00 [ 240.543180] ffff8802034c96e0 ffff88020333c4a0 ffff880205105b80 0000000000000001 [ 240.563579] Call Trace: [ 240.583607] [<ffffffffa03c01dd>] ocfs2_wait_for_recovery+0x7d/0xd0 [ocfs2] [ 240.604038] [<ffffffff81087fb0>] ? autoremove_wake_function+0x0/0x40 [ 240.624265] [<ffffffffa03a8088>] ocfs2_inode_lock_full_nested+0x268/0x6a0 [ocfs2] [ 240.645091] [<ffffffffa03b6026>] ? ocfs2_node_map_set_bit+0x46/0x60 [ocfs2] [ 240.666163] [<ffffffffa03bc018>] ocfs2_queue_orphans+0x68/0x260 [ocfs2] [ 240.687431] [<ffffffff81038c79>] ? default_spin_lock_flags+0x9/0x10 [ 240.708689] [<ffffffffa03bd3d4>] ocfs2_recover_orphans+0x54/0x230 [ocfs2] [ 240.729977] [<ffffffffa03bbe1c>] ? __ocfs2_wait_on_mount+0xcc/0x140 [ocfs2] [ 240.751404] [<ffffffff81087fb0>] ? autoremove_wake_function+0x0/0x40 [ 240.772669] [<ffffffffa03c0417>] ocfs2_complete_recovery+0x1e7/0x690 [ocfs2] [ 240.793894] [<ffffffffa03c0230>] ? ocfs2_complete_recovery+0x0/0x690 [ocfs2] [ 240.814874] [<ffffffff8108284d>] process_one_work+0x11d/0x420 [ 240.835518] [<ffffffff810832e9>] worker_thread+0x169/0x360 [ 240.855910] [<ffffffff81083180>] ? worker_thread+0x0/0x360 [ 240.876296] [<ffffffff81087866>] kthread+0x96/0xa0 [ 240.896305] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10 [ 240.915969] [<ffffffff810877d0>] ? kthread+0x0/0xa0 [ 240.935412] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10 [ 240.954898] INFO: task ureadahead:1384 blocked for more than 120 seconds. [ 240.974688] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 240.994902] ureadahead D 0000000000000001 0 1384 1 0x00000000 [ 241.015064] ffff880206991d08 0000000000000086 ffff880206991fd8 ffff880206990000 [ 241.035271] 0000000000013d00 ffff8802051ac858 ffff880206991fd8 0000000000013d00 [ 241.055316] ffff8802068216e0 ffff8802051ac4a0 ffff880206991ce8 0000000000000001 [ 241.075007] Call Trace: [ 241.094061] [<ffffffffa03c01dd>] ocfs2_wait_for_recovery+0x7d/0xd0 [ocfs2] [ 241.113431] [<ffffffff81087fb0>] ? autoremove_wake_function+0x0/0x40 [ 241.132382] [<ffffffffa03a8088>] ocfs2_inode_lock_full_nested+0x268/0x6a0 [ocfs2] [ 241.151399] [<ffffffffa03b8812>] ocfs2_inode_revalidate+0x72/0x2c0 [ocfs2] [ 241.170682] [<ffffffffa03b0839>] ocfs2_getattr+0x59/0x1d0 [ocfs2] [ 241.189648] [<ffffffff81169521>] vfs_getattr+0x51/0x120 [ 241.208176] [<ffffffff81169648>] vfs_fstatat+0x58/0x70 [ 241.226132] [<ffffffff8116969b>] vfs_stat+0x1b/0x20 [ 241.243766] [<ffffffff811698da>] sys_newstat+0x1a/0x40 [ 241.261254] [<ffffffff815c3955>] ? page_fault+0x25/0x30 [ 241.278749] [<ffffffff8100c002>] system_call_fastpath+0x16/0x1b
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Tony Rios [09.12.2011 09:38]:> To add to my previous message, > > After some time waiting, I try to bring the mount point up again, I > get these messages, appearing like it is going to work..... > > [ 127.520176] (mount.ocfs2,1388,0):dlm_join_domain:1857 Timed out > joining dlm domain A3AA504BE42E4D3D8A15248D8FCD49BB after 94000 > msecs [ 127.543603] ocfs2: Unmounting device (8,32) on (node 0) [ > 127.780023] o2net: no longer connected to node pedge38 (num 4) at > 10.88.0.38:7777 [ 745.068033] o2dlm: Nodes in domain > A3AA504BE42E4D3D8A15248D8FCD49BB: 1 [ 745.119791] ocfs2: Mounting > device (8,32) on (node 1, slot 1) with ordered data mode. [ > 745.141503] (ocfs2rec,2060,0):ocfs2_replay_journal:1601 Recovering > node 3 from slot 0 on device (8,32) [ 757.582921] o2net: accepted > connection from node pedge38 (num 4) at 10.88.0.38:7777 [ > 758.424804] o2net: accepted connection from node pedge36 (num 3) at > 10.88.0.36:7777 > > > Then I issue a df -k to see the magic, it locks up, I wait, and > eventually there is a kernel panic. > > Of course none of this is actually sending over netconsole to the > remote logging server. > > TonyHi Tony, since it is a timeout: what values have you configured? We had timeouts when the load went up on the boxes, resulting in occasional reboots every now and then. Plus, iSCSI data must pass over the same net, what might lead to line congestion. My values are: O2CB_HEARTBEAT_THRESHOLD=61 O2CB_IDLE_TIMEOUT_MS=60000 O2CB_KEEPALIVE_DELAY_MS=10000 O2CB_RECONNECT_DELAY_MS=10000 And since we use these settings, there was no OCFS2 timeout. Yet... Oh, and we use SLES 11 SP 1 inside ESXi VMs, with Raw Disk access to the RAID. HTH Werner -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk7iQb4ACgkQk33Krq8b42N8QQCdGPlOqM28Wl8/fKP/yBDbRjRd 6A0AmgLdoyotAAvTc/N4szS0r0thlI1U =Piu2 -----END PGP SIGNATURE-----