Michael Steinmann
2005-Nov-24 09:00 UTC
[Ocfs2-users] Heartbeat write timeout to device dm-6 after 12000 milliseconds
I'm having a hard time figuring out what's going on with our cluster. After 2 to 3 randomly fencing machines a day I have enabled 'HEARTBEAT allow' debug output, see extract of the logfile below. We have 5 volumes mounted from a Hitachi DF600F SAN over 2 multipath-tools controlled QLA2340. I hope the 12 second timeout is a non-issue here. SAN is healthy and file system performance is normal. Nodes randomly fence both under load and while idle. - What exactly is the meaning of 'changed 1 equal 0' in o2hb debug output? - How do I find the link between the disk and the heartbeat region to figure out which disk the failed heartbeat refers to? - How is the heartbeat timeout configured? - How can I debug further to find the cause of the fencing? Nov 24 13:50:01 spora01dbcb kernel: (9112,2):o2hb_check_slot:703 Slot 1 gen 0x7e62154b543a3665 cksum 0xf79c89bd seq 1132836601 last 1132836601 changed 1 equal 0 Nov 24 13:50:02 spora01dbcb kernel: (8896,3):o2hb_check_slot:703 Slot 0 gen 0x7c75fec12523ea9d cksum 0xb40682c6 seq 1132836605 last 1132836605 changed 1 equal 1 Nov 24 13:50:02 spora01dbcb kernel: (8896,3):o2hb_check_slot:703 Slot 1 gen 0xe0622786d272cb74 cksum 0x46600734 seq 1132836602 last 1132836602 changed 1 equal 0 Nov 24 13:50:02 spora01dbcb kernel: (8210,0):o2hb_check_slot:703 Slot 0 gen 0x17a46f5afd01938f cksum 0xd63a6b3 seq 1132836607 last 1132836607 changed 1 equal 0 Nov 24 13:50:02 spora01dbcb kernel: (8210,0):o2hb_check_slot:703 Slot 1 gen 0x6573425f3adee15f cksum 0x7e45fae1 seq 1132836602 last 1132836602 changed 1 equal 0 Nov 24 13:50:02 spora01dbcb kernel: (8658,2):o2hb_check_slot:703 Slot 0 gen 0xce60478d346ba3b9 cksum 0x596ca5a6 seq 1132836607 last 1132836607 changed 1 equal 0 Nov 24 13:50:02 spora01dbcb kernel: (8658,2):o2hb_check_slot:703 Slot 1 gen 0xe9b78acdc4601e0e cksum 0x85be4cf8 seq 1132836602 last 1132836602 changed 1 equal 0 Nov 24 13:50:03 spora01dbcb kernel: (9002,0):o2hb_check_slot:703 Slot 0 gen 0xd7e5ce722f261b84 cksum 0x36b381af seq 1132836607 last 1132836607 changed 1 equal 0 Nov 24 13:50:03 spora01dbcb kernel: (9002,0):o2hb_check_slot:703 Slot 1 gen 0x268f1d35bfa3b94d cksum 0xd785c8b seq 1132836603 last 1132836603 changed 1 equal 0 Nov 24 13:50:03 spora01dbcb kernel: (9112,2):o2hb_check_slot:703 Slot 0 gen 0x3ca5df451b7f0f14 cksum 0xa3693f93 seq 1132836607 last 1132836607 changed 1 equal 0 Nov 24 13:50:03 spora01dbcb kernel: (9112,2):o2hb_check_slot:703 Slot 1 gen 0x7e62154b543a3665 cksum 0xd7793fed seq 1132836603 last 1132836603 changed 1 equal 0 Nov 24 13:50:04 spora01dbcb kernel: (8896,3):o2hb_check_slot:703 Slot 0 gen 0x7c75fec12523ea9d cksum 0xf2aef875 seq 1132836609 last 1132836609 changed 1 equal 0 Nov 24 13:50:04 spora01dbcb kernel: (8896,3):o2hb_check_slot:703 Slot 1 gen 0xe0622786d272cb74 cksum 0x274eddc4 seq 1132836604 last 1132836604 changed 1 equal 0 Nov 24 13:50:04 spora01dbcb kernel: (8210,0):o2hb_check_slot:703 Slot 0 gen 0x17a46f5afd01938f cksum 0x6b2e6a50 seq 1132836609 last 1132836609 changed 1 equal 0 Nov 24 13:50:04 spora01dbcb kernel: (8210,0):o2hb_check_slot:703 Slot 1 gen 0x6573425f3adee15f cksum 0x1f6b2011 seq 1132836604 last 1132836604 changed 1 equal 0 Nov 24 13:50:04 spora01dbcb kernel: (8658,3):o2hb_check_slot:703 Slot 0 gen 0xce60478d346ba3b9 cksum 0x3f216945 seq 1132836609 last 1132836609 changed 1 equal 0 Nov 24 13:50:04 spora01dbcb kernel: (8658,3):o2hb_check_slot:703 Slot 1 gen 0xe9b78acdc4601e0e cksum 0xe4909608 seq 1132836604 last 1132836604 changed 1 equal 0 Nov 24 13:50:05 spora01dbcb kernel: (9002,3):o2hb_check_slot:703 Slot 0 gen 0xd7e5ce722f261b84 cksum 0x50fe4d4c seq 1132836609 last 1132836609 changed 1 equal 0 Nov 24 13:50:05 spora01dbcb kernel: (9002,3):o2hb_check_slot:703 Slot 1 gen 0x268f1d35bfa3b94d cksum 0x6c56867b seq 1132836605 last 1132836605 changed 1 equal 0 Nov 24 13:50:05 spora01dbcb kernel: (9112,2):o2hb_check_slot:703 Slot 0 gen 0x3ca5df451b7f0f14 cksum 0xc524f370 seq 1132836609 last 1132836609 changed 1 equal 0 Nov 24 13:50:05 spora01dbcb kernel: (9112,2):o2hb_check_slot:703 Slot 1 gen 0x7e62154b543a3665 cksum 0xb657e51d seq 1132836605 last 1132836605 changed 1 equal 0 Nov 24 13:50:06 spora01dbcb kernel: (8896,3):o2hb_check_slot:703 Slot 0 gen 0x7c75fec12523ea9d cksum 0xd24b4e25 seq 1132836611 last 1132836611 changed 1 equal 0 Nov 24 13:50:06 spora01dbcb kernel: (8896,3):o2hb_check_slot:703 Slot 1 gen 0xe0622786d272cb74 cksum 0x7ab6b94 seq 1132836606 last 1132836606 changed 1 equal 0 Nov 24 13:50:06 spora01dbcb kernel: (8210,3):o2hb_check_slot:703 Slot 0 gen 0x17a46f5afd01938f cksum 0x4bcbdc00 seq 1132836611 last 1132836611 changed 1 equal 0 Nov 24 13:50:06 spora01dbcb kernel: (8210,3):o2hb_check_slot:703 Slot 1 gen 0x6573425f3adee15f cksum 0x3f8e9641 seq 1132836606 last 1132836606 changed 1 equal 0 Nov 24 13:50:06 spora01dbcb kernel: (8658,3):o2hb_check_slot:703 Slot 0 gen 0xce60478d346ba3b9 cksum 0x1fc4df15 seq 1132836611 last 1132836611 changed 1 equal 0 Nov 24 13:50:06 spora01dbcb kernel: (8658,3):o2hb_check_slot:703 Slot 1 gen 0xe9b78acdc4601e0e cksum 0xc4752058 seq 1132836606 last 1132836606 changed 1 equal 0 Nov 24 13:50:07 spora01dbcb kernel: (9002,3):o2hb_check_slot:703 Slot 0 gen 0xd7e5ce722f261b84 cksum 0x701bfb1c seq 1132836611 last 1132836611 changed 1 equal 0 Nov 24 13:50:07 spora01dbcb kernel: (9002,3):o2hb_check_slot:703 Slot 1 gen 0x268f1d35bfa3b94d cksum 0x4cb3302b seq 1132836607 last 1132836607 changed 1 equal 0 Nov 24 13:50:07 spora01dbcb kernel: (9112,2):o2hb_check_slot:703 Slot 0 gen 0x3ca5df451b7f0f14 cksum 0xe5c14520 seq 1132836611 last 1132836611 changed 1 equal 0 Nov 24 13:50:07 spora01dbcb kernel: (9112,2):o2hb_check_slot:703 Slot 1 gen 0x7e62154b543a3665 cksum 0x96b2534d seq 1132836607 last 1132836607 changed 1 equal 0 Nov 24 13:50:08 spora01dbcb kernel: (8896,3):o2hb_check_slot:703 Slot 0 gen 0x7c75fec12523ea9d cksum 0xd24b4e25 seq 1132836611 last 1132836611 changed 0 equal 1 Nov 24 13:50:08 spora01dbcb kernel: (8896,3):o2hb_check_slot:703 Slot 1 gen 0xe0622786d272cb74 cksum 0x61e6a777 seq 1132836608 last 1132836608 changed 1 equal 0 Nov 24 13:50:08 spora01dbcb kernel: (8210,3):o2hb_check_slot:703 Slot 0 gen 0x17a46f5afd01938f cksum 0x2ae506f0 seq 1132836613 last 1132836613 changed 1 equal 0 Nov 24 13:50:08 spora01dbcb kernel: (8210,3):o2hb_check_slot:703 Slot 1 gen 0x6573425f3adee15f cksum 0x59c35aa2 seq 1132836608 last 1132836608 changed 1 equal 0 Nov 24 13:50:08 spora01dbcb kernel: (8658,3):o2hb_check_slot:703 Slot 0 gen 0xce60478d346ba3b9 cksum 0x7eea05e5 seq 1132836613 last 1132836613 changed 1 equal 0 Nov 24 13:50:08 spora01dbcb kernel: (8658,3):o2hb_check_slot:703 Slot 1 gen 0xe9b78acdc4601e0e cksum 0xa238ecbb seq 1132836608 last 1132836608 changed 1 equal 0 Nov 24 13:50:09 spora01dbcb kernel: (9002,3):o2hb_check_slot:703 Slot 0 gen 0xd7e5ce722f261b84 cksum 0x113521ec seq 1132836613 last 1132836613 changed 1 equal 0 Nov 24 13:50:09 spora01dbcb kernel: (9002,3):o2hb_check_slot:703 Slot 1 gen 0x268f1d35bfa3b94d cksum 0x2afefcc8 seq 1132836609 last 1132836609 changed 1 equal 0 Nov 24 13:50:09 spora01dbcb kernel: (9112,2):o2hb_check_slot:703 Slot 0 gen 0x3ca5df451b7f0f14 cksum 0x84ef9fd0 seq 1132836613 last 1132836613 changed 1 equal 0 Nov 24 13:50:09 spora01dbcb kernel: (9112,2):o2hb_check_slot:703 Slot 1 gen 0x7e62154b543a3665 cksum 0xf0ff9fae seq 1132836609 last 1132836609 changed 1 equal 0 Nov 24 13:50:10 spora01dbcb kernel: (8896,2):o2hb_check_slot:703 Slot 0 gen 0x7c75fec12523ea9d cksum 0x93802285 seq 1132836615 last 1132836615 changed 1 equal 1 Nov 24 13:50:10 spora01dbcb kernel: (8896,2):o2hb_check_slot:703 Slot 1 gen 0xe0622786d272cb74 cksum 0x41031127 seq 1132836610 last 1132836610 changed 1 equal 0 Nov 24 13:50:10 spora01dbcb kernel: (8210,3):o2hb_check_slot:703 Slot 0 gen 0x17a46f5afd01938f cksum 0xa00b0a0 seq 1132836615 last 1132836615 changed 1 equal 0 Nov 24 13:50:10 spora01dbcb kernel: (8210,3):o2hb_check_slot:703 Slot 1 gen 0x6573425f3adee15f cksum 0x7926ecf2 seq 1132836610 last 1132836610 changed 1 equal 0 Nov 24 13:50:10 spora01dbcb kernel: (8658,2):o2hb_check_slot:703 Slot 0 gen 0xce60478d346ba3b9 cksum 0x5e0fb3b5 seq 1132836615 last 1132836615 changed 1 equal 0 Nov 24 13:50:10 spora01dbcb kernel: (8658,2):o2hb_check_slot:703 Slot 1 gen 0xe9b78acdc4601e0e cksum 0x82dd5aeb seq 1132836610 last 1132836610 changed 1 equal 0 Nov 24 13:50:11 spora01dbcb kernel: (9002,3):o2hb_check_slot:703 Slot 0 gen 0xd7e5ce722f261b84 cksum 0x31d097bc seq 1132836615 last 1132836615 changed 1 equal 0 Nov 24 13:50:11 spora01dbcb kernel: (9002,3):o2hb_check_slot:703 Slot 1 gen 0x268f1d35bfa3b94d cksum 0xa1b4a98 seq 1132836611 last 1132836611 changed 1 equal 0 Nov 24 13:50:11 spora01dbcb kernel: (9112,2):o2hb_check_slot:703 Slot 0 gen 0x3ca5df451b7f0f14 cksum 0xa40a2980 seq 1132836615 last 1132836615 changed 1 equal 0 Nov 24 13:50:11 spora01dbcb kernel: (9112,2):o2hb_check_slot:703 Slot 1 gen 0x7e62154b543a3665 cksum 0xd01a29fe seq 1132836611 last 1132836611 changed 1 equal 0 Nov 24 13:50:12 spora01dbcb kernel: (8896,2):o2hb_check_slot:703 Slot 0 gen 0x7c75fec12523ea9d cksum 0x71382135 seq 1132836617 last 1132836617 changed 1 equal 0 Nov 24 13:50:12 spora01dbcb kernel: (8896,2):o2hb_check_slot:703 Slot 1 gen 0xe0622786d272cb74 cksum 0x202dcbd7 seq 1132836612 last 1132836612 changed 1 equal 0 Nov 24 13:50:12 spora01dbcb kernel: (8210,3):o2hb_check_slot:703 Slot 0 gen 0x17a46f5afd01938f cksum 0xe8b8b310 seq 1132836617 last 1132836617 changed 1 equal 0 Nov 24 13:50:12 spora01dbcb kernel: (8210,3):o2hb_check_slot:703 Slot 1 gen 0x6573425f3adee15f cksum 0x18083602 seq 1132836612 last 1132836612 changed 1 equal 0 Nov 24 13:50:13 spora01dbcb kernel: (8658,2):o2hb_check_slot:703 Slot 0 gen 0xce60478d346ba3b9 cksum 0xbcb7b005 seq 1132836617 last 1132836617 changed 1 equal 0 Nov 24 13:50:13 spora01dbcb kernel: (8658,2):o2hb_check_slot:703 Slot 1 gen 0xe9b78acdc4601e0e cksum 0xf3815b33 seq 1132836613 last 1132836613 changed 1 equal 0 Nov 24 13:50:13 spora01dbcb kernel: (9002,3):o2hb_check_slot:703 Slot 0 gen 0xd7e5ce722f261b84 cksum 0xd368940c seq 1132836617 last 1132836617 changed 1 equal 0 Nov 24 13:50:13 spora01dbcb kernel: (9002,3):o2hb_check_slot:703 Slot 1 gen 0x268f1d35bfa3b94d cksum 0x6b359068 seq 1132836613 last 1132836613 changed 1 equal 0 Nov 24 13:50:13 spora01dbcb kernel: (9112,2):o2hb_check_slot:703 Slot 0 gen 0x3ca5df451b7f0f14 cksum 0x46b22a30 seq 1132836617 last 1132836617 changed 1 equal 0 Nov 24 13:50:13 spora01dbcb kernel: (9112,2):o2hb_check_slot:703 Slot 1 gen 0x7e62154b543a3665 cksum 0xb134f30e seq 1132836613 last 1132836613 changed 1 equal 0 Nov 24 13:50:14 spora01dbcb kernel: (8896,2):o2hb_check_slot:703 Slot 0 gen 0x7c75fec12523ea9d cksum 0x71382135 seq 1132836617 last 1132836617 changed 0 equal 1 Nov 24 13:50:14 spora01dbcb kernel: (8896,2):o2hb_check_slot:703 Slot 1 gen 0xe0622786d272cb74 cksum 0xc87d87 seq 1132836614 last 1132836614 changed 1 equal 0 Nov 24 13:50:14 spora01dbcb kernel: (8210,2):o2hb_check_slot:703 Slot 0 gen 0x17a46f5afd01938f cksum 0xc85d0540 seq 1132836619 last 1132836619 changed 1 equal 0 Nov 24 13:50:14 spora01dbcb kernel: (8210,2):o2hb_check_slot:703 Slot 1 gen 0x6573425f3adee15f cksum 0x38ed8052 seq 1132836614 last 1132836614 changed 1 equal 0 Nov 24 13:50:15 spora01dbcb kernel: (9002,2):o2hb_check_slot:703 Slot 0 gen 0xd7e5ce722f261b84 cksum 0xf38d225c seq 1132836619 last 1132836619 changed 1 equal 0 Nov 24 13:50:15 spora01dbcb kernel: (9002,2):o2hb_check_slot:703 Slot 1 gen 0x268f1d35bfa3b94d cksum 0x4bd02638 seq 1132836615 last 1132836615 changed 1 equal 0 Nov 24 13:50:15 spora01dbcb kernel: (9112,2):o2hb_check_slot:703 Slot 0 gen 0x3ca5df451b7f0f14 cksum 0x66579c60 seq 1132836619 last 1132836619 changed 1 equal 0 Nov 24 13:50:15 spora01dbcb kernel: (9112,2):o2hb_check_slot:703 Slot 1 gen 0x7e62154b543a3665 cksum 0x91d1455e seq 1132836615 last 1132836615 changed 1 equal 0 Nov 24 13:50:16 spora01dbcb kernel: (8896,2):o2hb_check_slot:703 Slot 0 gen 0x7c75fec12523ea9d cksum 0x30f34d95 seq 1132836621 last 1132836621 changed 1 equal 1 Nov 24 13:50:16 spora01dbcb kernel: (8896,2):o2hb_check_slot:703 Slot 1 gen 0xe0622786d272cb74 cksum 0xe2707e37 seq 1132836616 last 1132836616 changed 1 equal 0 Nov 24 13:50:16 spora01dbcb kernel: (8210,2):o2hb_check_slot:703 Slot 0 gen 0x17a46f5afd01938f cksum 0xa973dfb0 seq 1132836621 last 1132836621 changed 1 equal 0 Nov 24 13:50:16 spora01dbcb kernel: (8210,2):o2hb_check_slot:703 Slot 1 gen 0x6573425f3adee15f cksum 0xda5583e2 seq 1132836616 last 1132836616 changed 1 equal 0 Nov 24 13:50:17 spora01dbcb kernel: (9002,2):o2hb_check_slot:703 Slot 0 gen 0xd7e5ce722f261b84 cksum 0x92a3f8ac seq 1132836621 last 1132836621 changed 1 equal 0 Nov 24 13:50:17 spora01dbcb kernel: (9002,2):o2hb_check_slot:703 Slot 1 gen 0x268f1d35bfa3b94d cksum 0xa9682588 seq 1132836617 last 1132836617 changed 1 equal 0 Nov 24 13:50:17 spora01dbcb kernel: (9112,2):o2hb_check_slot:703 Slot 0 gen 0x3ca5df451b7f0f14 cksum 0x7794690 seq 1132836621 last 1132836621 changed 1 equal 0 Nov 24 13:50:17 spora01dbcb kernel: (9112,2):o2hb_check_slot:703 Slot 1 gen 0x7e62154b543a3665 cksum 0x736946ee seq 1132836617 last 1132836617 changed 1 equal 0 Nov 24 13:50:18 spora01dbcb kernel: (8896,2):o2hb_check_slot:703 Slot 0 gen 0x7c75fec12523ea9d cksum 0x30f34d95 seq 1132836621 last 1132836621 changed 0 equal 1 Nov 24 13:50:18 spora01dbcb kernel: (8896,2):o2hb_check_slot:703 Slot 1 gen 0xe0622786d272cb74 cksum 0xc295c867 seq 1132836618 last 1132836618 changed 1 equal 0 Nov 24 13:50:18 spora01dbcb kernel: (8210,2):o2hb_check_slot:703 Slot 0 gen 0x17a46f5afd01938f cksum 0x899669e0 seq 1132836623 last 1132836623 changed 1 equal 0 Nov 24 13:50:18 spora01dbcb kernel: (8210,2):o2hb_check_slot:703 Slot 1 gen 0x6573425f3adee15f cksum 0xfab035b2 seq 1132836618 last 1132836618 changed 1 equal 0 Nov 24 13:50:19 spora01dbcb kernel: (9002,1):o2hb_check_slot:703 Slot 0 gen 0xd7e5ce722f261b84 cksum 0xb2464efc seq 1132836623 last 1132836623 changed 1 equal 0 Nov 24 13:50:19 spora01dbcb kernel: (9002,1):o2hb_check_slot:703 Slot 1 gen 0x268f1d35bfa3b94d cksum 0x898d93d8 seq 1132836619 last 1132836619 changed 1 equal 0 Nov 24 13:50:19 spora01dbcb kernel: (9112,1):o2hb_check_slot:703 Slot 0 gen 0x3ca5df451b7f0f14 cksum 0x279cf0c0 seq 1132836623 last 1132836623 changed 1 equal 0 Nov 24 13:50:19 spora01dbcb kernel: (9112,1):o2hb_check_slot:703 Slot 1 gen 0x7e62154b543a3665 cksum 0x538cf0be seq 1132836619 last 1132836619 changed 1 equal 0 Nov 24 13:50:20 spora01dbcb kernel: (8896,1):o2hb_check_slot:703 Slot 0 gen 0x7c75fec12523ea9d cksum 0x1016fbc5 seq 1132836623 last 1132836623 changed 1 equal 1 Nov 24 13:50:20 spora01dbcb kernel: (8896,1):o2hb_check_slot:703 Slot 1 gen 0xe0622786d272cb74 cksum 0xa3bb1297 seq 1132836620 last 1132836620 changed 1 equal 0 Nov 24 13:50:20 spora01dbcb kernel: (8210,2):o2hb_check_slot:703 Slot 0 gen 0x17a46f5afd01938f cksum 0xb772de91 seq 1132836625 last 1132836625 changed 1 equal 0 Nov 24 13:50:20 spora01dbcb kernel: (8210,2):o2hb_check_slot:703 Slot 1 gen 0x6573425f3adee15f cksum 0x9b9eef42 seq 1132836620 last 1132836620 changed 1 equal 0 Nov 24 13:50:21 spora01dbcb kernel: (9002,1):o2hb_check_slot:703 Slot 0 gen 0xd7e5ce722f261b84 cksum 0x8ca2f98d seq 1132836625 last 1132836625 changed 1 equal 0 Nov 24 13:50:21 spora01dbcb kernel: (9002,1):o2hb_check_slot:703 Slot 1 gen 0x268f1d35bfa3b94d cksum 0xe8a34928 seq 1132836621 last 1132836621 changed 1 equal 0 Nov 24 13:50:21 spora01dbcb kernel: (9112,1):o2hb_check_slot:703 Slot 0 gen 0x3ca5df451b7f0f14 cksum 0x197847b1 seq 1132836625 last 1132836625 changed 1 equal 0 Nov 24 13:50:21 spora01dbcb kernel: (9112,1):o2hb_check_slot:703 Slot 1 gen 0x7e62154b543a3665 cksum 0x32a22a4e seq 1132836621 last 1132836621 changed 1 equal 0 Nov 24 13:50:22 spora01dbcb kernel: (8896,1):o2hb_check_slot:703 Slot 0 gen 0x7c75fec12523ea9d cksum 0x2ef24cb4 seq 1132836625 last 1132836625 changed 1 equal 0 Nov 24 13:50:22 spora01dbcb kernel: (8896,1):o2hb_check_slot:703 Slot 1 gen 0xe0622786d272cb74 cksum 0x835ea4c7 seq 1132836622 last 1132836622 changed 1 equal 0 Nov 24 13:50:22 spora01dbcb kernel: (8210,1):o2hb_check_slot:703 Slot 0 gen 0x17a46f5afd01938f cksum 0x979768c1 seq 1132836627 last 1132836627 changed 1 equal 0 Nov 24 13:50:22 spora01dbcb kernel: (8210,1):o2hb_check_slot:703 Slot 1 gen 0x6573425f3adee15f cksum 0xbb7b5912 seq 1132836622 last 1132836622 changed 1 equal 0 Nov 24 13:50:22 spora01dbcb kernel: (12,2):o2hb_write_timeout:165 ERROR: Heartbeat write timeout to device dm-6 after 12000 milliseconds Nov 24 13:50:22 spora01dbcb kernel: (12,2):o2hb_stop_all_regions:1728 ERROR: stopping heartbeat on all active regions. --mike