Ocfs2 users - May 2010 - O2CB_HEARTBEAT_THRESHOLD won't take changes

Hello All,

I have multiple OCFS2 clusters on SLES10 SP2 running Xen. We needed to increase
the O2CB_HEARTBEAT_THRESHOLD from 31 up to 61 and did so successfully on 2 of
our 3 clusters.

However on one of the three clusters we are not able to change the value. The
/etc/sysconfig/o2cb file contains 61 as the threshold after reconfiguring via
/etc/init.d/o2cb configure, we reconfigure all 3 nodes at the same time after
unmounting the volume and stopping ocfs2 and o2cb on all nodes in the cluster
but on restart the value is reset to 31. We have even gone so far as to change
the default value in the /etc/init.d/o2cb script from 31 to 61 thinking that
must be where the setting of 31 is coming from and even then, the value is set
back to 31. Oh, and yes, we have rebooted with the new values in both the
/etc/sysconfig/o2cb config and with our change of the default value in
/etc/init.d/o2cb with no luck.

Is anyone aware of a condition where the O2CB_HEARTBEAT_THRESHOLD stays at 31
regardless of what it is changed to? Is the heartbeat region on disk possibly
causing this?

Any guidance in tracking this down would be appreciated.


Elliott Perrin
Senior Systems Architect

Summit with Tenzing

Direct: +1 877 767 5577 (x405)
Mobile: +1 416 995 2888
Web: www.tenzing.com
email: elliott.perrin at tenzing.com<mailto:elliott.perrin at tenzing.com>


[cid:image001.jpg at 01CB009B.E2899770]<http://www.tenzing.com/>

---------------------------------------------
Confidentiality Warning: This message and any attachments are intended only for
the use of the intended recipient(s), contain confidential proprietary business
information, and may be privileged. If you are not the intended recipient, you
are hereby notified that any review, retransmission, conversion to hard copy,
copying, circulation or other use of this message and any attachments is
strictly prohibited. If you are not the intended recipient, please notify the
sender immediately by return e-mail, and delete this message and any attachments
from your system. Thank-you!

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://oss.oracle.com/pipermail/ocfs2-users/attachments/20100531/12a0bd23/attachment.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 11621 bytes
Desc: image001.jpg
Url :
http://oss.oracle.com/pipermail/ocfs2-users/attachments/20100531/12a0bd23/attachment.jpg

Hi Elliott,
Elliott Perrin wrote:
>
> Hello All,
>
>  
>
> I have multiple OCFS2 clusters on SLES10 SP2 running Xen. We needed to 
> increase the O2CB_HEARTBEAT_THRESHOLD from 31 up to 61 and did so 
> successfully on 2 of our 3 clusters.
>
>  
>
> However on one of the three clusters we are not able to change the 
> value. The /etc/sysconfig/o2cb file contains 61 as the threshold after 
> reconfiguring via /etc/init.d/o2cb configure, we reconfigure all 3 
> nodes at the same time after unmounting the volume and stopping ocfs2 
> and o2cb on all nodes in the cluster but on restart the value is reset 
> to 31. We have even gone so far as to change the default value in the 
> /etc/init.d/o2cb script from 31 to 61 thinking that must be where the 
> setting of 31 is coming from and even then, the value is set back to 
> 31. Oh, and yes, we have rebooted with the new values in both the 
> /etc/sysconfig/o2cb config and with our change of the default value in 
> /etc/init.d/o2cb with no luck.
>
I would deem that you said that  
/sys/kernel/config/cluster/ocfs2/heartbeat/dead_threshold can't be set 
to 61, right?
You said even with restart, the heartbeat threadshold is still 31? That 
would be too strange.
So could you please do the following steps:
1. /etc/init.d/o2cb offline
this will make the cluster offline. Please make sure you didn't see 
anything in /sys/kernel/config/cluster/.
2.  grep HEARTBEAT /etc/sysconfig/o2cb and paste here so that we know 
the configuration file has been changed.
3. strace /etc/init.d/o2cb online and attach the output here. let's see 
what's going on.
4. cat /sys/kernel/config/cluster/ocfs2/heartbeat/dead_threshold to see 
the new value.

Regards,
Tao

Hello Tao,
> Hi Elliott,
> Elliott Perrin wrote:
> >
> > Hello All,
> >
> >
> >
> > I have multiple OCFS2 clusters on SLES10 SP2 running Xen. We needed
> to
> > increase the O2CB_HEARTBEAT_THRESHOLD from 31 up to 61 and did so
> > successfully on 2 of our 3 clusters.
> >
> >
> >
> > However on one of the three clusters we are not able to change the
> > value. The /etc/sysconfig/o2cb file contains 61 as the threshold
> after
> > reconfiguring via /etc/init.d/o2cb configure, we reconfigure all 3
> > nodes at the same time after unmounting the volume and stopping ocfs2
> > and o2cb on all nodes in the cluster but on restart the value is
> reset
> > to 31. We have even gone so far as to change the default value in the
> > /etc/init.d/o2cb script from 31 to 61 thinking that must be where the
> > setting of 31 is coming from and even then, the value is set back to
> > 31. Oh, and yes, we have rebooted with the new values in both the
> > /etc/sysconfig/o2cb config and with our change of the default value
> in
> > /etc/init.d/o2cb with no luck.
> >
> I would deem that you said that
> /sys/kernel/config/cluster/ocfs2/heartbeat/dead_threshold can't be set
> to 61, right?
Yes that is correct. 
> You said even with restart, the heartbeat threadshold is still 31? That
> would be too strange.
I thought the same as well. 
> So could you please do the following steps:
> 1. /etc/init.d/o2cb offline
> this will make the cluster offline. Please make sure you didn't see
> anything in /sys/kernel/config/cluster/.
That folder did disappear during restart but I am not sure if the admin I was
working with did an offline or a stop. We will do an offline during scheduled
maintenance tomorrow.
> 2.  grep HEARTBEAT /etc/sysconfig/o2cb and paste here so that we know
> the configuration file has been changed.
elliott.perrin at XXXXXXXXXXXX:~> grep HEART /etc/sysconfig/o2cb
# O2CB_HEARTBEAT_THRESHOLD: Iterations before a node is considered dead.
O2CB_HEARTBEAT_THRESHOLD=61
# O2CB_HEARTBEAT_MODE: Whether to use the native "kernel" or the
"user"
O2CB_HEARTBEAT_MODE="kernel"
> 3. strace /etc/init.d/o2cb online and attach the output here. let's see
> what's going on.
> 4. cat /sys/kernel/config/cluster/ocfs2/heartbeat/dead_threshold to see
> the new value.
elliott.perrin at XXXXXXXXXXX:~> cat
/sys/kernel/config/cluster/XXXXXXX/heartbeat/dead_threshold
31

Thank you Tao, we will be performing maintenance on this cluster in the next
couple days and will try these steps.

Cheers,
Elliott