Ocfs2 users - Mar 2011 - What could cause slow down betwen OCFS2 1.2.9 and 1.4.4

We upgraded our production database cluster (6 node) from EL4 Update 5 to EL5
Update 5, including upgrading OCFS2 from 1.2.9 to 1.4.4.

We are now noticing slowdown of batch jobs in Oracle, while hotbackup runs
faster. One thing we saw is that journal mode changed from write-back to
ordered, as we don't specify journal mode during mount. Oracle sees this as
slowdown based on higher IO latency, going from 6-8ms to 13-15ms for single
block IO. Total IO throughput has dropped.

Can this be caused by the journal mode being ordered?

Ulf.

Verifying the journal mode is easy enough. Remount with data=writeback.
It can be done one node at a time.

But since you upgraded from 4.5 to 5.5, you may have to cast a wider net
considering the entire kernel also changed.

On 03/11/2011 02:22 PM, Ulf Zimmermann wrote:
> We upgraded our production database cluster (6 node) from EL4 Update 5 to
EL5 Update 5, including upgrading OCFS2 from 1.2.9 to 1.4.4.
>
> We are now noticing slowdown of batch jobs in Oracle, while hotbackup runs
faster. One thing we saw is that journal mode changed from write-back to
ordered, as we don't specify journal mode during mount. Oracle sees this as
slowdown based on higher IO latency, going from 6-8ms to 13-15ms for single
block IO. Total IO throughput has dropped.
>
> Can this be caused by the journal mode being ordered?
>
> Ulf.
>
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users

May I ask how you change these parameter ?

Also, for a 90 percent read ocfs2, should I change to writeback as well?
-----Original Message-----
From: Ulf Zimmermann
Sent:  12/03/2011, 12:11  PM
To: Sunil Mushran
Cc: ocfs2-users at oss.oracle.com
Subject: Re: [Ocfs2-users] What could cause slow down betwen OCFS2 1.2.9 and
1.4.4


Another change we found is we used scheduler deadline, we are doing downtime
tonight to change scheduler and journal mode.


Ulf Zimmermann | Senior System Architect
OPENLANE, Inc | 2200 Bridge Parkway, Suite 202
Redwood City, CA 94065 | (650) 412-4042
ulf at openlane.com


On Mar 11, 2011, at 14:32, "Sunil Mushran" <sunil.mushran at
oracle.com> wrote:
> Verifying the journal mode is easy enough. Remount with data=writeback.
> It can be done one node at a time.
> 
> But since you upgraded from 4.5 to 5.5, you may have to cast a wider net
> considering the entire kernel also changed.
> 
> On 03/11/2011 02:22 PM, Ulf Zimmermann wrote:
>> We upgraded our production database cluster (6 node) from EL4 Update 5
to EL5 Update 5, including upgrading OCFS2 from 1.2.9 to 1.4.4.
>> 
>> We are now noticing slowdown of batch jobs in Oracle, while hotbackup
runs faster. One thing we saw is that journal mode changed from write-back to
ordered, as we don't specify journal mode during mount. Oracle sees this as
slowdown based on higher IO latency, going from 6-8ms to 13-15ms for single
block IO. Total IO throughput has dropped.
>> 
>> Can this be caused by the journal mode being ordered?
>> 
>> Ulf.
>> 
>> 
>> _______________________________________________
>> Ocfs2-users mailing list
>> Ocfs2-users at oss.oracle.com
>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
> 
_______________________________________________
Ocfs2-users mailing list
Ocfs2-users at oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

the IO scheduler can be changed by adding "elevator=<name of scheduler
to use>" to the kernel line in grub. But in EL5 you can do this also on
the fly, by "echo <name of scheduler to use> >
/sys/block/<devicename>/queue/scheduler"

> -----Original Message-----
> From: Thomas Lau [mailto:thomaslau at esun.com]
> Sent: Friday, March 11, 2011 8:31 PM
> To: Sunil Mushran; Ulf Zimmermann
> Cc: ocfs2-users at oss.oracle.com
> Subject: RE: [Ocfs2-users] What could cause slow down betwen OCFS2 1.2.9
> and 1.4.4
> 
> May I ask how you change these parameter ?
> 
> Also, for a 90 percent read ocfs2, should I change to writeback as well?
> -----Original Message-----
> From: Ulf Zimmermann
> Sent:  12/03/2011, 12:11  PM
> To: Sunil Mushran
> Cc: ocfs2-users at oss.oracle.com
> Subject: Re: [Ocfs2-users] What could cause slow down betwen OCFS2 1.2.9
> and 1.4.4
> 
> 
> Another change we found is we used scheduler deadline, we are doing
> downtime tonight to change scheduler and journal mode.
> 
> 
> Ulf Zimmermann | Senior System Architect
> OPENLANE, Inc | 2200 Bridge Parkway, Suite 202
> Redwood City, CA 94065 | (650) 412-4042
> ulf at openlane.com
> 
> 
> On Mar 11, 2011, at 14:32, "Sunil Mushran" <sunil.mushran at
oracle.com>
> wrote:
> 
> > Verifying the journal mode is easy enough. Remount with
data=writeback.
> > It can be done one node at a time.
> >
> > But since you upgraded from 4.5 to 5.5, you may have to cast a wider
net
> > considering the entire kernel also changed.
> >
> > On 03/11/2011 02:22 PM, Ulf Zimmermann wrote:
> >> We upgraded our production database cluster (6 node) from EL4
Update 5
> to EL5 Update 5, including upgrading OCFS2 from 1.2.9 to 1.4.4.
> >>
> >> We are now noticing slowdown of batch jobs in Oracle, while
hotbackup
> runs faster. One thing we saw is that journal mode changed from write-back
> to ordered, as we don't specify journal mode during mount. Oracle sees
this
> as slowdown based on higher IO latency, going from 6-8ms to 13-15ms for
> single block IO. Total IO throughput has dropped.
> >>
> >> Can this be caused by the journal mode being ordered?
> >>
> >> Ulf.
> >>
> >>
> >> _______________________________________________
> >> Ocfs2-users mailing list
> >> Ocfs2-users at oss.oracle.com
> >> http://oss.oracle.com/mailman/listinfo/ocfs2-users
> >
> 
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users