thr3ads.net - Ocfs users - [Ocfs-users] Disappointing Performance Using 9i RAC with OCFS on Linux [Nov 2003]

If this information is useful, please help other people find it:
Share via:

Kevin Miller

2003-Nov-12 10:42 UTC

[Ocfs-users] Disappointing Performance Using 9i RAC with OCFS on Linux

Skipped content of type multipart/alternative

Kevin Miller

2003-Nov-13 05:04 UTC

head link

[Ocfs-users] Disappointing Performance Using 9i RAC with OCFS on Linux

Wim,

Thanks for your prompt response on this. The tpmC figures look very
impressive, and tpmC is read intensive.

I had already read note 236679.1 "Comparing Performance Between RAW IO
vs OCFS vs EXT2/3" which I guess is the article to which you are
referring; it made me suspect that the poor performance was due to the
lack of an OS IO cache but I wasn't sure.

The database is running with a small buffer cache currently so there is
a lot of scope to increase its size, each node in the cluster has 7Gb
RAM with no other application competing for it, although we will have to
configure a shmfs to go to a very large SGA.

I will post the results of my tuning efforts.

Kevin

-----Original Message-----
From: Wim Coekaerts [mailto:wim.coekaerts@oracle.com] 
Sent: 12 November 2003 17:40
To: Kevin Miller
Cc: ocfs-devel@oss.oracle.com; ocfs-users@oss.oracle.com
Subject: Re: [Ocfs-users] Disappointing Performance Using 9i RAC with
OCFS on Linux

yes there is a very easy explanation fro this

you do selects
so read only
every access t a local filesystem goes through the linxu buffercache
(same with solaris)

so if you have enough memory or you hav a small database and plenty of
free ram, all that free ram gets used as a free cache, raeds are
basically going to be buffered reads

for ocfs and raw the reads have to also come from disk

what the "right" test would be is to setup the system so that you have
oracle fully utilize the memory so taht all the buffereing is done in
oracle

you are one of the many people that have this problem initially and when
the system is well tuned and set up it works just fine 

example :

performance numbers I promised :

Following table illustrates the performance between ext2/ext3/ocfs/raw 
with AIO.

FS Type tpmC (TPS*) Total CPU % (User  /  Sys) BInput (Mb/s) BOutput
(Mb/s) Runid
ext2-aio 2039 (130) 35.59 ( 24.59 / 11.00 ) 11.5 1.82 2142
ext3-aio (ordered) 2243 (129) 34.3 ( 22.06 / 12.25 ) 11.7 1.9 2162
ext3-aio (journal) 2380 (130) 37.44 ( 25.28 / 12.16 ) 12.2 3.1 2156
raw-aio 10742 (349) 63.97 ( 47.44 / 16.53 ) 7.3 5.4 2191
ocfs-aio 10756 (352) 65.28 ( 47.84 / 17.44 ) 7.2 5 2092


Swap is used for ext2/ext3 (about 240Mb). No swapping for raw and ocfs.

All above runs with following parameters:
DB: Oracle 9iR2 (9.2.0.3)
TPCC Kit: 1000 Warehouse
Users: 100
Kernel : e.25 (RHAS2.1)


as you can see, ext3 and ext2 are 1/4th to 1/5th of raw/ocfs

this is on a big box, 1000 warehouses, thats like what, 500-600gb of
data.

the really really short answer is this :
Every benchmark we ever do has been on raw, because
1- filesystems don't scale
2- filesystems don't give the same raw throughput

even on single node

we would be using filesystems if those were faster, trust me.

look on oraclem etalink for articles on ocfs raw and performance, I
wrote something a while ago explain the above in more detail.

Wim

On Wed, Nov 12, 2003 at 04:43:36PM -0000, Kevin Miller
wrote:> I am currently involved in setting up a new Oracle environment using
9i> RAC with OCFS on Linux. I have a handful of complex SELECT statements
> which I am using to get an initial feel for performance, comparing the
> results with a number of existing systems, and the results are
> disappointing.
> 
> Test ID  Linux non-RAC  Solaris  RAC-OCFS  RAC-RAW
>       1              2        4        45       19
>       2             13       13        53       26
>       3              5       13        54       27
>       4              2       12        51       23
>       5             17       10        86      102
>       6              6        3        52       71 
> 
> The results are elapsed time in seconds to complete each test. They
are> average values from several runs, the results from each individual run
> were fairly consistent. Neither the server nor the database were
> restarted between runs.
> 
> The data operated on by the SELECT statements was more-or-less
identical> in each database.
> 
> The RAC-OCFS and RAC-RAW systems are just different tablespaces within
> the same RAC database, one created on OCFS, the other using a raw
> partition.
> 
> SQL-Trace output shows that on the RAC system more-or-less all of the
> time is spent waiting for db file sequential/scattered reads.
> 
> The RAC system is a much higher specification than either the Linux
> non-RAC or the Solaris systems.
> 
> The next step is to build a non-RAC database on one of the cluster
nodes> to get a view of the performance over ext3.
> 
> 
> The specification of the systems is roughly:
> 
> Oracle 9.2.0.4.
> 
> Linux non-RAC
> RedHat 8.0 on Dual 1.8GHz Xeon (with hyperthreading), 1Gb RAM, 2 Ultra
> 320 SCSI disks (not striped or mirrored)
> 
> Solaris
> Solaris 8 on Sunfire 280R, Dual CPU, 2Gb RAM, Disk subsystem details
not> known
> 
> RAC-OCFS & RAC-RAW
> 2*(RedHat AS2.1 on IBM x360, Quad 1.5GHz Xeon, 7Gb RAM), Fibre
connected> FastT Disk Array
> 
> 
> Any comments?
> 
> 
> Kevin Miller
> Oracle Database Administrator
> Burns e-Commerce Solutions
>  
> 
> _______________________________________________
> Ocfs-users mailing list
> Ocfs-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs-users

Kevin Miller

2003-Nov-14 07:20 UTC

head link

[Ocfs-users] Disappointing Performance Using 9i RAC with OCFS on Linux

Wim,

As discussed, the buffer cache has been increased to 1Gb. The results
are as follows:

Test ID  Linux non-RAC  Solaris  RAC-OCFS  RAC-RAW   RAC-OCFS-1Gb
      1              2        4        45       19              1
      2             13       13        53       26              8
      3              5       13        54       27              6
      4              2       12        51       23              4
      5             17       10        86      102             17
      6              6        3        52       71              5

As you predicted, performance has improved significantly.

We are now moving into a new phase of testing using the real
application, I will post any interesting results.

Thanks for your help!

Regards,

Kevin

-----Original Message-----
From: Wim Coekaerts [mailto:wim.coekaerts@oracle.com] 
Sent: 12 November 2003 17:40
To: Kevin Miller
Cc: ocfs-devel@oss.oracle.com; ocfs-users@oss.oracle.com
Subject: Re: [Ocfs-users] Disappointing Performance Using 9i RAC with
OCFS on Linux

yes there is a very easy explanation fro this

you do selects
so read only
every access t a local filesystem goes through the linxu buffercache
(same with solaris)

so if you have enough memory or you hav a small database and plenty of
free ram, all that free ram gets used as a free cache, raeds are
basically going to be buffered reads

for ocfs and raw the reads have to also come from disk

what the "right" test would be is to setup the system so that you have
oracle fully utilize the memory so taht all the buffereing is done in
oracle

you are one of the many people that have this problem initially and when
the system is well tuned and set up it works just fine 

example :

performance numbers I promised :

Following table illustrates the performance between ext2/ext3/ocfs/raw 
with AIO.

FS Type tpmC (TPS*) Total CPU % (User  /  Sys) BInput (Mb/s) BOutput
(Mb/s) Runid
ext2-aio 2039 (130) 35.59 ( 24.59 / 11.00 ) 11.5 1.82 2142
ext3-aio (ordered) 2243 (129) 34.3 ( 22.06 / 12.25 ) 11.7 1.9 2162
ext3-aio (journal) 2380 (130) 37.44 ( 25.28 / 12.16 ) 12.2 3.1 2156
raw-aio 10742 (349) 63.97 ( 47.44 / 16.53 ) 7.3 5.4 2191
ocfs-aio 10756 (352) 65.28 ( 47.84 / 17.44 ) 7.2 5 2092


Swap is used for ext2/ext3 (about 240Mb). No swapping for raw and ocfs.

All above runs with following parameters:
DB: Oracle 9iR2 (9.2.0.3)
TPCC Kit: 1000 Warehouse
Users: 100
Kernel : e.25 (RHAS2.1)


as you can see, ext3 and ext2 are 1/4th to 1/5th of raw/ocfs

this is on a big box, 1000 warehouses, thats like what, 500-600gb of
data.

the really really short answer is this :
Every benchmark we ever do has been on raw, because
1- filesystems don't scale
2- filesystems don't give the same raw throughput

even on single node

we would be using filesystems if those were faster, trust me.

look on oraclem etalink for articles on ocfs raw and performance, I
wrote something a while ago explain the above in more detail.

Wim

On Wed, Nov 12, 2003 at 04:43:36PM -0000, Kevin Miller
wrote:> I am currently involved in setting up a new Oracle environment using
9i> RAC with OCFS on Linux. I have a handful of complex SELECT statements
> which I am using to get an initial feel for performance, comparing the
> results with a number of existing systems, and the results are
> disappointing.
> 
> Test ID  Linux non-RAC  Solaris  RAC-OCFS  RAC-RAW
>       1              2        4        45       19
>       2             13       13        53       26
>       3              5       13        54       27
>       4              2       12        51       23
>       5             17       10        86      102
>       6              6        3        52       71 
> 
> The results are elapsed time in seconds to complete each test. They
are> average values from several runs, the results from each individual run
> were fairly consistent. Neither the server nor the database were
> restarted between runs.
> 
> The data operated on by the SELECT statements was more-or-less
identical> in each database.
> 
> The RAC-OCFS and RAC-RAW systems are just different tablespaces within
> the same RAC database, one created on OCFS, the other using a raw
> partition.
> 
> SQL-Trace output shows that on the RAC system more-or-less all of the
> time is spent waiting for db file sequential/scattered reads.
> 
> The RAC system is a much higher specification than either the Linux
> non-RAC or the Solaris systems.
> 
> The next step is to build a non-RAC database on one of the cluster
nodes> to get a view of the performance over ext3.
> 
> 
> The specification of the systems is roughly:
> 
> Oracle 9.2.0.4.
> 
> Linux non-RAC
> RedHat 8.0 on Dual 1.8GHz Xeon (with hyperthreading), 1Gb RAM, 2 Ultra
> 320 SCSI disks (not striped or mirrored)
> 
> Solaris
> Solaris 8 on Sunfire 280R, Dual CPU, 2Gb RAM, Disk subsystem details
not> known
> 
> RAC-OCFS & RAC-RAW
> 2*(RedHat AS2.1 on IBM x360, Quad 1.5GHz Xeon, 7Gb RAM), Fibre
connected> FastT Disk Array
> 
> 
> Any comments?
> 
> 
> Kevin Miller
> Oracle Database Administrator
> Burns e-Commerce Solutions
>  
> 
> _______________________________________________
> Ocfs-users mailing list
> Ocfs-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs-users

Possibly Parallel Threads

Search for more apparently analagous threads

Ocfs users - Nov 2003 - Disappointing Performance Using 9i RAC with OCFS on Linux

[Ocfs-users] Disappointing Performance Using 9i RAC with OCFS on Linux

[Ocfs-users] Disappointing Performance Using 9i RAC with OCFS on Linux

[Ocfs-users] Disappointing Performance Using 9i RAC with OCFS on Linux

Possibly Parallel Threads