Ocfs users - Nov 2003 - Disappointing IMPORT Performance Using 9i RAC with OCFS on Linux

Skipped content of type multipart/alternative

well, you should compare it with rhas21 isntead of rh8

one thing the rhas21 kernel wasn't good at was io grouping, it submitted
really small io's seperately.

which kernel is this e.25 ? 

are you doing loads in parallel on 2 nodes or just a single node and you
just have 2 running ?

you should download the aio patch and try with aio enabled.
lemme dig up the oracle patch you can grab

Wim


On Fri, Nov 21, 2003 at 05:08:58PM -0000, Kevin Miller
wrote:
> I am currently involved in setting up a new Oracle environment using 9i
> RAC with OCFS on Linux. Part of the set-up involves importing the
> application schema which comprises 5.5Gb data and 3.7Gb indexes. The
> initial results are disappointing:
> 
> 	Linux non-RAC    Linux RAC-OCFS
>                  60               143
> 
> The results are elapsed time in minutes to complete the full schema
> import (using imp). They are average values from several runs, the
> results from each individual run were fairly consistent. Neither the
> server nor the database were restarted between runs.
> 
> The same dump file was used for each import.
> 
> The top five wait events for Linux RAC-OCFS database are:
> 
> Top 5 Timed Events
> ~~~~~~~~~~~~~~~~~~                                                     %
> Total
> Event                                               Waits    Time (s)
> Ela Time
> -------------------------------------------- ------------ -----------
> --------
> io done                                         1,457,317       5,550
> 51.84
> CPU time                                                        1,628
> 15.20
> log buffer space                                   17,957       1,271
> 11.88
> log file sequential read                            6,174         666
> 6.22
> log file parallel write                            19,632         591
> 5.52
> 
> 
> For the Linux non-RAC database:
> 
> Top 5 Timed Events
> ~~~~~~~~~~~~~~~~~~                                                     %
> Total
> Event                                               Waits    Time (s)
> Ela Time
> -------------------------------------------- ------------ -----------
> --------
> log file sequential read                            6,254       1,280
> 28.25
> CPU time                                                        1,242
> 27.40
> log buffer space                                    8,673         999
> 22.03
> db file scattered read                             51,788         375
> 8.28
> free buffer waits                                     744         330
> 7.28
> 
> 
> First glance would suggest that the Linux RAC-OCFS database is pretty
> much IO bound, but this system has a much higher specification IO
> subsystem then the Linux non-RAC system (see below).
> 
> The specification of the systems is roughly:
> 
> Oracle 9.2.0.4.
> 
> Linux non-RAC
> RedHat 8.0 on Dual 1.8GHz Xeon (with hyperthreading), 1Gb RAM, 2 Ultra
> 320 SCSI disks (not striped or mirrored)
> 
> Linux RAC-OCFS
> 2*(RedHat AS2.1 on IBM x360, Quad 1.5GHz Xeon, 7Gb RAM), fibre connected
> FastT Disk Array
> 
> 
> Any comments?
> 
> (By the way, because I don't do full schema imports very often I'm
not
> excited about import performance in its own right, but I am worried that
> the relatively poor performance is symptomatic of a more general issue
> that will affect the application itself)
> 
> 
> Kevin Miller
> Oracle Database Administrator
> Burns e-Commerce Solutions
> 
> 
> 
> 
> 
> 
> 	
> _______________________________________________
> Ocfs-users mailing list
> Ocfs-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs-users

Wim,
> well, you should compare it with rhas21 isntead of rh8
I agree.
> which kernel is this e.25 ?
2.4.9-e.24enterprise
> are you doing loads in parallel on 2 nodes or just a single node and
you
just have 2 running

Single node, no (application) activity on the second node.

Thanks for your help.

Kevin.

-----Original Message-----
From: Wim Coekaerts [mailto:wim.coekaerts@oracle.com] 
Sent: 21 November 2003 19:41
To: Kevin Miller
Cc: ocfs-users@oss.oracle.com; ocfs-devel@oss.oracle.com
Subject: Re: [Ocfs-users] Disappointing IMPORT Performance Using 9i RAC
with OCFS on Linux

well, you should compare it with rhas21 isntead of rh8

one thing the rhas21 kernel wasn't good at was io grouping, it submitted
really small io's seperately.

which kernel is this e.25 ? 

are you doing loads in parallel on 2 nodes or just a single node and you
just have 2 running ?

you should download the aio patch and try with aio enabled.
lemme dig up the oracle patch you can grab

Wim


On Fri, Nov 21, 2003 at 05:08:58PM -0000, Kevin Miller
wrote:
> I am currently involved in setting up a new Oracle environment using
9i
> RAC with OCFS on Linux. Part of the set-up involves importing the
> application schema which comprises 5.5Gb data and 3.7Gb indexes. The
> initial results are disappointing:
> 
> 	Linux non-RAC    Linux RAC-OCFS
>                  60               143
> 
> The results are elapsed time in minutes to complete the full schema
> import (using imp). They are average values from several runs, the
> results from each individual run were fairly consistent. Neither the
> server nor the database were restarted between runs.
> 
> The same dump file was used for each import.
> 
> The top five wait events for Linux RAC-OCFS database are:
> 
> Top 5 Timed Events
> ~~~~~~~~~~~~~~~~~~
%
> Total
> Event                                               Waits    Time (s)
> Ela Time
> -------------------------------------------- ------------ -----------
> --------
> io done                                         1,457,317       5,550
> 51.84
> CPU time                                                        1,628
> 15.20
> log buffer space                                   17,957       1,271
> 11.88
> log file sequential read                            6,174         666
> 6.22
> log file parallel write                            19,632         591
> 5.52
> 
> 
> For the Linux non-RAC database:
> 
> Top 5 Timed Events
> ~~~~~~~~~~~~~~~~~~
%
> Total
> Event                                               Waits    Time (s)
> Ela Time
> -------------------------------------------- ------------ -----------
> --------
> log file sequential read                            6,254       1,280
> 28.25
> CPU time                                                        1,242
> 27.40
> log buffer space                                    8,673         999
> 22.03
> db file scattered read                             51,788         375
> 8.28
> free buffer waits                                     744         330
> 7.28
> 
> 
> First glance would suggest that the Linux RAC-OCFS database is pretty
> much IO bound, but this system has a much higher specification IO
> subsystem then the Linux non-RAC system (see below).
> 
> The specification of the systems is roughly:
> 
> Oracle 9.2.0.4.
> 
> Linux non-RAC
> RedHat 8.0 on Dual 1.8GHz Xeon (with hyperthreading), 1Gb RAM, 2 Ultra
> 320 SCSI disks (not striped or mirrored)
> 
> Linux RAC-OCFS
> 2*(RedHat AS2.1 on IBM x360, Quad 1.5GHz Xeon, 7Gb RAM), fibre
connected
> FastT Disk Array
> 
> 
> Any comments?
> 
> (By the way, because I don't do full schema imports very often I'm
not
> excited about import performance in its own right, but I am worried
that
> the relatively poor performance is symptomatic of a more general issue
> that will affect the application itself)
> 
> 
> Kevin Miller
> Oracle Database Administrator
> Burns e-Commerce Solutions
> 
> 
> 
> 
> 
> 
> 	
> _______________________________________________
> Ocfs-users mailing list
> Ocfs-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs-users

Correction, kernel is 2.4.9-e.27enterprise

-----Original Message-----
From: Wim Coekaerts [mailto:wim.coekaerts@oracle.com] 
Sent: 21 November 2003 19:41
To: Kevin Miller
Cc: ocfs-users@oss.oracle.com; ocfs-devel@oss.oracle.com
Subject: Re: [Ocfs-users] Disappointing IMPORT Performance Using 9i RAC
with OCFS on Linux

well, you should compare it with rhas21 isntead of rh8

one thing the rhas21 kernel wasn't good at was io grouping, it submitted
really small io's seperately.

which kernel is this e.25 ? 

are you doing loads in parallel on 2 nodes or just a single node and you
just have 2 running ?

you should download the aio patch and try with aio enabled.
lemme dig up the oracle patch you can grab

Wim


On Fri, Nov 21, 2003 at 05:08:58PM -0000, Kevin Miller
wrote:
> I am currently involved in setting up a new Oracle environment using
9i
> RAC with OCFS on Linux. Part of the set-up involves importing the
> application schema which comprises 5.5Gb data and 3.7Gb indexes. The
> initial results are disappointing:
> 
> 	Linux non-RAC    Linux RAC-OCFS
>                  60               143
> 
> The results are elapsed time in minutes to complete the full schema
> import (using imp). They are average values from several runs, the
> results from each individual run were fairly consistent. Neither the
> server nor the database were restarted between runs.
> 
> The same dump file was used for each import.
> 
> The top five wait events for Linux RAC-OCFS database are:
> 
> Top 5 Timed Events
> ~~~~~~~~~~~~~~~~~~
%
> Total
> Event                                               Waits    Time (s)
> Ela Time
> -------------------------------------------- ------------ -----------
> --------
> io done                                         1,457,317       5,550
> 51.84
> CPU time                                                        1,628
> 15.20
> log buffer space                                   17,957       1,271
> 11.88
> log file sequential read                            6,174         666
> 6.22
> log file parallel write                            19,632         591
> 5.52
> 
> 
> For the Linux non-RAC database:
> 
> Top 5 Timed Events
> ~~~~~~~~~~~~~~~~~~
%
> Total
> Event                                               Waits    Time (s)
> Ela Time
> -------------------------------------------- ------------ -----------
> --------
> log file sequential read                            6,254       1,280
> 28.25
> CPU time                                                        1,242
> 27.40
> log buffer space                                    8,673         999
> 22.03
> db file scattered read                             51,788         375
> 8.28
> free buffer waits                                     744         330
> 7.28
> 
> 
> First glance would suggest that the Linux RAC-OCFS database is pretty
> much IO bound, but this system has a much higher specification IO
> subsystem then the Linux non-RAC system (see below).
> 
> The specification of the systems is roughly:
> 
> Oracle 9.2.0.4.
> 
> Linux non-RAC
> RedHat 8.0 on Dual 1.8GHz Xeon (with hyperthreading), 1Gb RAM, 2 Ultra
> 320 SCSI disks (not striped or mirrored)
> 
> Linux RAC-OCFS
> 2*(RedHat AS2.1 on IBM x360, Quad 1.5GHz Xeon, 7Gb RAM), fibre
connected
> FastT Disk Array
> 
> 
> Any comments?
> 
> (By the way, because I don't do full schema imports very often I'm
not
> excited about import performance in its own right, but I am worried
that
> the relatively poor performance is symptomatic of a more general issue
> that will affect the application itself)
> 
> 
> Kevin Miller
> Oracle Database Administrator
> Burns e-Commerce Solutions
> 
> 
> 
> 
> 
> 
> 	
> _______________________________________________
> Ocfs-users mailing list
> Ocfs-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs-users

Wim,

I have some further surprising results.

I wanted to isolate the write-intensive performance problem so I wrote a
bit of SQL which Cartesian-joins a couple of small-ish tables to create
a 1.2Gb table. I did this NOLOGGING to eliminate redo generation from
the test:

    EXT3        RAW       OCFS
     101        187        508

The results are elapsed time in seconds to create the table, averaged
over several runs (apart from the ext3 test which was a single run).

The RAW/OCFS tests were run within the same database; the RAW test used
a  tablespace created on a RAW partition, the OCFS test used a
tablespace created on an OCFS partition.

The EXT3 test was actually performed within the RMAN repository which is
non-RAC and runs on one node of the cluster.

Ok, these tests are not very scientific: the RAW partition is on a
different set of disks to the OCFS partition, the OCFS partition is on
the same set of disks as other bits of the database. But something just
doesn't feel right. And I really want to get this working. Its always
possible of course that the guys who set up the disk array/filesystems
have got it wrong and its nothing to do with the DB/OCFS layers. 

Kevin

-----Original Message-----
From: Wim Coekaerts [mailto:wim.coekaerts@oracle.com] 
Sent: 21 November 2003 19:41
To: Kevin Miller
Cc: ocfs-users@oss.oracle.com; ocfs-devel@oss.oracle.com
Subject: Re: [Ocfs-users] Disappointing IMPORT Performance Using 9i RAC
with OCFS on Linux

well, you should compare it with rhas21 isntead of rh8

one thing the rhas21 kernel wasn't good at was io grouping, it submitted
really small io's seperately.

which kernel is this e.25 ? 

are you doing loads in parallel on 2 nodes or just a single node and you
just have 2 running ?

you should download the aio patch and try with aio enabled.
lemme dig up the oracle patch you can grab

Wim


On Fri, Nov 21, 2003 at 05:08:58PM -0000, Kevin Miller
wrote:
> I am currently involved in setting up a new Oracle environment using
9i
> RAC with OCFS on Linux. Part of the set-up involves importing the
> application schema which comprises 5.5Gb data and 3.7Gb indexes. The
> initial results are disappointing:
> 
> 	Linux non-RAC    Linux RAC-OCFS
>                  60               143
> 
> The results are elapsed time in minutes to complete the full schema
> import (using imp). They are average values from several runs, the
> results from each individual run were fairly consistent. Neither the
> server nor the database were restarted between runs.
> 
> The same dump file was used for each import.
> 
> The top five wait events for Linux RAC-OCFS database are:
> 
> Top 5 Timed Events
> ~~~~~~~~~~~~~~~~~~
%
> Total
> Event                                               Waits    Time (s)
> Ela Time
> -------------------------------------------- ------------ -----------
> --------
> io done                                         1,457,317       5,550
> 51.84
> CPU time                                                        1,628
> 15.20
> log buffer space                                   17,957       1,271
> 11.88
> log file sequential read                            6,174         666
> 6.22
> log file parallel write                            19,632         591
> 5.52
> 
> 
> For the Linux non-RAC database:
> 
> Top 5 Timed Events
> ~~~~~~~~~~~~~~~~~~
%
> Total
> Event                                               Waits    Time (s)
> Ela Time
> -------------------------------------------- ------------ -----------
> --------
> log file sequential read                            6,254       1,280
> 28.25
> CPU time                                                        1,242
> 27.40
> log buffer space                                    8,673         999
> 22.03
> db file scattered read                             51,788         375
> 8.28
> free buffer waits                                     744         330
> 7.28
> 
> 
> First glance would suggest that the Linux RAC-OCFS database is pretty
> much IO bound, but this system has a much higher specification IO
> subsystem then the Linux non-RAC system (see below).
> 
> The specification of the systems is roughly:
> 
> Oracle 9.2.0.4.
> 
> Linux non-RAC
> RedHat 8.0 on Dual 1.8GHz Xeon (with hyperthreading), 1Gb RAM, 2 Ultra
> 320 SCSI disks (not striped or mirrored)
> 
> Linux RAC-OCFS
> 2*(RedHat AS2.1 on IBM x360, Quad 1.5GHz Xeon, 7Gb RAM), fibre
connected
> FastT Disk Array
> 
> 
> Any comments?
> 
> (By the way, because I don't do full schema imports very often I'm
not
> excited about import performance in its own right, but I am worried
that
> the relatively poor performance is symptomatic of a more general issue
> that will affect the application itself)
> 
> 
> Kevin Miller
> Oracle Database Administrator
> Burns e-Commerce Solutions
> 
> 
> 
> 
> 
> 
> 	
> _______________________________________________
> Ocfs-users mailing list
> Ocfs-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs-users

Wim,

This may be a false alarm.

I have re-written the test so that there are virtually no READs in order
to build the large table, which was impacting the OCFS test more than
the RAW test. So the tests are very close to measuring the pure WRITE
performance by Oracle into the two tablespace types. 

The results are now similar between RAW and OCFS.

Regards,

Kevin

-----Original Message-----
From: Kevin Miller 
Sent: 26 November 2003 11:19
To: 'Wim Coekaerts'
Cc: ocfs-users@oss.oracle.com; ocfs-devel@oss.oracle.com
Subject: RE: [Ocfs-users] Disappointing IMPORT Performance Using 9i RAC
with OCFS on Linux

Wim,

I have some further surprising results.

I wanted to isolate the write-intensive performance problem so I wrote a
bit of SQL which Cartesian-joins a couple of small-ish tables to create
a 1.2Gb table. I did this NOLOGGING to eliminate redo generation from
the test:

    EXT3        RAW       OCFS
     101        187        508

The results are elapsed time in seconds to create the table, averaged
over several runs (apart from the ext3 test which was a single run).

The RAW/OCFS tests were run within the same database; the RAW test used
a  tablespace created on a RAW partition, the OCFS test used a
tablespace created on an OCFS partition.

The EXT3 test was actually performed within the RMAN repository which is
non-RAC and runs on one node of the cluster.

Ok, these tests are not very scientific: the RAW partition is on a
different set of disks to the OCFS partition, the OCFS partition is on
the same set of disks as other bits of the database. But something just
doesn't feel right. And I really want to get this working. Its always
possible of course that the guys who set up the disk array/filesystems
have got it wrong and its nothing to do with the DB/OCFS layers. 

Kevin

-----Original Message-----
From: Wim Coekaerts [mailto:wim.coekaerts@oracle.com] 
Sent: 21 November 2003 19:41
To: Kevin Miller
Cc: ocfs-users@oss.oracle.com; ocfs-devel@oss.oracle.com
Subject: Re: [Ocfs-users] Disappointing IMPORT Performance Using 9i RAC
with OCFS on Linux

well, you should compare it with rhas21 isntead of rh8

one thing the rhas21 kernel wasn't good at was io grouping, it submitted
really small io's seperately.

which kernel is this e.25 ? 

are you doing loads in parallel on 2 nodes or just a single node and you
just have 2 running ?

you should download the aio patch and try with aio enabled.
lemme dig up the oracle patch you can grab

Wim


On Fri, Nov 21, 2003 at 05:08:58PM -0000, Kevin Miller
wrote:
> I am currently involved in setting up a new Oracle environment using
9i
> RAC with OCFS on Linux. Part of the set-up involves importing the
> application schema which comprises 5.5Gb data and 3.7Gb indexes. The
> initial results are disappointing:
> 
> 	Linux non-RAC    Linux RAC-OCFS
>                  60               143
> 
> The results are elapsed time in minutes to complete the full schema
> import (using imp). They are average values from several runs, the
> results from each individual run were fairly consistent. Neither the
> server nor the database were restarted between runs.
> 
> The same dump file was used for each import.
> 
> The top five wait events for Linux RAC-OCFS database are:
> 
> Top 5 Timed Events
> ~~~~~~~~~~~~~~~~~~
%
> Total
> Event                                               Waits    Time (s)
> Ela Time
> -------------------------------------------- ------------ -----------
> --------
> io done                                         1,457,317       5,550
> 51.84
> CPU time                                                        1,628
> 15.20
> log buffer space                                   17,957       1,271
> 11.88
> log file sequential read                            6,174         666
> 6.22
> log file parallel write                            19,632         591
> 5.52
> 
> 
> For the Linux non-RAC database:
> 
> Top 5 Timed Events
> ~~~~~~~~~~~~~~~~~~
%
> Total
> Event                                               Waits    Time (s)
> Ela Time
> -------------------------------------------- ------------ -----------
> --------
> log file sequential read                            6,254       1,280
> 28.25
> CPU time                                                        1,242
> 27.40
> log buffer space                                    8,673         999
> 22.03
> db file scattered read                             51,788         375
> 8.28
> free buffer waits                                     744         330
> 7.28
> 
> 
> First glance would suggest that the Linux RAC-OCFS database is pretty
> much IO bound, but this system has a much higher specification IO
> subsystem then the Linux non-RAC system (see below).
> 
> The specification of the systems is roughly:
> 
> Oracle 9.2.0.4.
> 
> Linux non-RAC
> RedHat 8.0 on Dual 1.8GHz Xeon (with hyperthreading), 1Gb RAM, 2 Ultra
> 320 SCSI disks (not striped or mirrored)
> 
> Linux RAC-OCFS
> 2*(RedHat AS2.1 on IBM x360, Quad 1.5GHz Xeon, 7Gb RAM), fibre
connected
> FastT Disk Array
> 
> 
> Any comments?
> 
> (By the way, because I don't do full schema imports very often I'm
not
> excited about import performance in its own right, but I am worried
that
> the relatively poor performance is symptomatic of a more general issue
> that will affect the application itself)
> 
> 
> Kevin Miller
> Oracle Database Administrator
> Burns e-Commerce Solutions
> 
> 
> 
> 
> 
> 
> 	
> _______________________________________________
> Ocfs-users mailing list
> Ocfs-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs-users