Kevin Miller
2003-Nov-21 11:07 UTC
[Ocfs-users] Disappointing IMPORT Performance Using 9i RAC with OCFS on Linux
Skipped content of type multipart/alternative
Wim Coekaerts
2003-Nov-21 13:41 UTC
[Ocfs-users] Disappointing IMPORT Performance Using 9i RAC with OCFS on Linux
well, you should compare it with rhas21 isntead of rh8 one thing the rhas21 kernel wasn't good at was io grouping, it submitted really small io's seperately. which kernel is this e.25 ? are you doing loads in parallel on 2 nodes or just a single node and you just have 2 running ? you should download the aio patch and try with aio enabled. lemme dig up the oracle patch you can grab Wim On Fri, Nov 21, 2003 at 05:08:58PM -0000, Kevin Miller wrote:> I am currently involved in setting up a new Oracle environment using 9i > RAC with OCFS on Linux. Part of the set-up involves importing the > application schema which comprises 5.5Gb data and 3.7Gb indexes. The > initial results are disappointing: > > Linux non-RAC Linux RAC-OCFS > 60 143 > > The results are elapsed time in minutes to complete the full schema > import (using imp). They are average values from several runs, the > results from each individual run were fairly consistent. Neither the > server nor the database were restarted between runs. > > The same dump file was used for each import. > > The top five wait events for Linux RAC-OCFS database are: > > Top 5 Timed Events > ~~~~~~~~~~~~~~~~~~ % > Total > Event Waits Time (s) > Ela Time > -------------------------------------------- ------------ ----------- > -------- > io done 1,457,317 5,550 > 51.84 > CPU time 1,628 > 15.20 > log buffer space 17,957 1,271 > 11.88 > log file sequential read 6,174 666 > 6.22 > log file parallel write 19,632 591 > 5.52 > > > For the Linux non-RAC database: > > Top 5 Timed Events > ~~~~~~~~~~~~~~~~~~ % > Total > Event Waits Time (s) > Ela Time > -------------------------------------------- ------------ ----------- > -------- > log file sequential read 6,254 1,280 > 28.25 > CPU time 1,242 > 27.40 > log buffer space 8,673 999 > 22.03 > db file scattered read 51,788 375 > 8.28 > free buffer waits 744 330 > 7.28 > > > First glance would suggest that the Linux RAC-OCFS database is pretty > much IO bound, but this system has a much higher specification IO > subsystem then the Linux non-RAC system (see below). > > The specification of the systems is roughly: > > Oracle 9.2.0.4. > > Linux non-RAC > RedHat 8.0 on Dual 1.8GHz Xeon (with hyperthreading), 1Gb RAM, 2 Ultra > 320 SCSI disks (not striped or mirrored) > > Linux RAC-OCFS > 2*(RedHat AS2.1 on IBM x360, Quad 1.5GHz Xeon, 7Gb RAM), fibre connected > FastT Disk Array > > > Any comments? > > (By the way, because I don't do full schema imports very often I'm not > excited about import performance in its own right, but I am worried that > the relatively poor performance is symptomatic of a more general issue > that will affect the application itself) > > > Kevin Miller > Oracle Database Administrator > Burns e-Commerce Solutions > > > > > > >> _______________________________________________ > Ocfs-users mailing list > Ocfs-users@oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs-users
Kevin Miller
2003-Nov-24 03:25 UTC
[Ocfs-users] Disappointing IMPORT Performance Using 9i RAC with OCFS on Linux
Wim,> well, you should compare it with rhas21 isntead of rh8I agree.> which kernel is this e.25 ?2.4.9-e.24enterprise> are you doing loads in parallel on 2 nodes or just a single node andyou just have 2 running Single node, no (application) activity on the second node. Thanks for your help. Kevin. -----Original Message----- From: Wim Coekaerts [mailto:wim.coekaerts@oracle.com] Sent: 21 November 2003 19:41 To: Kevin Miller Cc: ocfs-users@oss.oracle.com; ocfs-devel@oss.oracle.com Subject: Re: [Ocfs-users] Disappointing IMPORT Performance Using 9i RAC with OCFS on Linux well, you should compare it with rhas21 isntead of rh8 one thing the rhas21 kernel wasn't good at was io grouping, it submitted really small io's seperately. which kernel is this e.25 ? are you doing loads in parallel on 2 nodes or just a single node and you just have 2 running ? you should download the aio patch and try with aio enabled. lemme dig up the oracle patch you can grab Wim On Fri, Nov 21, 2003 at 05:08:58PM -0000, Kevin Miller wrote:> I am currently involved in setting up a new Oracle environment using9i> RAC with OCFS on Linux. Part of the set-up involves importing the > application schema which comprises 5.5Gb data and 3.7Gb indexes. The > initial results are disappointing: > > Linux non-RAC Linux RAC-OCFS > 60 143 > > The results are elapsed time in minutes to complete the full schema > import (using imp). They are average values from several runs, the > results from each individual run were fairly consistent. Neither the > server nor the database were restarted between runs. > > The same dump file was used for each import. > > The top five wait events for Linux RAC-OCFS database are: > > Top 5 Timed Events > ~~~~~~~~~~~~~~~~~~%> Total > Event Waits Time (s) > Ela Time > -------------------------------------------- ------------ ----------- > -------- > io done 1,457,317 5,550 > 51.84 > CPU time 1,628 > 15.20 > log buffer space 17,957 1,271 > 11.88 > log file sequential read 6,174 666 > 6.22 > log file parallel write 19,632 591 > 5.52 > > > For the Linux non-RAC database: > > Top 5 Timed Events > ~~~~~~~~~~~~~~~~~~%> Total > Event Waits Time (s) > Ela Time > -------------------------------------------- ------------ ----------- > -------- > log file sequential read 6,254 1,280 > 28.25 > CPU time 1,242 > 27.40 > log buffer space 8,673 999 > 22.03 > db file scattered read 51,788 375 > 8.28 > free buffer waits 744 330 > 7.28 > > > First glance would suggest that the Linux RAC-OCFS database is pretty > much IO bound, but this system has a much higher specification IO > subsystem then the Linux non-RAC system (see below). > > The specification of the systems is roughly: > > Oracle 9.2.0.4. > > Linux non-RAC > RedHat 8.0 on Dual 1.8GHz Xeon (with hyperthreading), 1Gb RAM, 2 Ultra > 320 SCSI disks (not striped or mirrored) > > Linux RAC-OCFS > 2*(RedHat AS2.1 on IBM x360, Quad 1.5GHz Xeon, 7Gb RAM), fibreconnected> FastT Disk Array > > > Any comments? > > (By the way, because I don't do full schema imports very often I'm not > excited about import performance in its own right, but I am worriedthat> the relatively poor performance is symptomatic of a more general issue > that will affect the application itself) > > > Kevin Miller > Oracle Database Administrator > Burns e-Commerce Solutions > > > > > > >> _______________________________________________ > Ocfs-users mailing list > Ocfs-users@oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs-users
Kevin Miller
2003-Nov-24 03:32 UTC
[Ocfs-users] Disappointing IMPORT Performance Using 9i RAC with OCFS on Linux
Correction, kernel is 2.4.9-e.27enterprise -----Original Message----- From: Wim Coekaerts [mailto:wim.coekaerts@oracle.com] Sent: 21 November 2003 19:41 To: Kevin Miller Cc: ocfs-users@oss.oracle.com; ocfs-devel@oss.oracle.com Subject: Re: [Ocfs-users] Disappointing IMPORT Performance Using 9i RAC with OCFS on Linux well, you should compare it with rhas21 isntead of rh8 one thing the rhas21 kernel wasn't good at was io grouping, it submitted really small io's seperately. which kernel is this e.25 ? are you doing loads in parallel on 2 nodes or just a single node and you just have 2 running ? you should download the aio patch and try with aio enabled. lemme dig up the oracle patch you can grab Wim On Fri, Nov 21, 2003 at 05:08:58PM -0000, Kevin Miller wrote:> I am currently involved in setting up a new Oracle environment using9i> RAC with OCFS on Linux. Part of the set-up involves importing the > application schema which comprises 5.5Gb data and 3.7Gb indexes. The > initial results are disappointing: > > Linux non-RAC Linux RAC-OCFS > 60 143 > > The results are elapsed time in minutes to complete the full schema > import (using imp). They are average values from several runs, the > results from each individual run were fairly consistent. Neither the > server nor the database were restarted between runs. > > The same dump file was used for each import. > > The top five wait events for Linux RAC-OCFS database are: > > Top 5 Timed Events > ~~~~~~~~~~~~~~~~~~%> Total > Event Waits Time (s) > Ela Time > -------------------------------------------- ------------ ----------- > -------- > io done 1,457,317 5,550 > 51.84 > CPU time 1,628 > 15.20 > log buffer space 17,957 1,271 > 11.88 > log file sequential read 6,174 666 > 6.22 > log file parallel write 19,632 591 > 5.52 > > > For the Linux non-RAC database: > > Top 5 Timed Events > ~~~~~~~~~~~~~~~~~~%> Total > Event Waits Time (s) > Ela Time > -------------------------------------------- ------------ ----------- > -------- > log file sequential read 6,254 1,280 > 28.25 > CPU time 1,242 > 27.40 > log buffer space 8,673 999 > 22.03 > db file scattered read 51,788 375 > 8.28 > free buffer waits 744 330 > 7.28 > > > First glance would suggest that the Linux RAC-OCFS database is pretty > much IO bound, but this system has a much higher specification IO > subsystem then the Linux non-RAC system (see below). > > The specification of the systems is roughly: > > Oracle 9.2.0.4. > > Linux non-RAC > RedHat 8.0 on Dual 1.8GHz Xeon (with hyperthreading), 1Gb RAM, 2 Ultra > 320 SCSI disks (not striped or mirrored) > > Linux RAC-OCFS > 2*(RedHat AS2.1 on IBM x360, Quad 1.5GHz Xeon, 7Gb RAM), fibreconnected> FastT Disk Array > > > Any comments? > > (By the way, because I don't do full schema imports very often I'm not > excited about import performance in its own right, but I am worriedthat> the relatively poor performance is symptomatic of a more general issue > that will affect the application itself) > > > Kevin Miller > Oracle Database Administrator > Burns e-Commerce Solutions > > > > > > >> _______________________________________________ > Ocfs-users mailing list > Ocfs-users@oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs-users
Kevin Miller
2003-Nov-26 05:17 UTC
[Ocfs-users] Disappointing IMPORT Performance Using 9i RAC with OCFS on Linux
Wim,
I have some further surprising results.
I wanted to isolate the write-intensive performance problem so I wrote a
bit of SQL which Cartesian-joins a couple of small-ish tables to create
a 1.2Gb table. I did this NOLOGGING to eliminate redo generation from
the test:
EXT3 RAW OCFS
101 187 508
The results are elapsed time in seconds to create the table, averaged
over several runs (apart from the ext3 test which was a single run).
The RAW/OCFS tests were run within the same database; the RAW test used
a tablespace created on a RAW partition, the OCFS test used a
tablespace created on an OCFS partition.
The EXT3 test was actually performed within the RMAN repository which is
non-RAC and runs on one node of the cluster.
Ok, these tests are not very scientific: the RAW partition is on a
different set of disks to the OCFS partition, the OCFS partition is on
the same set of disks as other bits of the database. But something just
doesn't feel right. And I really want to get this working. Its always
possible of course that the guys who set up the disk array/filesystems
have got it wrong and its nothing to do with the DB/OCFS layers.
Kevin
-----Original Message-----
From: Wim Coekaerts [mailto:wim.coekaerts@oracle.com]
Sent: 21 November 2003 19:41
To: Kevin Miller
Cc: ocfs-users@oss.oracle.com; ocfs-devel@oss.oracle.com
Subject: Re: [Ocfs-users] Disappointing IMPORT Performance Using 9i RAC
with OCFS on Linux
well, you should compare it with rhas21 isntead of rh8
one thing the rhas21 kernel wasn't good at was io grouping, it submitted
really small io's seperately.
which kernel is this e.25 ?
are you doing loads in parallel on 2 nodes or just a single node and you
just have 2 running ?
you should download the aio patch and try with aio enabled.
lemme dig up the oracle patch you can grab
Wim
On Fri, Nov 21, 2003 at 05:08:58PM -0000, Kevin Miller
wrote:> I am currently involved in setting up a new Oracle environment using
9i> RAC with OCFS on Linux. Part of the set-up involves importing the
> application schema which comprises 5.5Gb data and 3.7Gb indexes. The
> initial results are disappointing:
>
> Linux non-RAC Linux RAC-OCFS
> 60 143
>
> The results are elapsed time in minutes to complete the full schema
> import (using imp). They are average values from several runs, the
> results from each individual run were fairly consistent. Neither the
> server nor the database were restarted between runs.
>
> The same dump file was used for each import.
>
> The top five wait events for Linux RAC-OCFS database are:
>
> Top 5 Timed Events
> ~~~~~~~~~~~~~~~~~~
%> Total
> Event Waits Time (s)
> Ela Time
> -------------------------------------------- ------------ -----------
> --------
> io done 1,457,317 5,550
> 51.84
> CPU time 1,628
> 15.20
> log buffer space 17,957 1,271
> 11.88
> log file sequential read 6,174 666
> 6.22
> log file parallel write 19,632 591
> 5.52
>
>
> For the Linux non-RAC database:
>
> Top 5 Timed Events
> ~~~~~~~~~~~~~~~~~~
%> Total
> Event Waits Time (s)
> Ela Time
> -------------------------------------------- ------------ -----------
> --------
> log file sequential read 6,254 1,280
> 28.25
> CPU time 1,242
> 27.40
> log buffer space 8,673 999
> 22.03
> db file scattered read 51,788 375
> 8.28
> free buffer waits 744 330
> 7.28
>
>
> First glance would suggest that the Linux RAC-OCFS database is pretty
> much IO bound, but this system has a much higher specification IO
> subsystem then the Linux non-RAC system (see below).
>
> The specification of the systems is roughly:
>
> Oracle 9.2.0.4.
>
> Linux non-RAC
> RedHat 8.0 on Dual 1.8GHz Xeon (with hyperthreading), 1Gb RAM, 2 Ultra
> 320 SCSI disks (not striped or mirrored)
>
> Linux RAC-OCFS
> 2*(RedHat AS2.1 on IBM x360, Quad 1.5GHz Xeon, 7Gb RAM), fibre
connected> FastT Disk Array
>
>
> Any comments?
>
> (By the way, because I don't do full schema imports very often I'm
not
> excited about import performance in its own right, but I am worried
that> the relatively poor performance is symptomatic of a more general issue
> that will affect the application itself)
>
>
> Kevin Miller
> Oracle Database Administrator
> Burns e-Commerce Solutions
>
>
>
>
>
>
>
> _______________________________________________
> Ocfs-users mailing list
> Ocfs-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs-users
Kevin Miller
2003-Nov-26 09:05 UTC
[Ocfs-users] Disappointing IMPORT Performance Using 9i RAC with OCFS on Linux
Wim,
This may be a false alarm.
I have re-written the test so that there are virtually no READs in order
to build the large table, which was impacting the OCFS test more than
the RAW test. So the tests are very close to measuring the pure WRITE
performance by Oracle into the two tablespace types.
The results are now similar between RAW and OCFS.
Regards,
Kevin
-----Original Message-----
From: Kevin Miller
Sent: 26 November 2003 11:19
To: 'Wim Coekaerts'
Cc: ocfs-users@oss.oracle.com; ocfs-devel@oss.oracle.com
Subject: RE: [Ocfs-users] Disappointing IMPORT Performance Using 9i RAC
with OCFS on Linux
Wim,
I have some further surprising results.
I wanted to isolate the write-intensive performance problem so I wrote a
bit of SQL which Cartesian-joins a couple of small-ish tables to create
a 1.2Gb table. I did this NOLOGGING to eliminate redo generation from
the test:
EXT3 RAW OCFS
101 187 508
The results are elapsed time in seconds to create the table, averaged
over several runs (apart from the ext3 test which was a single run).
The RAW/OCFS tests were run within the same database; the RAW test used
a tablespace created on a RAW partition, the OCFS test used a
tablespace created on an OCFS partition.
The EXT3 test was actually performed within the RMAN repository which is
non-RAC and runs on one node of the cluster.
Ok, these tests are not very scientific: the RAW partition is on a
different set of disks to the OCFS partition, the OCFS partition is on
the same set of disks as other bits of the database. But something just
doesn't feel right. And I really want to get this working. Its always
possible of course that the guys who set up the disk array/filesystems
have got it wrong and its nothing to do with the DB/OCFS layers.
Kevin
-----Original Message-----
From: Wim Coekaerts [mailto:wim.coekaerts@oracle.com]
Sent: 21 November 2003 19:41
To: Kevin Miller
Cc: ocfs-users@oss.oracle.com; ocfs-devel@oss.oracle.com
Subject: Re: [Ocfs-users] Disappointing IMPORT Performance Using 9i RAC
with OCFS on Linux
well, you should compare it with rhas21 isntead of rh8
one thing the rhas21 kernel wasn't good at was io grouping, it submitted
really small io's seperately.
which kernel is this e.25 ?
are you doing loads in parallel on 2 nodes or just a single node and you
just have 2 running ?
you should download the aio patch and try with aio enabled.
lemme dig up the oracle patch you can grab
Wim
On Fri, Nov 21, 2003 at 05:08:58PM -0000, Kevin Miller
wrote:> I am currently involved in setting up a new Oracle environment using
9i> RAC with OCFS on Linux. Part of the set-up involves importing the
> application schema which comprises 5.5Gb data and 3.7Gb indexes. The
> initial results are disappointing:
>
> Linux non-RAC Linux RAC-OCFS
> 60 143
>
> The results are elapsed time in minutes to complete the full schema
> import (using imp). They are average values from several runs, the
> results from each individual run were fairly consistent. Neither the
> server nor the database were restarted between runs.
>
> The same dump file was used for each import.
>
> The top five wait events for Linux RAC-OCFS database are:
>
> Top 5 Timed Events
> ~~~~~~~~~~~~~~~~~~
%> Total
> Event Waits Time (s)
> Ela Time
> -------------------------------------------- ------------ -----------
> --------
> io done 1,457,317 5,550
> 51.84
> CPU time 1,628
> 15.20
> log buffer space 17,957 1,271
> 11.88
> log file sequential read 6,174 666
> 6.22
> log file parallel write 19,632 591
> 5.52
>
>
> For the Linux non-RAC database:
>
> Top 5 Timed Events
> ~~~~~~~~~~~~~~~~~~
%> Total
> Event Waits Time (s)
> Ela Time
> -------------------------------------------- ------------ -----------
> --------
> log file sequential read 6,254 1,280
> 28.25
> CPU time 1,242
> 27.40
> log buffer space 8,673 999
> 22.03
> db file scattered read 51,788 375
> 8.28
> free buffer waits 744 330
> 7.28
>
>
> First glance would suggest that the Linux RAC-OCFS database is pretty
> much IO bound, but this system has a much higher specification IO
> subsystem then the Linux non-RAC system (see below).
>
> The specification of the systems is roughly:
>
> Oracle 9.2.0.4.
>
> Linux non-RAC
> RedHat 8.0 on Dual 1.8GHz Xeon (with hyperthreading), 1Gb RAM, 2 Ultra
> 320 SCSI disks (not striped or mirrored)
>
> Linux RAC-OCFS
> 2*(RedHat AS2.1 on IBM x360, Quad 1.5GHz Xeon, 7Gb RAM), fibre
connected> FastT Disk Array
>
>
> Any comments?
>
> (By the way, because I don't do full schema imports very often I'm
not
> excited about import performance in its own right, but I am worried
that> the relatively poor performance is symptomatic of a more general issue
> that will affect the application itself)
>
>
> Kevin Miller
> Oracle Database Administrator
> Burns e-Commerce Solutions
>
>
>
>
>
>
>
> _______________________________________________
> Ocfs-users mailing list
> Ocfs-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs-users