I have two identical servers. The only difference is that the first
one has Maxtor 250G drives and the second one has Seagate 320G drives.
OS: CentOS-4.4 (fully patched)
CPU: dual Opteron 280
Memory: 16GB
Raid card: 3ware 9550Sx-8LP
Raid volume: 4-disk Raid 5 with NCQ and Write Cache enabled
On the first server I have decent performance. Nothing spectacular,
but good enough. The second one has about 1/3 the write speed. I
can't find any difference between the systems. Both of them have the
same stripe size, both have ext3 filesystems, both have write caching
and NCQ turned on. I have already increased the read ahead setting to
16384 on both servers.
I ran the tests like this:
# sync; bonnie++ -d /iotest -s 50g -n 0 -b -f
(I have removed some extra information from the reports for brevity)
And here are the results for the two servers:
------Output------- --Input--
--Block-- -Rewrite- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP /sec %CP
First 50G 62893 25 46763 12 160672 19 120.6 1
Second 50G 18835 7 44025 12 194719 24 122.8 1
As you can see, the write performance of the second server is
terrible. Anyone have any suggestions of what I can look for? I keep
thinking there must be something I tweaked on the first server that I
forgot about for the second one, but so far I haven't been able to
find it.
Any suggestions appreciated!
--
Bowie
Have both controller the same firmware? -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 4285 bytes Desc: S/MIME Cryptographic Signature URL: <http://lists.centos.org/pipermail/centos/attachments/20061012/98742221/attachment-0002.bin>
On Thu, 12 Oct 2006 at 4:22pm, Bowie Bailey wrote> I have two identical servers. The only difference is that the first > one has Maxtor 250G drives and the second one has Seagate 320G drives. > > OS: CentOS-4.4 (fully patched) > CPU: dual Opteron 280 > Memory: 16GB > Raid card: 3ware 9550Sx-8LP > Raid volume: 4-disk Raid 5 with NCQ and Write Cache enabled > > On the first server I have decent performance. Nothing spectacular, > but good enough. The second one has about 1/3 the write speed. I > can't find any difference between the systems. Both of them have the > same stripe size, both have ext3 filesystems, both have write caching > and NCQ turned on. I have already increased the read ahead setting to > 16384 on both servers.Turn off NCQ. Last I knew, this was still 3ware's recommendation. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Frank B?ttner wrote:> Have both controller the same firmware?Yes. They both have FE9X 3.04.00.005. -- Bowie
Joshua Baker-LePain wrote:> On Thu, 12 Oct 2006 at 4:22pm, Bowie Bailey wrote > > > I have two identical servers. The only difference is that the first > > one has Maxtor 250G drives and the second one has Seagate 320G > > drives. > > > > OS: CentOS-4.4 (fully patched) > > CPU: dual Opteron 280 > > Memory: 16GB > > Raid card: 3ware 9550Sx-8LP > > Raid volume: 4-disk Raid 5 with NCQ and Write Cache enabled > > > > On the first server I have decent performance. Nothing spectacular, > > but good enough. The second one has about 1/3 the write speed. I > > can't find any difference between the systems. Both of them have > > the same stripe size, both have ext3 filesystems, both have write > > caching and NCQ turned on. I have already increased the read ahead > > setting to 16384 on both servers. > > Turn off NCQ. Last I knew, this was still 3ware's recommendation.I can try that, but when I tested on the first server, I found that disabling NCQ increased the read performance by 10M/s and decreased the write performance by 10M/s. Maybe I'll get different results from the Seagate drives. -- Bowie
chrism at imntv.com wrote:> Bowie Bailey wrote: > > Joshua Baker-LePain wrote: > > > On Thu, 12 Oct 2006 at 4:22pm, Bowie Bailey wrote > > > > > > > I have two identical servers. The only difference is that the > > > > first one has Maxtor 250G drives and the second one has Seagate > > > > 320G drives. > > > > > > > > OS: CentOS-4.4 (fully patched) > > > > CPU: dual Opteron 280 > > > > Memory: 16GB > > > > Raid card: 3ware 9550Sx-8LP > > > > Raid volume: 4-disk Raid 5 with NCQ and Write Cache enabled > > > > > > > > On the first server I have decent performance. Nothing > > > > spectacular, but good enough. The second one has about 1/3 the > > > > write speed. I can't find any difference between the systems. > > > > Both of them have the same stripe size, both have ext3 > > > > filesystems, both have write caching and NCQ turned on. I have > > > > already increased the read ahead setting to 16384 on both > > > > servers. > > > Turn off NCQ. Last I knew, this was still 3ware's recommendation. > > > > I can try that, but when I tested on the first server, I found that > > disabling NCQ increased the read performance by 10M/s and decreased > > the write performance by 10M/s. Maybe I'll get different results > > from the Seagate drives. > > Here's what I'm getting from a dual Opteron 275 box, Centos 4.4-64bit, > 2gb RAM and an 8 drive array of 750gig Seagates: > > Version 1.03 ------Sequential Output------ --Sequential Input- > --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP > /sec %CP > localhost.locald 4G 248901 96 138957 40 308874 > 42 625.8 > > I've been applying all the various tricks that came up during the last > couple of threads. Turning NCQ off definitely made a big difference > for me.With NCQ off, I get these results: Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP Second 50G 20853 8 41899 11 192713 23 116.9 1 Write performance increased by 2M/s and read performance decreased by 2M/s. What tweaks have you applied so far? I would expect a 4-drive array to be slower than an 8-drive array, but not this much! -- Bowie
I saw a definite improvement by turning off NCQ and setting StorSave to 'Balanced.' Are these 1.5GB/Sec or 3.0GB/Sec SATA drives? During my testing I changed from non-interleaved memory and 1.5GB to interleaved and 3.0GB. Made a big difference in bonnie++ results. Unfortunately, I can't say which was more important. If you have the patience, read through my recent (but lengthy) thread on the 3Ware 9550 titled "Calling All FS Fanatics." There's a lot of good info from many helpful people. I've only gotten full performance using JFS or XFS. Kirk Bocek Bowie Bailey wrote:> I have two identical servers. The only difference is that the first > one has Maxtor 250G drives and the second one has Seagate 320G drives. > > OS: CentOS-4.4 (fully patched) > CPU: dual Opteron 280 > Memory: 16GB > Raid card: 3ware 9550Sx-8LP > Raid volume: 4-disk Raid 5 with NCQ and Write Cache enabled > > On the first server I have decent performance. Nothing spectacular, > but good enough. The second one has about 1/3 the write speed. I > can't find any difference between the systems. Both of them have the > same stripe size, both have ext3 filesystems, both have write caching > and NCQ turned on. I have already increased the read ahead setting to > 16384 on both servers. > > I ran the tests like this: > > # sync; bonnie++ -d /iotest -s 50g -n 0 -b -f > (I have removed some extra information from the reports for brevity) > > And here are the results for the two servers: > > ------Output------- --Input-- > --Block-- -Rewrite- --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP /sec %CP > First 50G 62893 25 46763 12 160672 19 120.6 1 > Second 50G 18835 7 44025 12 194719 24 122.8 1 > > As you can see, the write performance of the second server is > terrible. Anyone have any suggestions of what I can look for? I keep > thinking there must be something I tweaked on the first server that I > forgot about for the second one, but so far I haven't been able to > find it. > > Any suggestions appreciated! >
What's the motherboard or manufacturer and model? Bowie Bailey wrote:> OS: CentOS-4.4 (fully patched) > CPU: dual Opteron 280 > Memory: 16GB > Raid card: 3ware 9550Sx-8LP > Raid volume: 4-disk Raid 5 with NCQ and Write Cache enabled
Non-interleaved memory sounds interesting. Where do I change that
setting?
I don't want to change the StorSave setting. I'm not really looking
for maximum performance here, just reasonable. If I can get the same
60 M/s speed that I got from the first server, I'll be happy.
The drives are all running at 3.0GB/Sec according to the drive
information in 3DM.
I have read the entire "Calling All FS Fanatics" thread as it passed
through the list. It was quite interesting, but the main things I got
from it are:
- Increase the read ahead settings
- XFS, JFS, and ReiserFS are faster, but less fault-tolerant, than
ext3
I have already changed the read ahead on both the physical volume and
the logical volume. This gives me increased read performance, but has
no effect on writes. I am sticking with ext3 since I am much more
concerned about stability than performance.
Bowie
Kirk Bocek wrote:> I saw a definite improvement by turning off NCQ and setting StorSave
> to 'Balanced.' Are these 1.5GB/Sec or 3.0GB/Sec SATA drives? During
> my testing I changed from non-interleaved memory and 1.5GB to
> interleaved and 3.0GB. Made a big difference in bonnie++ results.
> Unfortunately, I can't say which was more important.
>
> If you have the patience, read through my recent (but lengthy) thread
> on the 3Ware 9550 titled "Calling All FS Fanatics." There's a
lot of
> good info from many helpful people. I've only gotten full performance
> using JFS or XFS.
>
> Kirk Bocek
>
> Bowie Bailey wrote:
> > I have two identical servers. The only difference is that the first
> > one has Maxtor 250G drives and the second one has Seagate 320G
> > drives.
> >
> > OS: CentOS-4.4 (fully patched)
> > CPU: dual Opteron 280
> > Memory: 16GB
> > Raid card: 3ware 9550Sx-8LP
> > Raid volume: 4-disk Raid 5 with NCQ and Write Cache enabled
> >
> > On the first server I have decent performance. Nothing spectacular,
> > but good enough. The second one has about 1/3 the write speed. I
> > can't find any difference between the systems. Both of them have
> > the same stripe size, both have ext3 filesystems, both have write
> > caching and NCQ turned on. I have already increased the read ahead
> > setting to 16384 on both servers.
> >
> > I ran the tests like this:
> >
> > # sync; bonnie++ -d /iotest -s 50g -n 0 -b -f
> > (I have removed some extra information from the reports for brevity)
> >
> > And here are the results for the two servers:
> >
> > ------Output------- --Input--
> > --Block-- -Rewrite- --Block-- --Seeks--
> > Machine Size K/sec %CP K/sec %CP K/sec %CP /sec %CP
> > First 50G 62893 25 46763 12 160672 19 120.6 1
> > Second 50G 18835 7 44025 12 194719 24 122.8 1
> >
> > As you can see, the write performance of the second server is
> > terrible. Anyone have any suggestions of what I can look for? I
> > keep thinking there must be something I tweaked on the first server
> > that I forgot about for the second one, but so far I haven't been
> > able to find it.
> >
> > Any suggestions appreciated!
chrism at imntv.com wrote:> Kirk Bocek wrote: > > I saw a definite improvement by turning off NCQ and setting StorSave > > to 'Balanced.' Are these 1.5GB/Sec or 3.0GB/Sec SATA drives? During > > my testing I changed from non-interleaved memory and 1.5GB to > > interleaved and 3.0GB. Made a big difference in bonnie++ results. > > Unfortunately, I can't say which was more important. > > > > If you have the patience, read through my recent (but lengthy) > > thread on the 3Ware 9550 titled "Calling All FS Fanatics." There's > > a lot of good info from many helpful people. I've only gotten full > > performance using JFS or XFS. > > > > I'm doing this from memory as the machine is at another location now. > I think I did this: > > turned off ncq (per Josh's suggestion)Tried that, it made a small improvement (see my previous post).> turned on write caching (it's on an oversized ups and the data isn't > critical)Write caching is on. I've got a BBU, so the data is protected.> set storsave to "performance"I prefer to leave this one as is to take advantage of the write journaling.> changed the memory interleave (thanks to kirk's suggestion). It was > off by default.This one I would like to try. Can you tell me where I can make the change?> used parted to create gpt disklabelWhat is the advantage of gpt over msdos? Can I make this change without recreating the filesystem?> set noatime and one other option that was suggested here for the RAID > partitionAnyone know if noatime is recommended for a filesystem containing an Oracle database? What was the other option?> used mke2fs -j -b 4096 /dev/blahI used 'mkfs.ext3 /dev/blah'. It created 4K blocks.> That was it. > > Also, I tried the same array as a RAID0 device on the same box and the > performance was approximately the same (maybe ever so slightly > faster).Raid0 should be faster, particularly with writes since it doesn't have to calculate parity. -- Bowie
Kirk Bocek wrote:> chrism at imntv.com wrote: > > I'm doing this from memory as the machine is at another location > > now. I think I did this: > > > > turned off ncq (per Josh's suggestion) > > turned on write caching (it's on an oversized ups and the data > > isn't critical) set storsave to "performance" > > changed the memory interleave (thanks to kirk's suggestion). It > > was off by default. used parted to create gpt disklabel > > set noatime and one other option that was suggested here for the > > RAID partition used mke2fs -j -b 4096 /dev/blah > > > > That was it. > > Okay, now try installing kernel-module-xfs and xfsutils (located in > the centosplus repository), run mkfs.xfs on some unused space, mount > and re-run bonnie++.I'm sure that would be faster, but I'm sticking to ext3 for now. -- Bowie
Kirk Bocek wrote:> What's the motherboard or manufacturer and model?Knew I was forgetting something... :) OS: CentOS-4.4 (fully patched) Motherboard: Tyan Thunder K8HM CPU: dual Opteron 280 Memory: 16GB Raid card: 3ware 9550SX-8LP Raid volume: 4-disk Raid 5 with NCQ and Write Cache enabled -- Bowie
Kirk Bocek wrote:> chrism at imntv.com wrote: > > To what end? I'd absolutely NEVER use it in production that way. > > That's *plenty* fast for me. > > On the current system I built, the best I could get from ext3 on a > 4-drive raid-5 array was 95MB/Sec writes. With XFS I'm getting > 220MB/Sec+. For a media server, I want the extra speed.For me, ext3 on a 4-drive raid5 array is 20MB/Sec. I've GOT to be doing something wrong. -- Bowie
Joshua Baker-LePain wrote:> On Thu, 12 Oct 2006 at 7:58pm, Bowie Bailey wrote > > > With NCQ off, I get these results: > > > > Version 1.03 ------Sequential Output------ --Sequential > > Input- > > --Random- > > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > > --Seeks-- > > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec > > %CP /sec %CP Second 50G 20853 8 41899 11 > > 192713 23 116.9 1 > > > > Write performance increased by 2M/s and read performance decreased > > by 2M/s. > > > > What tweaks have you applied so far? I would expect a 4-drive array > > to be slower than an 8-drive array, but not this much! > > What's your StorSave profile set to?StorSave is set to Protection. Data integrity is very important on this system, so I need to leave it as is. Besides, I don't want to disable the data journaling after paying $100 for a BBU to protect it. :) -- Bowie
Kirk Bocek wrote:> Responding to several of your posts here. I understand your need for > stability, but can't you set StorSave down and see what it does to > your speed?I could, but I don't see the point. I have one machine with StorSave set at "Protect" which writes at 60MB/Sec. The other identical system which is also set to "Protect" only writes at 20MB/Sec. Therefore, there must be something else at work. As we speak, I have swapped the hard drives in these two systems and I am running another test. This will at least tell me if the problem is based in the hard drives & OS or elsewhere in hardware.> Some of my 95MB/Sec was achieved with some of the suggested ext3 > journal settings. Look back through my previous thread. > > Memory interleaving on my system (Supermicro X7DVL-E) is turned on > automatically when you install two pairs of memory sticks. With 16GB > on your system, I *assume* you have at least 4 sticks. Check the > manual and see if there is a BIOS setting.I found the interleaving settings in the BIOS. Both systems currently have it turned off, so I haven't touched it at this point. Once I have determined why the performance is so different between these two machines, I may experiment with that setting to see if it helps. Right now, I am not looking for maximum performance. I am just trying to determine why there is a 3x difference in write performance between two identical machines. -- Bowie
Bowie Bailey spake the following on 10/12/2006 1:22 PM:> I have two identical servers. The only difference is that the first > one has Maxtor 250G drives and the second one has Seagate 320G drives. > > OS: CentOS-4.4 (fully patched) > CPU: dual Opteron 280 > Memory: 16GB > Raid card: 3ware 9550Sx-8LP > Raid volume: 4-disk Raid 5 with NCQ and Write Cache enabled > > On the first server I have decent performance. Nothing spectacular, > but good enough. The second one has about 1/3 the write speed. I > can't find any difference between the systems. Both of them have the > same stripe size, both have ext3 filesystems, both have write caching > and NCQ turned on. I have already increased the read ahead setting to > 16384 on both servers. > > I ran the tests like this: > > # sync; bonnie++ -d /iotest -s 50g -n 0 -b -f > (I have removed some extra information from the reports for brevity) > > And here are the results for the two servers: > > ------Output------- --Input-- > --Block-- -Rewrite- --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP /sec %CP > First 50G 62893 25 46763 12 160672 19 120.6 1 > Second 50G 18835 7 44025 12 194719 24 122.8 1 > > As you can see, the write performance of the second server is > terrible. Anyone have any suggestions of what I can look for? I keep > thinking there must be something I tweaked on the first server that I > forgot about for the second one, but so far I haven't been able to > find it. > > Any suggestions appreciated! >Is there a difference in the write performance, speed, or cache size of the drives? You didn't list specifics of the drives, so I had to ask. -- MailScanner is like deodorant... You hope everybody uses it, and you notice quickly if they don't!!!!
William L. Maltby wrote:> On Fri, 2006-10-13 at 11:44 -0400, Bowie Bailey wrote: > > Kirk Bocek wrote: > > > <snip> > > > Right now, I am not looking for maximum performance. I am just > > trying to determine why there is a 3x difference in write > > performance between two identical machines. > > 60 vs. 40 conductor cables on UDMA? have you wiggled/reseated them?These are SATA drives, but I have checked the cable connections.> PS sizes similar and simailar load?Identical PS. Load should be similar. Tests have been done at runlevel 5 with nobody using the system.> /var/log/messages BogoMips the same?I'm not familiar with this.> Mem config same?Yep. They've both got 16GB of RAM (6GB allocated to Huge Pages and about 9GB free).> Same "load". Does top in a "long" screen show any diference in number > of loads or total load being imposed? Keep an eye out for excessive > wait channels in top (have to change defaults of fields displayed and > order) or in ps?I haven't seen that. I'll keep an eye on it next time.> Are you using the sar reports, which can be helpful in the case of > large unknowns.I haven't used the sar reports. Sar tends to be a bit cryptic. If you can give me a useful command line, I'll check it out. Thanks! -- Bowie
Scott Silva wrote:> Bowie Bailey spake the following on 10/12/2006 1:22 PM: > > I have two identical servers. The only difference is that the first > > one has Maxtor 250G drives and the second one has Seagate 320G > > drives. > > > > OS: CentOS-4.4 (fully patched) > > CPU: dual Opteron 280 > > Memory: 16GB > > Raid card: 3ware 9550Sx-8LP > > Raid volume: 4-disk Raid 5 with NCQ and Write Cache enabled > > > > On the first server I have decent performance. Nothing spectacular, > > but good enough. The second one has about 1/3 the write speed. I > > can't find any difference between the systems. Both of them have > > the same stripe size, both have ext3 filesystems, both have write > > caching and NCQ turned on. I have already increased the read ahead > > setting to 16384 on both servers. > > > > I ran the tests like this: > > > > # sync; bonnie++ -d /iotest -s 50g -n 0 -b -f > > (I have removed some extra information from the reports for brevity) > > > > And here are the results for the two servers: > > > > ------Output------- --Input-- > > --Block-- -Rewrite- --Block-- --Seeks-- > > Machine Size K/sec %CP K/sec %CP K/sec %CP /sec %CP > > First 50G 62893 25 46763 12 160672 19 120.6 1 > > Second 50G 18835 7 44025 12 194719 24 122.8 1 > > > > As you can see, the write performance of the second server is > > terrible. Anyone have any suggestions of what I can look for? I > > keep thinking there must be something I tweaked on the first server > > that I forgot about for the second one, but so far I haven't been > > able to find it. > > > > Any suggestions appreciated! > > > Is there a difference in the write performance, speed, or cache size > of the drives? > You didn't list specifics of the drives, so I had to ask.There may be minor differences, but I would not expect this much difference. Actually, further testing has shown that the write performance on the second machine varies anywhere from 20MB/Sec to 54MB/Sec depending on when I test it. Unfortunately, it appears random from my point of view. I'm still running tests to try to find a cause or at least a related symptom. The First system has Maxtor 7V250F0 drives (250GB). The second system has Seagate ST3500641AS drives (320GB). They are both 7200RPM SATA II drives with 16MB cache and NCQ support. -- Bowie
DamianS wrote:> > The First system has Maxtor 7V250F0 drives (250GB). The second > > system has Seagate ST3500641AS drives (320GB). They are both > > 7200RPM SATA II > > drives with 16MB cache and NCQ support. > > > > Have you eliminated the hard drives as being responsible, by testing > the Seagates on first system, and Maxtors on second system?Here is where things currently stand. I have taken the two arrays and swapped them between the two systems. The Seagates are still slower. However, since the OS is installed on these arrays, I don't know if this is due to the drives or some OS setting. I have also taken one spare drive of each type and tested each one as a single-drive unit in both systems. These tests all give similar numbers and show that the Seagate drives are marginally faster than the Maxtors. -- Bowie