I know there are a few 3Ware fans here and I was hoping to find some help. I
just built a new server using a 3Ware 9550SX-4LP with four disks in raid 5.
The array is fully initialized but I'm not getting the write performance I
was
hoping for -- only 40 to 45MB/Sec.
3Ware's site advertises 300MB/Sec writes using 8 disks on the PCI Express
version of this card (the 9580 I think.) I'm guessing 4x PCI Express has
more
bandwidth but 133Mhz 64-bit PCI-X is no slouch. The only two things I need to
change are to put the disks in 3GB/Sec SATA mode (they're in 1.5GB/Sec mode
right now) and to enable memory interleaving on the motherboard (I only have
two sticks of memory, this M/B needs 4 for interleaving.) But I don't think
these things should drop me down to the write performance of a single IDE disk.
Does anyone have any suggestions on improving write performance?
The Details (I know the devil is in here somewhere):
Supermicro X7DVL-E Motherboard
Dual Xeon 5130 (2 Ghz)
2GB RAM
3Ware 9550SX-4LP in a 133Mhz 64-bit slot.
4x Seagate ST3400620AS 400GB SATA Drives
NCQ Enabled but currently in 1.5GB/Sec mode
CentOS 4.4, Kernel 2.6.9-42.0.2.ELsmp
Stock Kernel 3Ware Driver 2.26.04.010
Bonnie++ 1.03a 50GB Benchmark run four times on an ext3 filesystem:
------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
Beryl 50G:64k 44572 77 53084 14 40624 7 47790 72 162504 12 96.5 0
Beryl 50G:64k 45960 79 53068 14 39972 7 47701 73 162510 12 96.8 0
Beryl 50G:64k 36846 64 37579 10 36785 6 48090 72 162609 12 96.9 0
Beryl 50G:64k 45205 77 53621 15 40757 7 48325 73 162331 12 96.4 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
Beryl 16 3928 99 +++++ +++ +++++ +++ 3961 98 +++++ +++ 12442 99
Beryl 16 3929 98 +++++ +++ +++++ +++ 3957 99 +++++ +++ 12373 100
Beryl 16 3718 94 +++++ +++ +++++ +++ 3389 84 +++++ +++ 12307 99
Beryl 16 3920 98 +++++ +++ +++++ +++ 3969 99 +++++ +++ 12421 99
On Sat, 30 Sep 2006 at 8:56pm, Kirk Bocek wrote> 3Ware's site advertises 300MB/Sec writes using 8 disks on the PCI Express > version of this card (the 9580 I think.) I'm guessing 4x PCI Express has more > bandwidth but 133Mhz 64-bit PCI-X is no slouch. The only two things I need to > change are to put the disks in 3GB/Sec SATA mode (they're in 1.5GB/Sec mode > right now) and to enable memory interleaving on the motherboard (I only have > two sticks of memory, this M/B needs 4 for interleaving.) But I don't think > these things should drop me down to the write performance of a single IDE > disk. > > Does anyone have any suggestions on improving write performance?1) Make sure the firmware on the board is up to date. 2) Turn off NCQ (last I knew, this was 3ware's advice). 3) Try putting the ext3 journal on another device. 4) Use XFS. The last one is not just a prejudice. There really seems to be some sort of odd (and rather bad) interaction between 3ware RAID5 and ext3. Going to XFS really helps (especially with writes). Oh, and be sure to 'blockdev --setra 16384' that device, which will help with reads. 16384 is what works for me -- YMMV, of course. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
> I know there are a few 3Ware fans here and I was hoping to find some help. > I just built a new server using a 3Ware 9550SX-4LP with four disks in raid > 5. The array is fully initialized but I'm not getting the write performance > I was hoping for -- only 40 to 45MB/Sec.I've got similar trouble on old Xeon MBs with 5 PCI-X slot. Try to use the other slot and clear PCI information on BIOS.> 3Ware's site advertises 300MB/Sec writes using 8 disks on the PCI Express > version of this card (the 9580 I think.) I'm guessing 4x PCI Express has > more bandwidth but 133Mhz 64-bit PCI-X is no slouch. The only two things I > need to change are to put the disks in 3GB/Sec SATA mode (they're in > 1.5GB/Sec mode right now) and to enable memory interleaving on the > motherboard (I only have two sticks of memory, this M/B needs 4 for > interleaving.) But I don't think these things should drop me down to the > write performance of a single IDE disk. > > Does anyone have any suggestions on improving write performance? > > The Details (I know the devil is in here somewhere): > > Supermicro X7DVL-E Motherboard > Dual Xeon 5130 (2 Ghz) > 2GB RAM > 3Ware 9550SX-4LP in a 133Mhz 64-bit slot. > 4x Seagate ST3400620AS 400GB SATA Drives > NCQ Enabled but currently in 1.5GB/Sec mode > CentOS 4.4, Kernel 2.6.9-42.0.2.ELsmp > Stock Kernel 3Ware Driver 2.26.04.010 > > Bonnie++ 1.03a 50GB Benchmark run four times on an ext3 filesystem: > > ------Sequential Output------ --Sequential Input- > --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec > %CP Beryl 50G:64k 44572 77 53084 14 40624 7 47790 72 162504 12 > 96.5 0 Beryl 50G:64k 45960 79 53068 14 39972 7 47701 73 162510 > 12 96.8 0 Beryl 50G:64k 36846 64 37579 10 36785 6 48090 72 > 162609 12 96.9 0 Beryl 50G:64k 45205 77 53621 15 40757 7 48325 > 73 162331 12 96.4 0 ------Sequential Create------ --------Random > Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- > files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > Beryl 16 3928 99 +++++ +++ +++++ +++ 3961 98 +++++ +++ 12442 99 > Beryl 16 3929 98 +++++ +++ +++++ +++ 3957 99 +++++ +++ 12373 100 > Beryl 16 3718 94 +++++ +++ +++++ +++ 3389 84 +++++ +++ 12307 99 > Beryl 16 3920 98 +++++ +++ +++++ +++ 3969 99 +++++ +++ 12421 99 > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos
On Sat, 2006-09-30 at 20:56 -0700, Kirk Bocek wrote:> I know there are a few 3Ware fans here and I was hoping to find some help. I > just built a new server using a 3Ware 9550SX-4LP with four disks in raid 5. > The array is fully initialized but I'm not getting the write performance I was > hoping for -- only 40 to 45MB/Sec. >If you are satisfied with the data integrity guarantees of XFS, which are weaker than what EXT3 gives you by default, you might want to mount your EXT3 filesystem with the data=writeback option. This turns off the extra guarantees that ext3 gives you. And if you do use XFS, make absolutely certain you can trust your UPS and that the system is configured to do a normal shutdown before the UPS runs out. I've heard too many horror stories about XFS filesystems destroyed by a power blip. Yes, I'm aware that it says in their FAQ that this was a problem that was fixed in 1.1. But from what I'm hearing, it is perhaps better, but not fixed. Just to compare the guarantees: Ext3 (data=journal): After an unclean shutdown, Fileysystem and metadata are guaranteed to be consistent. If the application was told that a data write occurred, it did. (Slowest writes.) Ext3 (data=ordered): After a power outage, the filesystem and metadata are guaranteed to be consistent. The file contents are guaranteed not to be garbage. BUT, the data might not be the latest written. (Medium speed writes.) Ext3 (data=writeback): This is the level that most journaled filesystems give you. After an unclean shutdown, filesystem and metadata are guaranteed to be consistent. File can contain garbage. (Fastest writes.) XFS: Same as data=writeback for ext3, but due to the way XFS nulls out data blocks, and also delays allocation, data loss is more likely. Someone more familiar with XFS might want to elaborate or correct my sketchy description of the reasons for XFS's relative fragility. Be advised that changing the data journaling option for a *root* filesystem is more involved than just changing it in /etc/fstab. For /home, /var, etc. you can just add it to fstab and it will work. -Steve
Kirk Bocek wrote:> 3Ware's site advertises 300MB/Sec writes using 8 disks on the PCI > Express version of this card (the 9580 I think.) I'm guessing 4x PCI > Express has more bandwidth but 133Mhz 64-bit PCI-X is no slouch. The > only two things I need to change are to put the disks in 3GB/Sec SATA > mode (they're in 1.5GB/Sec mode right now) and to enable memory > interleaving on the motherboard (I only have two sticks of memory, > this M/B needs 4 for interleaving.) But I don't think these things > should drop me down to the write performance of a single IDE disk.Has somebody made experience with PCI-X-133MHz and PCI-X-66MHz? With my board there's not much difference. I even have the feeling, that driving a 9550SXU-4LP at PCI-X-66MHz is a bit faster than at 133. At least bonnie++ gives me slightly faster values @ 66.> > Does anyone have any suggestions on improving write performance?see above. :) TIA - Michael -- Michael Kress, kress at hal.saar.de http://www.michael-kress.de / http://kress.net P E N G U I N S A R E C O O L