zfs discuss - Feb 2010 - Speed question: 8-disk RAIDZ2 vs 10-disk RAIDZ3

I currently am getting good speeds out of my existing system (8x 2TB in a
RAIDZ2 exported over fibre channel) but there''s no such thing as too
much
speed, and these other two drive bays are just begging for drives in
them.... If I go to 10x 2TB in a RAIDZ3, will the extra spindles increase
speed, or will the extra parity writes reduce speed, or will the two factors
offset and leave things a wash?
    (My goal is to be able to survive one controller failure, so if I add
more drives I''ll have to add redundancy to compensate for the fact that
one
controller would then be able to take out three drives.)
    I''ve considered adding a drive for the ZIL instead, but my
experiments
in disabling the ZIL (using the evil tuning guide at
<http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Disabl
ing_the_ZIL_.28Don.27t.29>) didn''t show any speed increase. (I know
it''s a
bad idea run the system with ZIL disabled; I disabled it only to measure its
impact on my write speeds and re-enabled it after testing was complete.)

Current system:
OpenSolaris dev release b132
Intel S5500BC mainboard (latest firmware)
Intel E5506 Xeon 2.13GHz
8GB RAM
3x LSI 3018 PCIe SATA controllers (latest IT firmware)
8x 2TB Hitachi 7200RPM SATA drives (2 connected to each LSI and 2 to
motherboard SATA ports)
2x 60GB Imation M-class SSD (boot mirror)
Qlogic 2440 PCIe Fibre Channel HBA
-- 
Dave Pooser, ACSA
Manager of Information Services
Alford Media  http://www.alfordmedia.com

> If I go to 10x 2TB in a RAIDZ3, will the extra spindles increase
> speed, or will the extra parity writes reduce speed, or will the two
factors
> offset and leave things a wash?
I should mention that the usage of this system is as storage for large
(5-300GB) video files, so what''s most important is sequential write
speed.
-- 
Dave Pooser, ACSA
Manager of Information Services
Alford Media  http://www.alfordmedia.com

On Tue, 16 Feb 2010, Dave Pooser wrote:
>> If I go to 10x 2TB in a RAIDZ3, will the extra spindles increase
>> speed, or will the extra parity writes reduce speed, or will the two
factors
>> offset and leave things a wash?
>
> I should mention that the usage of this system is as storage for large
> (5-300GB) video files, so what''s most important is sequential
write speed.
With 10 disks, I would go for two raidz2 vdevs of 5 disks each.  This 
is not as space efficient as your one raidz3 vdev with 10 disks but it 
is also likely to be a bit more responsive, and withstand a 
(temporary) slowdown due to a single slow disk a bit better.  With a 
single raidz3 vdev, write performance will go into the toilet if even 
one disk becomes a bit balky.  Resilver times when a disk is replaced 
are also likely to be longer.

Disks are far more likely to fail than the controller they are 
attached to, and disk failures are often slow to occur, or not obvious 
as failures without close scrutiny with tools like ''iostat
-x''.

There have been many reports here about raidz-based pools which became 
very slow, with the eventual finding that the slowness was due to just 
one balky disk.  I think that this is what you need to prepare for, 
particularly with hardware going out on a truck to the field.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/