zfs discuss - Apr 2009 - Unexpectedly poor 10-disk RAID-Z2 performance?

Hail, caesar.

I''ve got a 10-disk RAID-Z2 backed by the 1.5 TB Seagate drives
everyone''s so fond of. They''ve all received a firmware upgrade
(the
sane one, not the one that caused your drives to brick if the internal
event log hit the wrong number on boot).
They''re attached to an ARC-1280ML, a reasonably good SATA controller,
which has 1 GB of ECC DDR2 for caching purposes.

The machine this is all inside of is backed by an nForce 4 chipset
[hahaha], with a dual-core 1.8 GHz Opteron and 4 GB of ECC RAM
onboard. No swap is enabled.

I''m running snv_112.

I expect to get very nice performance out of the above configuration,
both locally and remotely. And I do, sometimes. I get ~70 MB/s
sustained random reads and something to the tune of 40-50 MB/s random
writes.

But a lot of the time, random small reads and writes will block for
several seconds (!), while zpool iostat won''t show any particularly
exciting throughput issues (~20 MB/s read in use on any given disk).

I tried attaching a 4GB USB flash device as a dedicated ZIL, which
improved performance a fair bit in the cases where it was already
performing well, and didn''t help at all in the cases where it was
performing poorly.

I then tried attaching a 2GB flash device as a cache device for the
pool, and it rapidly filled (obviously), but didn''t appear to give any
interesting performance wins.

I then tried attaching a 500GB 7200 RPM HD instead as a cache device,
and despite giving it several days to fill, as with the ZIL, it
improved performance marginally in cases where it was already fine and
doesn''t appear to have helped with the poor cases.

It''s not that the system is under heavy CPU load (load average is
never above .5), or memory pressure (600 MB free +/- 50 MB).

I presume I''m missing something, but I have no idea what. Halp?

Thanks,
- Rich

On Thu, 23 Apr 2009, Rince wrote:
>
> I presume I''m missing something, but I have no idea what. Halp?
The main thing to be aware of with raidz and raidz2 is that the 
devices in a vdev are basically "chained together" so that you get an 
effective one disk''s worth of IOPS per vdev.  They are very space 
efficient but not so good for multi-user use.

Reporting random read or write thoughput is not a useful measure 
unless you know how many simultaneous users are going to need that 
throughput at the same time.  The blocksize also makes a difference. 
Thoughput with 128K blocks is obviously different than with 8K blocks.

Since you have 10 disks in your raidz2, they all need to be running 
similarly very well in order to obtain the optimum performance.  Even 
one pokey disk will cause the whole vdev to suffer.  You can use 
''iostat -nx 10'' (or similar) to discover if you have a pokey
disk
which is bottlenecking your pool.

If your pool design is bottlenecked for IOPS then a single heavy IOPS 
consumer could cause performance for other users to suffer. 
Sometimes IOPS consumers may come from unexpected places such as 
FireFox.  Scripts from the DTrace Toolkit can help identify which 
processes are consuming the IOPS.

Using a SSD as your log device will help with higher-level synchronous 
write performance but it will not be able to heal a problem with your 
raidz2 vdev, which needs to do individual disk transactions at a lower 
level.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/