Hello,
Short version: Pool A is fast, pool B is slow. Writing to pool A is
fast. Writing to pool B is slow. Writing to pool B WHILE writing to
pool A is fast on both pools. Explanation?
Long version:
I have an existing two-disk pool consisting of two SATA drives. Call
this pool "pool1". This has always been as fast as I would have
expected; 30-50 MB/second write, 105 MB/second read.
I have now added four additional drives (A, B, C and D) to the
machine, that I wanted to use for a raidz. For initial testing I chose
a striped pool just to see what kind of performance I would get.
The initial two drives (pool1) are on their own controller. A and B
are on a second controller and C and D on a third controller. All of
the controllers are SiL3512.
Here comes the very interesting bit. For the purpose of the diagram
below, "good" performance means 20-50 mb/sec write and ~70-80 mb/sec
read. "Bad" performance means 3-5 mb/sec (!!!!) write and ~70-80
mb/sec read.
disk layout in pool | other I/O | performance
------------------------------------------------------------------------
A + B | none | good
C + D | none | good
A + B + C + D | none | bad
A + B + C | none | bad
A + B + C + D | write to pool1 | goodish (!!!)
(some tested combinations omitted)
In other words: Initially it looked like write performance went down
the drain as soon as I combined drives from multiple controllers into
one pool, while performance was fine as long as I stayed within one
controller.
However, writing to the "slow" A+B+C+D pool *WHILE ALSO WRITING TO
POOL1* actually *INCREASES* performance. The write to pool1 and the
otherwise "slow" pool are not quite up to normal "good"
level, but
that is probably to be expected even under normal circumstances.
CPU usage during the writing (when slow) is almost non-existent. There
is no spike similar to what you seem to get every five second or so
normally (during transaction commits?).
Also, at least once I saw the write performance on the "slow" pool
spike at 19 mb/second for a single second period (zpool iostat) when I
initiated the write, then it went down again and remains very
constant, not really varying outside 3.4-4.5. Often EXACTLY at 3.96.
"writing" and "reading" means dd:ing (to /dev/null, from
/dev/zero)
with bs=$((1024*1024)).
Pools created with "zpool create speedtest c4d0 c5d0 c6d0 c7d0" and
variations of that for the different combinations. The pool with all
four drives is 1.16T in size.
--
/ Peter Schuller, InfiDyne Technologies HB
PGP userID: 0xE9758B7D or ''Peter Schuller <peter.schuller at
infidyne.com>''
Key retrieval: Send an E-Mail to getpgpkey at scode.org
E-Mail: peter.schuller at infidyne.com Web: http://www.scode.org