zfs discuss - May 2010 - mirror writes 10x slower than individual writes

I have an odd setup at present, because I''m testing while still
building my machine.

It''s an Intel Atom D510 mobo running snv_134 2GB RAM with 2 SATA drives
(AHCI):
1: Samsung 250GB old laptop drive
2: WD Green 1.5TB drive (idle3 turned off)

Ultimately, it will be a time machine backup for my Mac laptop. So I have
installed Netatalk 2.1.1 which is working great.

Read performance from the mirror, via gigabit ethernet rocks, easily sustaining
50MB/s off the two drives mirrored.

However, write performance is terrible, typically no better than 1-2MB/s on
average.

I just thought to detach the WD drive from the mirror and test the drives
individually, so with the system still running on drive 1 I create an
independent zpool on the other drive and a netatalk share to it.

Using `dd` to copy a single large file, to each drive the results are:

Drive 1: Samsung (rpool, and there''s a scrub going on)
1565437216 bytes transferred in 98.236700 secs (15935360 bytes/sec)

Drive 2: Western Digital 1.5TB green drive:
1565437216 bytes transferred in 71.745737 secs (21819237 bytes/sec)

However, when the two drives were mirrored, after all resilvering completed and
there was no background I/O, the write performance was about 10x worse.

Watching `zpool iostat -v 2` I could see that quite often drive 1 would write a
big chunk of data and then wait for ages for drive 2 to write the same data to
disc.

Could it be that there is a separate cache for the mirror that was stalling
waiting on the cache for the larger drive??

Would this scenario be caused because the drives are so different in size? 250GB
and 1500GB??

Once the scrub finishes, I''ll re-attach the mirror, and re-test
tomorrow, reporting the `zpool iostat` in detail...
-- 
This message posted from opensolaris.org

On Mon, 31 May 2010, Matt Connolly wrote:
> Watching `zpool iostat -v 2` I could see that quite often drive 1 
> would write a big chunk of data and then wait for ages for drive 2 
> to write the same data to disc.
>
> Could it be that there is a separate cache for the mirror that was 
> stalling waiting on the cache for the larger drive??
>
> Would this scenario be caused because the drives are so different in 
> size? 250GB and 1500GB??
I think that the problem is likely that this Western Digital drive is 
one of the new ones using 4K sectors internally.  Almost everyone 
obtains terrible write performance from zfs using these drives.  I 
suggest getting rid of it and replace it with a drive which still uses 
standard 512 byte sectors internally.  It will be like night and day.

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Mon, May 31, 2010 at 1:19 PM, Bob Friesenhahn
<bfriesen at simple.dallas.tx.us> wrote:
> I think that the problem is likely that this Western Digital drive is one
of
> the new ones using 4K sectors internally. ?Almost everyone obtains terrible
> write performance from zfs using these drives. ?I suggest getting rid of it
> and replace it with a drive which still uses standard 512 byte sectors
> internally. ?It will be like night and day.
You can verify if this is the case by running format.

      10. c2t7d0 <ATA-WDC WD10EADS-00L-1A01-931.51GB>
          /pci at 0,0/pci10de,376 at a/pci8086,32c at 0/pci11ab,11ab at 4/disk
at 7,0

I have 1TB WD Green drives that are NOT 4k sectors, the WD10EADS. The
EADS drives have not been available for purchase since December 09 or
so. If you bought your drive this year, it''s probably using 4k
sectors.

The 4k sector ("Advanced Format") are WD10EARS, WD15EARS and WD20EARS.
Someone else has posted that they got good performance with these
drives in a mirror, but not in a raidz, so mirroring two of them may
give you better performance, too.

-B

-- 
Brandon High : bhigh at freaks.com