zfs discuss - Dec 2009 - Benchmarks results for ZFS + NFS, using SSD''s as slog devices (ZIL)

Hello, 


We have performed several tests to measure the performance 
using SSD drives for the ZIL. 

Tests are performed using a X4540 "Thor" with a zpool consisting of
3 14-disk RaidZ2 vdevs. This fileserver is connected to a Centos 5.4
machine which mounts a filesystem on the zpool via NFS, over a 
dedicated, direct, 1Gb ethernet link. 

The issue we''re trying to resolve by using SSD''s, is the
much-discussed
slow NFS performance when using synchronous IO. Unfortunately,
asynchronous IO is not possible, since the Solaris NFS server is 
synchronous by default, and the linux clients are unable to request
asynchronous NFS traffic.

The SSD devices we''ve used are OCZ Vertex Turbo 30Gb disks.

Data was gathered using iozone from the centos machine: 
(iozone -c -e -i 0 -i 1 -i 2 -o -a).

The raw data as well as the graphs that I created are available on
request, should people be interested.

Since we are interested in using the Thor as an NFS file server
for homedirectories, we are mostly concerned about random write
performance.

We have made the following observations:

 - Using SSD devices as ZIL logs yields a factor 2 improvement in throughput
   when using a recordsize <= 128k, in comparison to using the internal
   ZIL devices of the pool (ie. not setting up slog devices).

 - With recordsizes of 1MB and up, having the ZIL reside on the raw disks of
   pool (no separate slog devices) outperforms using SSD''s as a slog
device.

 - Disabling the ZIL altogether yields significantly better performance 
   (at least a factor 10).

We had hoped that using SSD''s would yield better performance.  It is
possible we will see an improvement with Intel X25-E series SSD''s,
but those haven''t arrived yet so we can''t test that.

An alternative test we performed was extracting a 138Mb tarfile
consisting of ~2000 small files. With the ZIL disabled, extracting
the file took 4 seconds.

With the ZIL enabled, but with no specific slog devices in the pool
(thus using the disks in the pool), extraction took 72seconds. Using
the SSD''s as log devices, the time required was reduced to 34 seconds.
This corresponds to the ~factor 2 improvement we noticed using our
iozone benchmark.  For this specific workload, we noticed no difference
in using 1 or 2 (striped) slog SSD''s.


At the bottom line, lets end up with a few specific questions: 

1. Is this performance using SSD''s as expected? Can we expect better
performance
   using Intel X25-E SSD''s?
2. Disabling the ZIL looks like a serious option, after these performance 
   benchmarks. I would expect to see Disabling the ZIL as an "officially
supported
   option", given that we all have used UFS for years, which is no better
in
   terms of reliability. Is there an "Official Sun Response" to this?



with kind regards,
Auke Folkerts
University of Amsterdam
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091223/fce2e4d3/attachment.bin>

Some questions below...

On Dec 23, 2009, at 8:27 AM, Auke Folkerts wrote:
> Hello,
>
>
> We have performed several tests to measure the performance
> using SSD drives for the ZIL.
>
> Tests are performed using a X4540 "Thor" with a zpool consisting
of
> 3 14-disk RaidZ2 vdevs. This fileserver is connected to a Centos 5.4
> machine which mounts a filesystem on the zpool via NFS, over a
> dedicated, direct, 1Gb ethernet link.
>
> The issue we''re trying to resolve by using SSD''s, is the
much-
> discussed
> slow NFS performance when using synchronous IO. Unfortunately,
> asynchronous IO is not possible, since the Solaris NFS server is
> synchronous by default, and the linux clients are unable to request
> asynchronous NFS traffic.
>
> The SSD devices we''ve used are OCZ Vertex Turbo 30Gb disks.
>
> Data was gathered using iozone from the centos machine:
> (iozone -c -e -i 0 -i 1 -i 2 -o -a).
>
> The raw data as well as the graphs that I created are available on
> request, should people be interested.
Yes, can you post somewhere?
> Since we are interested in using the Thor as an NFS file server
> for homedirectories, we are mostly concerned about random write
> performance.
>
> We have made the following observations:
>
> - Using SSD devices as ZIL logs yields a factor 2 improvement in  
> throughput
>   when using a recordsize <= 128k, in comparison to using the internal
>   ZIL devices of the pool (ie. not setting up slog devices).
>
> - With recordsizes of 1MB and up, having the ZIL reside on the raw  
> disks of
>   pool (no separate slog devices) outperforms using SSD''s as a
slog
> device.
>
> - Disabling the ZIL altogether yields significantly better performance
>   (at least a factor 10).
>
> We had hoped that using SSD''s would yield better performance.  It
is
> possible we will see an improvement with Intel X25-E series SSD''s,
> but those haven''t arrived yet so we can''t test that.
>
> An alternative test we performed was extracting a 138Mb tarfile
> consisting of ~2000 small files. With the ZIL disabled, extracting
> the file took 4 seconds.
>
> With the ZIL enabled, but with no specific slog devices in the pool
> (thus using the disks in the pool), extraction took 72seconds. Using
> the SSD''s as log devices, the time required was reduced to 34
seconds.
> This corresponds to the ~factor 2 improvement we noticed using our
> iozone benchmark.  For this specific workload, we noticed no  
> difference
> in using 1 or 2 (striped) slog SSD''s.
Questions:
	1. Client wsize?
	2. Client NFS version?
	3. logbias settings?
	4. Did you test with a Solaris NFS client?  If not, why not?
> At the bottom line, lets end up with a few specific questions:
>
> 1. Is this performance using SSD''s as expected? Can we expect
better
> performance
>   using Intel X25-E SSD''s?
> 2. Disabling the ZIL looks like a serious option, after these  
> performance
>   benchmarks. I would expect to see Disabling the ZIL as an  
> "officially supported
>   option", given that we all have used UFS for years, which is no  
> better in
>   terms of reliability. Is there an "Official Sun Response" to
this?
UFS is a totally different issue, sync writes are always sync''ed.

I don''t work for Sun, but it would be unusual for a company to accept
willful negligence as a policy.  Ambulance chasing lawyers love that
kind of thing.
  -- richard

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

Hi Richard, ZFS-discuss.
> Message: 2
> Date: Wed, 23 Dec 2009 09:49:18 -0800
> From: Richard Elling <richard.elling at gmail.com>
> To: Auke Folkerts <folkerts at science.uva.nl>
> Cc: zfs-discuss at opensolaris.org
> Subject: Re: [zfs-discuss] Benchmarks results for ZFS + NFS,    using
>     SSD''s as slog devices (ZIL)
> Message-ID: <40070921-F894-4146-9E4C-7570D52C83E0 at gmail.com>
> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
>
> Some questions below...
>
> On Dec 23, 2009, at 8:27 AM, Auke Folkerts wrote:
>

Filling in for Auke here,
>> > The raw data as well as the graphs that I created are available on
>> > request, should people be interested.
>
> Yes, can you post somewhere?
I''ve put the results here, tests are run under nv129:

http://www.science.uva.nl/~jeroen/solaris11_iozone_nfs2zfs

Original measurements (with iozone headers) are in:

http://www.science.uva.nl/~jeroen/solaris11_iozone_nfs2zfs/originals/

>
> Questions:
>     1. Client wsize?
We usually set these to 342768 but this was tested with CenOS
defaults: 8192 (were doing this over NFSv3)
>     2. Client NFS version?
NFSv3 (earlier tests show about 15% improvement using v4, but we still
use v3 in production).

>     3. logbias settings?
Throughput for runs labeled "throughput" otherwise defaults.

>     4. Did you test with a Solaris NFS client?  If not, why not?
We didn''t, because our production environment consists of Solaris
servers and Linux/MS Windows clients.

> UFS is a totally different issue, sync writes are always sync''ed.
>
> I don''t work for Sun, but it would be unusual for a company to
accept
> willful negligence as a policy.  Ambulance chasing lawyers love that
> kind of thing.
The Thor replaces a geriatric Enterprise system running Solaris 8 over
UFS. For these workloads it beat the pants out of our current setup
and somehow the "but you''re safer now" argument
doesn''t go over very
well :)

We are under the impression that a setup that server NFS over UFS has
the same assurance level than a setup using "ZFS without ZIL". Is this
impression false?

If it isn''t then offering a tradeoff between "same assurance level
as
you are used to with better performance" or "better assurance level
but for random-IO significant performance hits" doesn''t seem too
wrong
to me. In the first case you still have the ZFS guarantees once data
is "on disk"...

Thanks in advance for your insights,

With kind regards,

Jeroen

- --
Jeroen Roodhart            
IT Consultant
                             University of Amsterdam   
j.r.roodhart at uva.nl          Informatiseringscentrum 
Tel. 020 525 7203
- --
See http://www.science.uva.nl/~jeroen for openPGP public key
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFLMqKT37AP1zFtDU0RAxeCAKDcglo2n0Q8Sx0tGyzx+MEGJt90TwCfWm59
JbGdTavhenqSrQEtGUvPZaw=K25S
-----END PGP SIGNATURE-----

On Thu, Dec 24, 2009 at 12:07:03AM +0100, Jeroen Roodhart
wrote:
> We are under the impression that a setup that server NFS over UFS has
> the same assurance level than a setup using "ZFS without ZIL". Is
this
> impression false?
Completely.  It''s closer to "UFS mount -o async", without the
risk of
UFS corruption, but still with the risk of client-side corruption of
the NFS write semantics. 

--
Dan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091224/6cd11236/attachment.bin>

>> UFS is a totally different issue, sync writes are always
sync''ed.
>>
>> I don''t work for Sun, but it would be unusual for a company to
accept
>> willful negligence as a policy. ?Ambulance chasing lawyers love that
>> kind of thing.
>
> The Thor replaces a geriatric Enterprise system running Solaris 8 over
> UFS. For these workloads it beat the pants out of our current setup
> and somehow the "but you''re safer now" argument
doesn''t go over very
> well :)
>
> We are under the impression that a setup that server NFS over UFS has
> the same assurance level than a setup using "ZFS without ZIL". Is
this
> impression false?
That impression is false!

No ZIL is especially bad for NFS applications.

If you have disabled ZIL and you reboot your NFS server while a client
is writing you will have writes that have "disappeared" and no error
will be logged on the server or the client. Thus the result till be
corrupted files and no way to know that the are corrupted.

UFS (with or without the log) will behave as ZFS with ZIL.

The ZIL is _not_ optional as the log is in UFS.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

Mattias Pantzare wrote:
>
> The ZIL is _not_ optional as the log is in UFS.
Right, thanks (also to Richard and Daniel) for the explanation. I was
afraid this was to good to be true, nice to see it stated this clearly
though.

That  would leave us with three options;

1) Deal with it and accept performance as it is.
2) Find a way to speed things up further for this workload
3) Stop trying to use ZFS for this workload

Option 1 is not going to be feasible, so we''re left with 2 and 3.

We will have to do some more benchmarks in the new year. Maybe if
trying different NFS wsize-s  results in different figures. Also we''ll
look at UFS on the Thor, although I am not looking forward to handle
large amounts of data on anything other than ZFS. Spoiled for life
probably :)

In the mean time, if any of you would have time to look at our iozone
data and spotted glaring mistakes, we would definitely appreciate your
comments.

Thanks for your help,

With kind regards,

Jeroen

- --
Jeroen Roodhart           
IT Consultant
                             University of Amsterdam  
j.r.roodhart at uva.nl          Informatiseringscentrum
Tel. 020 525 7203
- --
See http://www.science.uva.nl/~jeroen for openPGP public key
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD4DBQFLMynd37AP1zFtDU0RA9dNAJdhfwGH7Dj3cXBCX3MS/zTaV/c+AKCozNJn
kmxtdS9Vu/sM/icXTE0hsA==fP4d
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

Jeroen Roodhart wrote:
>> Questions: 1. Client wsize?
>
> We usually set these to 342768 but this was tested with CenOS
> defaults: 8192 (were doing this over NFSv3)
Is stand corrected here. Looking at proc/mounts I see we are in fact
using different values:

...
10.0.0.1:/mypool/test_FS /nfs nfs
rw,vers=3,rsize=1048576,wsize=1048576,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=10.0.0.1
0 0

So wsize was 1048576 during the iozone tests. That''ll teach me to rely
on manuals :) So
repeating these tests with different wsizes seems to be a smart thing
to do.

With kind regards,

Jeroen

- --
Jeroen Roodhart              
IT Consultant
                             University of Amsterdam     
j.r.roodhart at uva.nl          Informatiseringscentrum   
Tel. 020 525 7203
- --
See http://www.science.uva.nl/~jeroen for openPGP public key
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFLMy0737AP1zFtDU0RA5OmAKDtt7LMRu/a36PBa+Fg5vL2pDKdGACdFIkU
fWkmBPJvNwoQpm2A4Y3SorA=gaX7
-----END PGP SIGNATURE-----

On Dec 24, 2009, at 12:44 AM, Jeroen Roodhart wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: RIPEMD160
>
> Mattias Pantzare wrote:
>>
>> The ZIL is _not_ optional as the log is in UFS.
>
> Right, thanks (also to Richard and Daniel) for the explanation. I was
> afraid this was to good to be true, nice to see it stated this clearly
> though.
>
> That  would leave us with three options;
>
> 1) Deal with it and accept performance as it is.
> 2) Find a way to speed things up further for this workload
> 3) Stop trying to use ZFS for this workload
>
> Option 1 is not going to be feasible, so we''re left with 2 and 3.
>
> We will have to do some more benchmarks in the new year. Maybe if
> trying different NFS wsize-s  results in different figures. Also
we''ll
> look at UFS on the Thor, although I am not looking forward to handle
> large amounts of data on anything other than ZFS. Spoiled for life
> probably :)
UFS is really not an option. The absolute maximum possible size for
a UFS file system is 16 TB, but 1 TB is the practical maximum.  For a
thumper or thor, you''d be crazy trying to manage 24 UFS file systems.
Not a good return for the effort.
> In the mean time, if any of you would have time to look at our iozone
> data and spotted glaring mistakes, we would definitely appreciate your
> comments.
How about posting the data somewhere we can see it?
  -- richard

[revisiting the OP]

On Dec 23, 2009, at 8:27 AM, Auke Folkerts wrote:
> Hello,
>
>
> We have performed several tests to measure the performance
> using SSD drives for the ZIL.
>
> Tests are performed using a X4540 "Thor" with a zpool consisting
of
> 3 14-disk RaidZ2 vdevs. This fileserver is connected to a Centos 5.4
> machine which mounts a filesystem on the zpool via NFS, over a
> dedicated, direct, 1Gb ethernet link.
>
> The issue we''re trying to resolve by using SSD''s, is the
much-
> discussed
> slow NFS performance when using synchronous IO. Unfortunately,
> asynchronous IO is not possible, since the Solaris NFS server is
> synchronous by default, and the linux clients are unable to request
> asynchronous NFS traffic.
>
> The SSD devices we''ve used are OCZ Vertex Turbo 30Gb disks.
>
> Data was gathered using iozone from the centos machine:
> (iozone -c -e -i 0 -i 1 -i 2 -o -a).
For the archives, these options are interpreted as:

	-a    test a number file and I/O sizes, 64 KB to 512 MB and 4 KB
	      to 16 MB respectively.

	-c    measure close() timing (note: Solaris file systems enforce
	      sync-on-close, some Linux file systems make this optional
	      thereby risking data)

	-e   include flush in timing calculations (note: before you measure
	      this, make sure your NFS server actually respects syncs,
	      Solaris does)

	-i 0  run write/rewrite test

	-i 1  run read/re-read test  (note: this tests server side read cache,
	      and is probably not a very useful test unless both your client  
and
	      server is memory constrained)

	-i 2  run random read/write test

	-o    writes are synchronously written to the disk. Files are opened
	      with the O_SYNC flag.  This would make sense if your workload
	      is a database, like Oracle, which opens its datafiles O_DSYNC.
	      It is not at all representative of normal file system use.

OK, so you are interested in finding out how fast sync writes can be
written on a Solaris file system and NFS service that ensures sync
writes are flushed to media. To fully understand how this is impacted
by ZFS, you need to understand how the ZIL works and the impact
of the logbias setting. This is why I asked about wsize, because by
default, that is automatically adjusted by an algorithm that you can
override with logbias settings. For tests that do a number of different
I/O sizes, you should be able to see the knee in the curve as a
result of this algorithm. In short, you can optimize for latency or
bandwidth. Don''t expect good latency when you are optimized for
bandwidth.
  -- richard

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

Hi Richard,

Richard Elling wrote:
> How about posting the data somewhere we can see it?
As stated in an earlier posting it should be accessible at:

http://init.science.uva.nl/~jeroen/solaris11_iozone_nfs2zfs

Happy holidays!

~Jeroen

- --
Jeroen Roodhart           
IT Consultant
                             University of Amsterdam  
j.r.roodhart at uva.nl          Informatiseringscentrum
Tel. 020 525 7203
- --
See http://www.science.uva.nl/~jeroen for openPGP public key
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFLM9Pf37AP1zFtDU0RA0ZxAJ9IJmn8fc6sBWi2KG7sVlL8RLiZ0wCbBcbL
4+5IT9mxhgIqRm+j5Mx9Kqk=nKUC
-----END PGP SIGNATURE-----

> Mattias Pantzare wrote:
> That  would leave us with three options;
> 
> 1) Deal with it and accept performance as it is.
> 2) Find a way to speed things up further for this
> workload
> 3) Stop trying to use ZFS for this workload
Option 4 is to re-do your pool, using fewer disks per raidz2 vdev, giving more
vdevs to the pool, and thus increasing the IOps for the whole pool.

14 disks in a single raidz2 vdev is going to give horrible IO, regardless of how
fast the individual disks are.

Redoing it with 6-disk raidz2 vdevs, or even 8-drive raidz2 vdevs will give you
much better throughput.

Freddie
-- 
This message posted from opensolaris.org

On Dec 24, 2009, at 5:34 PM, Freddie Cash wrote:
>> Mattias Pantzare wrote:
>> That  would leave us with three options;
>>
>> 1) Deal with it and accept performance as it is.
>> 2) Find a way to speed things up further for this
>> workload
>> 3) Stop trying to use ZFS for this workload
>
> Option 4 is to re-do your pool, using fewer disks per raidz2 vdev,  
> giving more vdevs to the pool, and thus increasing the IOps for the  
> whole pool.
>
> 14 disks in a single raidz2 vdev is going to give horrible IO,  
> regardless of how fast the individual disks are.
>
> Redoing it with 6-disk raidz2 vdevs, or even 8-drive raidz2 vdevs  
> will give you much better throughput.
At this point it is useful to know that if you do not have a
separate log, then the ZIL uses the pool and its data protection
scheme.  In other words, each ZIL write will be a raidz2 stripe
with its associated performance.
  -- richard

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

Hi Freddie, list,
> Option 4 is to re-do your pool, using fewer disks per raidz2 vdev,
> giving more vdevs to the pool, and thus increasing the IOps for the
> whole pool.
>
> 14 disks in a single raidz2 vdev is going to give horrible IO,
> regardless of how fast the individual disks are.
>
> Redoing it with 6-disk raidz2 vdevs, or even 8-drive raidz2 vdevs
> will give you much better throughput.
We are aware of the configuration being possibly suboptimal. However,
before we had the SSDs, we did test earlier with 6x7 Z2 and even 2way
mirrorset setups. These gave better IOPS but not significantly enough
improvement (I would expect roughly a bit more than double the
performance in 14x3 vs 6x7) .  In the end it is indeed a choice
between performance, space and security.  Our hope is that the SSD
slogs "serialise" the  data flow  enough  to make this work. But you
have a fair point and we will also look into the combination of SSDs
and pool-configurations.

Also, possibly the Vortex Turbo SSDs aren''t as good latency wise as I
expected. Maybe the Sun SSDs will do a lot better. We will find this
out when they arrive (due somewhere in february).

With kind regards,

Jeroen

- --
Jeroen Roodhart              
IT Consultant
                             University of Amsterdam     
j.r.roodhart at uva.nl          Informatiseringscentrum   
Tel. 020 525 7203
- --
See http://www.science.uva.nl/~jeroen for openPGP public key
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFLNURP37AP1zFtDU0RA0QqAKDg/hr68JPjLvc0gaOmCe4RxPXY3QCg1G+g
e6BTEKqq6QxpePonnn54fOo=JQcb
-----END PGP SIGNATURE-----

Jeroen Roodhart wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: RIPEMD160
>
> Hi Freddie, list,
>
>   
>> Option 4 is to re-do your pool, using fewer disks per raidz2 vdev,
>> giving more vdevs to the pool, and thus increasing the IOps for the
>> whole pool.
>>
>> 14 disks in a single raidz2 vdev is going to give horrible IO,
>> regardless of how fast the individual disks are.
>>
>> Redoing it with 6-disk raidz2 vdevs, or even 8-drive raidz2 vdevs
>> will give you much better throughput.
>>     
>
> We are aware of the configuration being possibly suboptimal. However,
> before we had the SSDs, we did test earlier with 6x7 Z2 and even 2way
> mirrorset setups. These gave better IOPS but not significantly enough
> improvement (I would expect roughly a bit more than double the
> performance in 14x3 vs 6x7) .  In the end it is indeed a choice
> between performance, space and security.  Our hope is that the SSD
> slogs "serialise" the  data flow  enough  to make this work. But
you
> have a fair point and we will also look into the combination of SSDs
> and pool-configurations.
>
> Also, possibly the Vortex Turbo SSDs aren''t as good latency wise
as I
> expected. Maybe the Sun SSDs will do a lot better. We will find this
> out when they arrive (due somewhere in february).
>
> With kind regards,
>
> Jeroen
>
> - 
I haven''t seen this mentioned before, but the OCZ Vertex Turbo is still
an MLC-based SSD, and is /substantially/ inferior to an Intel X25-E in 
terms of random write performance, which is what a ZIL device does 
almost exclusively in the case of NFS traffic.

In fact, I think that the Vertex''s sustained random write IOPs 
performance is actually inferior to a 15k SAS drive.



-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA
Timezone: US/Pacific (GMT-0800)

On Dec 25, 2009, at 4:15 PM, Erik Trimble wrote:
>
> I haven''t seen this mentioned before, but the OCZ Vertex Turbo is
> still an MLC-based SSD, and is /substantially/ inferior to an Intel  
> X25-E in terms of random write performance, which is what a ZIL  
> device does almost exclusively in the case of NFS traffic.
ZIL traffic tends to be sequential on separate logs. But may be of  
different sizes.
  -- richard
> In fact, I think that the Vertex''s sustained random write IOPs  
> performance is actually inferior to a 15k SAS drive.
I read a benchmark report yesterday that might be interesting.  It  
seems that
there is a market for modest sized SSDs, which would be perfect for  
separate
logs + OS for servers.
http://benchmarkreviews.com/index.php?option=com_content&task=view&id=392&Itemid=60
  -- richard

On Dec 25, 2009, at 6:01 PM, Jeroen Roodhart <j.r.roodhart at uva.nl>  
wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: RIPEMD160
>
> Hi Freddie, list,
>
>> Option 4 is to re-do your pool, using fewer disks per raidz2 vdev,
>> giving more vdevs to the pool, and thus increasing the IOps for the
>> whole pool.
>>
>> 14 disks in a single raidz2 vdev is going to give horrible IO,
>> regardless of how fast the individual disks are.
>>
>> Redoing it with 6-disk raidz2 vdevs, or even 8-drive raidz2 vdevs
>> will give you much better throughput.
>
> We are aware of the configuration being possibly suboptimal. However,
> before we had the SSDs, we did test earlier with 6x7 Z2 and even 2way
> mirrorset setups. These gave better IOPS but not significantly enough
> improvement (I would expect roughly a bit more than double the
> performance in 14x3 vs 6x7) .  In the end it is indeed a choice
> between performance, space and security.  Our hope is that the SSD
> slogs "serialise" the  data flow  enough  to make this work. But
you
> have a fair point and we will also look into the combination of SSDs
> and pool-configurations.
>
> Also, possibly the Vortex Turbo SSDs aren''t as good latency wise
as I
> expected. Maybe the Sun SSDs will do a lot better. We will find this
> out when they arrive (due somewhere in february).
Understand too that every write is striped across the vdev which means  
the vdev has the IOPS of a single drive.

A 14 drive pool of mirrors would provide 7 times the IOPS then a 14  
drive raidz2/3

Richard Elling wrote:
> On Dec 25, 2009, at 4:15 PM, Erik Trimble wrote:
>>
>> I haven''t seen this mentioned before, but the OCZ Vertex Turbo
is
>> still an MLC-based SSD, and is /substantially/ inferior to an Intel 
>> X25-E in terms of random write performance, which is what a ZIL 
>> device does almost exclusively in the case of NFS traffic.
>
> ZIL traffic tends to be sequential on separate logs. But may be of 
> different sizes.
>  -- richard
Really?  Now that I think about it, that seems to make sense - I was 
assuming that each NFS write would be relatively small, but that''s 
likely not a valid general assumption.  I''d still think that the 
MLC-nature of the Vertex isn''t optimal for write-heavy applications
like
here, even with a modest SDRAM cache on the SSD.

>> In fact, I think that the Vertex''s sustained random write IOPs
>> performance is actually inferior to a 15k SAS drive.
>
> I read a benchmark report yesterday that might be interesting.  It 
> seems that
> there is a market for modest sized SSDs, which would be perfect for 
> separate
> logs + OS for servers.
>
http://benchmarkreviews.com/index.php?option=com_content&task=view&id=392&Itemid=60
>
>  -- richard
I''m still hoping that vendors realize that there definitely is a (very 
large) market for ~20GB high write IOPS SSDs.    I like my 18GB Zeus 
SSD, but it sure would be nice to be able to pay $2/Gb for it, instead 
of 10x that now...

-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA
Timezone: US/Pacific (GMT-0800)

On Dec 25, 2009, at 3:01 PM, Jeroen Roodhart wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: RIPEMD160
>
> Hi Freddie, list,
>
>> Option 4 is to re-do your pool, using fewer disks per raidz2 vdev,
>> giving more vdevs to the pool, and thus increasing the IOps for the
>> whole pool.
>>
>> 14 disks in a single raidz2 vdev is going to give horrible IO,
>> regardless of how fast the individual disks are.
>>
>> Redoing it with 6-disk raidz2 vdevs, or even 8-drive raidz2 vdevs
>> will give you much better throughput.
>
> We are aware of the configuration being possibly suboptimal. However,
> before we had the SSDs, we did test earlier with 6x7 Z2 and even 2way
> mirrorset setups. These gave better IOPS but not significantly enough
> improvement (I would expect roughly a bit more than double the
> performance in 14x3 vs 6x7) .  In the end it is indeed a choice
> between performance, space and security.  Our hope is that the SSD
> slogs "serialise" the  data flow  enough  to make this work. But
you
> have a fair point and we will also look into the combination of SSDs
> and pool-configurations.
For your benchmark, there will not be a significant difference for any
combination of HDDs. They all have at least 4 ms of write latency.
Going from 10 ms down to 4 ms will not be nearly as noticeable as
going from 10 ms to 0.01 ms :-)
  -- richard