zfs discuss - Aug 2010 - Best usage of SSD-disk in ZFS system

Hello!

I have built a OpenSolaris / ZFS based storage system for one of our 
customers. The configuration is about this:

Motherboard/CPU: SuperMicro X7SBE / Xeon (something, sorry - can''t 
remember and do not have my specification nearby)
RAM: 8GB ECC (X7SBE won''t take more)
Drives for storage: 16*1.5TB Seagate ST31500341AS, connected to two 
AOC-SAT2-MV8 controllers
Drives for operating system: 2*80GB Intel X25-M (mirror)

ZFS configuration: Two vdevs, raid-z of 7+1 disks per set, striped 
together (gives a zpool with about 21TB storage space)

Disk performance: around 700-800MB/s, tested and timed with
''mkfile'' and
''time'' (a 40GB file is created in just about a minute)
I have a spare X25-M drive of 40GB to use for cache or log (or both), 
but since the disk array is a lot faster than the SSD-disk, I can not 
see the advantage in using it as a cache device.

Is there any advantages for using a separate log or cache device in this 
case ?

Regards,
   PeO

hello

i would say: it depends
if you fill your pool with large videos or media files,
i suppose its ok, but if you have things like databases or webserver,
you will need only good iops values, much more than you can have with spindles.
(ssd could be 100x better than disks for this use)

in this case, i would suggest to have a ssd read cache >= 80GB and a separate
(strongly recommend as mirror) log-write cache (could be much smaller, 4gb+ is
ok). do not try to build slices to use one ssd for read and write cache. you
will hate yourself in case of errors. even with newest zfs version, you cant
import a pool with missing log-drive on a new system for example after a system
crash .

also if you use dedup especially with only 8gb ram, you will not be happy about
without ssd caches.

you could use also much slower and smaller system drives.


gea  <br>
www.napp-it.org -free webgui for opensolaris-
-- 
This message posted from opensolaris.org

On Fri, Aug 6, 2010 at 6:44 AM, P-O Yliniemi <peo at bsd-guide.net> wrote:
>  Hello!
>
> I have built a OpenSolaris / ZFS based storage system for one of our
> customers. The configuration is about this:
>
> Motherboard/CPU: SuperMicro X7SBE / Xeon (something, sorry - can''t
remember
> and do not have my specification nearby)
> RAM: 8GB ECC (X7SBE won''t take more)
> Drives for storage: 16*1.5TB Seagate ST31500341AS, connected to two
> AOC-SAT2-MV8 controllers
> Drives for operating system: 2*80GB Intel X25-M (mirror)
>
> ZFS configuration: Two vdevs, raid-z of 7+1 disks per set, striped together
> (gives a zpool with about 21TB storage space)
>
> Disk performance: around 700-800MB/s, tested and timed with
''mkfile'' and
> ''time'' (a 40GB file is created in just about a minute)
> I have a spare X25-M drive of 40GB to use for cache or log (or both), but
> since the disk array is a lot faster than the SSD-disk, I can not see the
> advantage in using it as a cache device.
>
> Is there any advantages for using a separate log or cache device in this
> case ?
>
> Regards,
>  PeO
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
I can tell you for sure that there can be a really nice advantage for
sequential writes.

to see this yourself, do the following:

create a filesystem, share it out NFS
create a really big tar.gz file and put it in the filessytem

log in from a network client via nfs and extract the tar.ball using
something like:


time tar xzfv some.tar.gz


do this a few times to get an average, then add the SSD as a log device.

I have the exact same motherboard with a very similar setup, and i noticed a
400% nfs performance boost by doing this.


try it yourself =)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100806/5f9c8a45/attachment.html>

Our experience has been that a new out of the box SSD works well for the ZIL but
as soon as it''s completely full, performance drops to slower than a
regular SAS hard drive due to the write performance penalty in their fundamental
design, their LBA map strategy and the not yet released (to me at least) TRIM
support in OpenSolaris. Considering this, we only use (safe) DRAM based products
for our ZILs like the DDRDrive X1 which is incredible. For the L2ARC, SSDs are
ok but again, once they are full you still incur the write performance penalty
when writing to it so you can read from it which is also slower than a regular
SAS drive in some cases. More RAM costs more but works better.

I would recommend that you test not just initially but for at least a week. That
should give you time to fill up the drive and see what I call its
"native" performance which means how fast it can do the
read-erase-write operation under full load.
-- 
This message posted from opensolaris.org

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of P-O Yliniemi
> 
> Drives for storage: 16*1.5TB Seagate ST31500341AS, connected to two
> AOC-SAT2-MV8 controllers
> Drives for operating system: 2*80GB Intel X25-M (mirror)
>
> Is there any advantages for using a separate log or cache device in
> this
> case ?
>
That''s a bummer.  A waste of perfectly good SSD drives being used for
the
OS, which basically never sees a benefit from SSD''s.  Eh, I
don''t know.
Maybe you''re performing lots of random reads on your OS disk, although
that
seems pretty unlikely.

I saw your comment about the pool being faster than the SSD''s. 
Here''s what
you''re missing:
Spindle disks are faster than SSD''s for serial operations, sustained
throughput.  SSD''s are faster for random operations, smaller latency.

When you use a SSD for cache or log, you''re attempting to leverage the
performance benefits of both types of device.  

The ZIL is only used for sync writes.  And sync writes are naturally very
latency sensitive.  So if you add a SSD log device for dedicated ZIL, then
suddenly you''re shifting the work load profile to be more serial
operations
on the spindle disks, and more random operations on the SSD.  That is,
assuming you''re doing some sync writes.

If you aren''t doing any sync writes, then a SSD log device does you no
good
whatsoever, except to waste a drive bay.  ;-)

Thanks everyone for the suggestions. To summarize:

* There''s no point in using SSD''s for the operating system
(except an
incredibly fast boot speed, and probably a bit lower system system 
temperature / power usage - but because of the rest of the components in 
the system it doesn''t matter). I could probably use a USB key directly 
connected to the USB type A connector on the motherboard.
* Using a separate disk for logging might give problems if the log 
device goes wrong. To avoid this - keep the log on the disk pool, or 
mirror the log device.
* A SSD drive as a log device would increase read speed over network 
based sharing protocols (NFS or SMB, which will be used at the location 
of this server)
* Disk performance increased by adding SSD cache and log devices will 
die out (to a speed slower than the spindle disks) when the log and 
cache disks once have been filled up to its maximum capacity. So the use 
of SSD disks for cache and log is a waste of money and drive bays.
* Use DDR-based drives like DDRDrive (or i-RAM ?) for ZIL

I am still able to reorganize the use of the SSD disks (reinstall 
OpenSolaris), and even replace any one of them with 2.5" spindle disks, 
so any more suggestions are highly welcome.
I will do some more performance tests during my work day tomorrow.

Regards,
   PeO


P-O Yliniemi skrev 2010-08-06 12:44:
>  Hello!
>
> I have built a OpenSolaris / ZFS based storage system for one of our 
> customers. The configuration is about this:
>
> Motherboard/CPU: SuperMicro X7SBE / Xeon (something, sorry - can''t
> remember and do not have my specification nearby)
> RAM: 8GB ECC (X7SBE won''t take more)
> Drives for storage: 16*1.5TB Seagate ST31500341AS, connected to two 
> AOC-SAT2-MV8 controllers
> Drives for operating system: 2*80GB Intel X25-M (mirror)
>
> ZFS configuration: Two vdevs, raid-z of 7+1 disks per set, striped 
> together (gives a zpool with about 21TB storage space)
>
> Disk performance: around 700-800MB/s, tested and timed with
''mkfile''
> and ''time'' (a 40GB file is created in just about a
minute)
> I have a spare X25-M drive of 40GB to use for cache or log (or both), 
> but since the disk array is a lot faster than the SSD-disk, I can not 
> see the advantage in using it as a cache device.
>
> Is there any advantages for using a separate log or cache device in 
> this case ?
>
> Regards,
>   PeO
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of P-O Yliniemi
> 
> * Using a separate disk for logging might give problems if the log
> device goes wrong. To avoid this - keep the log on the disk pool, or
> mirror the log device.
Just a couple of corrections here:

You wouldn''t use a non-redundant disk in your pool, would you?  So
don''t use
a non-redundant log device either.  It''s nothing special to log
devices.
But yes, it''s true, "using a separate disk for log might give
problems if
the log device goes wrong."  Just like any other device (except cache). 
And
even this is false in later versions of zpool.  In later versions of zpool,
under most conditions, failing unmirrored log devices result in no harm, and
even in the worst conditions, result only in max 30sec data loss, not
whole-pool loss.

And the conclusion is:  (a) don''t use a dedicated log device, or (b)
use a
dedicated log device, but make sure it''s mirrored

> * A SSD drive as a log device would increase read speed over network
> based sharing protocols (NFS or SMB, which will be used at the location
> of this server)
SSD log device is used for sync writes only.  Will not affect read speed.
Perhaps you''re thinking of cache devices, which can sometimes increase
read
speeds.

> * Disk performance increased by adding SSD cache and log devices will
> die out (to a speed slower than the spindle disks) when the log and
> cache disks once have been filled up to its maximum capacity. So the
> use
> of SSD disks for cache and log is a waste of money and drive bays.
Disagree wholeheartedly.  

Even if it''s true that the SSD has to suffer the penalty of
read-modify-write (which is a questionable assertion) you''re still
comparing
3 operations of a 8,000 IOPS device, versus 1 operation on a 200 IOPS
device.  Even with this penalty (which is sometimes phantom) it''s still
an
order of magnitude performance gained.

> * Use DDR-based drives like DDRDrive (or i-RAM ?) for ZIL
Best performance this way, true.  But it''s not necessarily a solution
for
everyone.

> I am still able to reorganize the use of the SSD disks (reinstall
> OpenSolaris), and even replace any one of them with 2.5" spindle
disks,
> so any more suggestions are highly welcome.
You don''t need to reinstall osol.  Just replace those drives with some
other
drives.  And make sure you understand how to do that ... formatting, fdisk,
partition, slice, and grub.

> Best performance this way, true. But it''s not necessarily a
solution for
> everyone.
I agree.  Our product the DDRdrive X1, an NVRAM based SSD, is 
definitely not targeted towards the home end-user.  By nature of our 
component choices, fabrication tech, and ultimately price point we are a 
ZIL accelerator well matched to the 24/7 demands of enterprise use.

Thanks,

Christopher George
Founder/CTO
www.ddrdrive.com
-- 
This message posted from opensolaris.org