zfs discuss - Jan 2009 - Add SSD drive as L2ARC(?) cache to existing ZFS raid?

I understand Fishworks has a L2ARC cache, which as I have understood it, is a
SSD drive as a cache?

I have 5 terabyte discs in a raidz1. Could I add one SSD drive in a similar
vein? Would it be easy to do? What would be the impact? Has anyone tried this?
-- 
This message posted from opensolaris.org

Orvar Korvar wrote:
> I understand Fishworks has a L2ARC cache, which as I have understood it, is
a SSD drive as a cache?
>   
Fishworks is an engineering team, I hear they have many L2ARCs in
their lab :-)  Yes, the Sun Storage 7000 series systems can have
read-optimized SSDs for use as L2ARC devices.
> I have 5 terabyte discs in a raidz1. Could I add one SSD drive in a similar
vein?
Yes. You should look for a read-optimized SSD, for best performance
gains.
> Would it be easy to do? 
Yes.
> What would be the impact? 
It depends on the workload.  In general, it helps workloads which
do random reads, perhaps by 10x or more.
> Has anyone tried this?
>   
Yes.
 -- richard

On Wed, 28 Jan 2009, Richard Elling wrote:
> Orvar Korvar wrote:
>
>> I have 5 terabyte discs in a raidz1. Could I add one SSD drive in a 
>> similar vein? Would it be easy to do?
>
> Yes.
>
To be specific, you use the ''cache'' argument to zpool, as in:

 	zpool create <pool> <...> cache <cache-device>


Regards,
markm

Orvar,

In an existing RAIDZ configuration, you would add the cache device like
this:

# zpool add pool-name cache device-name

Currently, cache devices are only supported in the OpenSolaris and SXCE
releases.

The important thing is determining whether the cache device would
improve your workload''s performance, following Richard''s
advice.

Can you try and buy an SSD? :-)

Cindy


Mark J Musante wrote:
> On Wed, 28 Jan 2009, Richard Elling wrote:
> 
> 
>>Orvar Korvar wrote:
>>
>>
>>>I have 5 terabyte discs in a raidz1. Could I add one SSD drive in a 
>>>similar vein? Would it be easy to do?
>>
>>Yes.
>>
> 
> 
> To be specific, you use the ''cache'' argument to zpool, as
in:
> 
>  	zpool create <pool> <...> cache <cache-device>
> 
> 
> Regards,
> markm
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Thanx for your answers guys. :o)  

I dont contemplating trying this for my ZFS raid, as the SSD drives are
expensive right now. I just want to be able to answer questions when I convert
Windows/Linux to Solaris. And therefore collect info. Has anyone tried this on a
blogg? Would be cool to blog about. "Before" and "After".

BTW, did I tell you that Solaris rocks? :o)
-- 
This message posted from opensolaris.org

Imagine 10 SATA discs in raidz2 and one or two SSD drives as a cache. Each Vista
client reaches ~90MB/sec to the server, using Solaris CIFS and iSCSI. So you
want to use iSCSI with this. (iSCSI allows ZFS to export a file system as a
native SCSI disc to a desktop PC. The desktop PC can mount this iSCSI disk as a
native SCSI disk and format it with NTFS - on top of ZFS with snapshots, etc.
This is done from desktop PC bios).

Now, you install WinXP on the iSCSI ZFS volume, and clone it with a snapshot.
Then you can boot from the clone, with the iSCSI volume on a desktop PC. Thus,
your desktop PC doesnt need any hard drive at all. It uses the iSCSI volume on
the ZFS server as a native SCSI disk, which has WinXP installed.

This way, you can deploy lots of desktop PC in an instant. Using the cloned
WinXP snapshot. And if there are problems e.g virus, just destroy the clone and
create a new one in one second.




Has anyone does this? Does the SSD provide extra speed? Any stories to share?
(Ive read this iSCSI suggestion on a blogg with black background color,
it''s not my idea).

If I add a SSD disk as a cache, can I remove it? No? Will there be problems if I
remove it? Can I exchange it with a bigger? (Trying to convert these Windows
people)
-- 
This message posted from opensolaris.org

Kebabber,

You can''t expose zfs filesystems over iSCSI.

You only can expose ZFS volumes (raw volumes) over iscsi.

PS: 2 weeks ago I did a few tests, using filebench.

I saw little to no improvement using a 32GB Intel X25E SSD.

Maybe this is because filebench is flushing the cache in between tests.

I also compared iscsi boot time (using gpxe as boot loader) ,

We are using raidz storagepool (4disks). here again, adding the X25E as cache
device did not speedup the boot proccess. So I did not see real improvement.

PS: We have 2 master volumes (xp and vista) which we clone to provision
additional guests.

I''m now waiting for new SSD disks (STEC Zeus 18GB en STEC Mach 100GB.),
since those are used in SUN 7000 product. I hope they perform better.

Kristof
-- 
This message posted from opensolaris.org

How were you running this test?

were you running it locally on the machine, or were you running it over 
something like NFS?

What is the rest of your storage like? just direct-attached (SAS or 
SATA, for example) disks, or are you using a higher-end RAID controller?

-Greg

kristof wrote:
> Kebabber,
> 
> You can''t expose zfs filesystems over iSCSI.
> 
> You only can expose ZFS volumes (raw volumes) over iscsi.
> 
> PS: 2 weeks ago I did a few tests, using filebench.
> 
> I saw little to no improvement using a 32GB Intel X25E SSD.
> 
> Maybe this is because filebench is flushing the cache in between tests.
> 
> I also compared iscsi boot time (using gpxe as boot loader) ,
> 
> We are using raidz storagepool (4disks). here again, adding the X25E as
cache device did not speedup the boot proccess. So I did not see real
improvement.
> 
> PS: We have 2 master volumes (xp and vista) which we clone to provision
additional guests.
> 
> I''m now waiting for new SSD disks (STEC Zeus 18GB en STEC Mach
100GB.), since those are used in SUN 7000 product. I hope they perform better.
> 
> Kristof

So are there no guide lines how to add a SSD disk as a home user? Which is the
best SSD disk to add? What percentage improvements are typical? Or, will a home
user not benefit from adding a SSD drive? It is only enterprise SSD drives that
works, together with some esoteric software from Fishworks? It requires
Enterprise hardware to get a boost from SSD? Not possible? Or?

No one has done this yet? What does the Fishworks team say?
-- 
This message posted from opensolaris.org

Orvar,

With my testing, i''ve seen a 5x improvement with small file creation 
when working specifically with NFS. This is after I added an SSD for the 
ZIL.

I recommend Richard Elling''s zilstat (he posted links earlier).
It''ll
let you see if a dedicated device for the ZIL will help your specific 
workload.

My understanding is that you''ll get "more bang for the buck"
using an
SSD for the ZIL rather than the L2ARC. Performing some of your own 
benchmarks is really the only way see what will help improve performance 
for your specific workload. I recommend reading up on the ZFS ARC and 
L2ARC, to help try to determine if testing a dedicated L2ARC device is 
even worthwhile for your uses. I know it wasn''t really helpful for me, 
as our read performance is already great.

As for a specific SSD, I''ve tested the Intel X25E. It''s around
$600 or
so. It''s got about half the performance of the snazzy, pricey STEC Zeus
drives. With the specific workload I was trying to accelerate, I wasn''t
hitting any of the limits of the Intel SSDs (but I was definitely WAY 
past the performance limits of a standard hard disk). Again, all of this 
was for accelerating the ZIL, not for use on the L2ARC, so YMMV.

Fishworks does this. They use an SSD both for the read cache as well as 
the ZIL.

-Greg

Orvar Korvar wrote:
> So are there no guide lines how to add a SSD disk as a home user? Which is
the best SSD disk to add? What percentage improvements are typical? Or, will a
home user not benefit from adding a SSD drive? It is only enterprise SSD drives
that works, together with some esoteric software from Fishworks? It requires
Enterprise hardware to get a boost from SSD? Not possible? Or?
> 
> No one has done this yet? What does the Fishworks team say?

Orvar Korvar wrote:
> So are there no guide lines how to add a SSD disk as a home user? 
I''m not sure any guidelines are needed beyond that which is already
documented in the zpool man page or the ZFS Administration Guide.
What do you believe is missing from those docs?
> Which is the best SSD disk to add? 
The one that somebody else pays for :-)
> What percentage improvements are typical? Or, will a home user not benefit
from adding a SSD drive?
YMMV.  I do not believe there will be a way to provide a generic answer
to this question.
> It is only enterprise SSD drives that works, together with some esoteric
software from Fishworks?
The Sun Storage 7000 systems use ZFS to manage the storage pools.
> It requires Enterprise hardware to get a boost from SSD? Not possible? Or?
>
> No one has done this yet? 
Many people have experimented with the separate log features using a
wide variety of devices.
 -- richard
> What does the Fishworks team say?
>

On Tue, February 3, 2009 15:26, Richard Elling wrote:
> Orvar Korvar wrote:
>> Which is the best SSD disk to add?
>
> The one that somebody else pays for :-)
Does the access pattern for say the ZIL tickle the "stuttering" bug in
the
JMicron-controller multi-level-cell SSDs?  That is, in all the cheap ones?
 Somehow early reviews managed to miss it, so I''m stuck with one, and
will
eventually replace it with something faster -- and then will be looking
for a way to use it productively.  Or has all the testing centered around
enterprise-grade stuff and nobody knows about the cheaper drives''
performance?

(There''s a problem in the MLC SSD drives with particular JMicron
controller chips where some patterns of small writes trigger ~~10sec.
hangs, which is most often called "stuttering" in online discussions
of it
that I''ve seen. Using it as a Windows system disk unfortunately hits
these
patterns far too frequently.  My particular example is an OCZ Core 2 64GB
unit. Oh; I should say I''ve been through the list of registry tweaks,
and
I can''t see they''ve helped more than a little.)

-- 
David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

Is it possible for someone to put up a wiki page somewhere with the various
SSD, ZIL, L2ARC options with Pros, Cons and Benchmarks.
Especially with notes like the below.

Given this is a key area of interest for zfs at the moment, seems like it
would be a useful resource.

On Wed, Feb 4, 2009 at 11:01 AM, David Dyer-Bennet <dd-b at dd-b.net>
wrote:
>
> On Tue, February 3, 2009 15:26, Richard Elling wrote:
> > Orvar Korvar wrote:
>
> >> Which is the best SSD disk to add?
> >
> > The one that somebody else pays for :-)
>
> Does the access pattern for say the ZIL tickle the "stuttering"
bug in the
> JMicron-controller multi-level-cell SSDs?  That is, in all the cheap ones?
>  Somehow early reviews managed to miss it, so I''m stuck with one,
and will
> eventually replace it with something faster -- and then will be looking
> for a way to use it productively.  Or has all the testing centered around
> enterprise-grade stuff and nobody knows about the cheaper drives''
> performance?
>
> (There''s a problem in the MLC SSD drives with particular JMicron
> controller chips where some patterns of small writes trigger ~~10sec.
> hangs, which is most often called "stuttering" in online
discussions of it
> that I''ve seen. Using it as a Windows system disk unfortunately
hits these
> patterns far too frequently.  My particular example is an OCZ Core 2 64GB
> unit. Oh; I should say I''ve been through the list of registry
tweaks, and
> I can''t see they''ve helped more than a little.)
>
> --
> David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/
> Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
> Photos: http://dd-b.net/photography/gallery/
> Dragaera: http://dragaera.info
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090204/5887b738/attachment.html>

Not sure is best to put something like this.

There is wikis like
http://www.solarisinternals.com/wiki/index.php/Solaris_Internals_and_Performance_FAQ
http://wiki.genunix.org/wiki/index.php/WhiteBox_ZFSStorageServer

But I haven''t seen anything which has an active community like
http://www.thinkwiki.org/wiki/ThinkWiki
Is it worthwhile to setup a something like this?

Adding together information like:
http://jkontherun.com/2009/02/04/new-192gb-ssd-from-transcend-runs-at-high-speed/


I think would be very useful.



On Thu, Feb 5, 2009 at 6:30 AM, Miles Nordin <carton at ivy.net> wrote:
> In case you want to start one, here are the (incomplete) notes
I''ve
> kept from the list:
>
> http://www.vmetro.com/category4304.html -- ZFS log device, prestoserv of
> the future.  also Gigabyte iRAM.
>  there are some companies, Crucial and STEC come to mind,
>  sell SSDs which fit in disk form factors.  IIRC, Mac Book Air and EMC
>  use STEC''s  SSDs.  -- Richard Elling
>  the ones that Mtron sells were testing as the fastest ones on the market.
>  -- Brian Hechinger 2008-06-27
>  http://www.xbitlabs.com/articles/storage/display/ssd-iram_5.html
>  http://www.stec-inc.com/product/zeusiops.php
>  http://www.fusionio.com/
>
>
http://forums.storagereview.net/index.php?s=&showtopic=27190&view=findpost&p=253758
>  http://www.hyperossystems.co.uk/
> - Ross
>  http://www.opensolaris.org/jive/thread.jspa?threadID=65074&tstart=30
> - relling
>
>
http://www.marketwatch.com/news/story/STEC-Support-Suns-Unified-Storage/story.aspx?guid=%7B07043E00-7628-411D-B24A-2FFEC8B8F706%7D
>    The ZEUS product line makes a fine slog while the MACH8 product line
> works nicely for L2ARC. [from the Sun 7000]
> - Greg Mason
>  Intel X25E. It''s around $600 It''s got about half the
performance of STEC
> Zeus drives
> - David Dyer-Bennet
>   There''s a problem in the MLC SSD drives with particular JMicron
> controller chips where some patterns of small writes trigger ~~10sec.
> hangs, which is most often called "stuttering" in online
discussions of it
> that I''ve seen.  My particular example is an OCZ Core 2 64GB
> unit.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090205/b87aed00/attachment.html>

I''ll second that. A wiki page on the ZFS wiki, with best practices and
recommendations about adding SSD would be great. There is not much information
on this subject, I feel. Case scenarios, blogs, etc showing some numbers.
-- 
This message posted from opensolaris.org

Orvar Korvar wrote:
> I''ll second that. A wiki page on the ZFS wiki, with best practices
and recommendations about adding SSD would be great. There is not much
information on this subject, I feel. Case scenarios, blogs, etc showing some
numbers.
>   
Seriously, is it so complicated that a best practice page is needed? 

NB Best practices tend to be neither time nor version-specific, because
versions change over time.  For version-specific things, release notes
are more appropriate.
 -- richard

On Fri, Feb 6, 2009 at 11:29 AM, Richard Elling <richard.elling at
gmail.com>
 wrote:
>
> Seriously, is it so complicated that a best practice page is needed?

While you might be right about that, I think there is a need for a good
shared experiences site, howtos, etc.
For example, I want to put a new 2U 12 disk storage system together. 
I''ll
use a Supermicro chasis, but I wonder what is best for:

   - AMD or Intel?
   - Is the AOC-SAT2-MV8 still the best SATA controller or should I get a
   LSI card.
   - The AOC-USASLP-S8iR has battery back cache option.  Will this work with
   Solaris? Which release? Can it do JBOD? Will the battery back cache actually
   be useful?
   - Then of course there is all the SSD options. Which SSD units are better
   for write/ZIL, which is better for read/L2ARC?


All these questions are pretty general, and being asked often.  I think it
would be good to find some way to consolidate this stuff.

Nicholas
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090206/37c8a9d6/attachment.html>

Nicholas Lee wrote:
>
> On Fri, Feb 6, 2009 at 11:29 AM, Richard 
> Elling <richard.elling at gmail.com <mailto:richard.elling at
gmail.com>> wrote:
>
>
>     Seriously, is it so complicated that a best practice page is needed?
>
>  
> While you might be right about that, I think there is a need for a 
> good shared experiences site, howtos, etc.
>
> For example, I want to put a new 2U 12 disk storage system together. 
>  I''ll use a Supermicro chasis, but I wonder what is best for:
>
>     * AMD or Intel?
>     * Is the AOC-SAT2-MV8 still the best SATA controller or should I
>       get a LSI card.
>     * The AOC-USASLP-S8iR has battery back cache option.  Will this
>       work with Solaris? Which release? Can it do JBOD? Will the
>       battery back cache actually be useful?
>     * Then of course there is all the SSD options. Which SSD units are
>       better for write/ZIL, which is better for read/L2ARC?
>
>
> All these questions are pretty general, and being asked often.  I 
> think it would be good to find some way to consolidate this stuff.
Yes, but this becomes yet another HCL -- not best practices.  The
info will be outdated in weeks or even days.  Does the current HCL suit? 
If not, how would  you propose a better solution?
 -- richard

Is there an issue with having additional resource that at support each
other?
If information is well documented, then it will be easy to tell if it is out
of date.

Regardless does the current HCL answer the questions I posed?



On Fri, Feb 6, 2009 at 2:26 PM, Richard Elling <richard.elling at
gmail.com>wrote:
> Nicholas Lee wrote:
>
>
>> On Fri, Feb 6, 2009 at 11:29 AM, Richard Elling <richard.elling at
gmail.com<mailto:
>> richard.elling at gmail.com>> wrote:
>>
>>
>>    Seriously, is it so complicated that a best practice page is needed?
>>
>>  While you might be right about that, I think there is a need for a
good
>> shared experiences site, howtos, etc.
>>
>> For example, I want to put a new 2U 12 disk storage system together. 
I''ll
>> use a Supermicro chasis, but I wonder what is best for:
>>
>>    * AMD or Intel?
>>    * Is the AOC-SAT2-MV8 still the best SATA controller or should I
>>      get a LSI card.
>>    * The AOC-USASLP-S8iR has battery back cache option.  Will this
>>      work with Solaris? Which release? Can it do JBOD? Will the
>>      battery back cache actually be useful?
>>    * Then of course there is all the SSD options. Which SSD units are
>>      better for write/ZIL, which is better for read/L2ARC?
>>
>>
>> All these questions are pretty general, and being asked often.  I think
it
>> would be good to find some way to consolidate this stuff.
>>
>
> Yes, but this becomes yet another HCL -- not best practices.  The
> info will be outdated in weeks or even days.  Does the current HCL suit? If
> not, how would  you propose a better solution?
> -- richard
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090206/4e0d635e/attachment.html>