zfs discuss - Jul 2007 - How to take advantage of PSARC 2007/171: ZFS Separate Intent Log

PSARC 2007/171 will be available in b68. Any documentation anywhere on
how to take advantage of it?

Some of the Sun storage arrays contain NVRAM. It would be really nice
if the array NVRAM would be available for ZIL storage. It would also
be nice for extra hardware (PCI-X, PCIe card) that added NVRAM storage
to various sun low/mid-range servers that are currently acting as
ZFS/NFS servers. Or maybe someone knows of cheap SSD storage that
could be used for the ZIL? I think several HD''s are available with
SCSI/ATA interfaces.

-- 
albert chin (china at thewrittenword.com)

>PSARC 2007/171 will be available in b68. Any documentation anywhere on
>how to take advantage of it?
>
>Some of the Sun storage arrays contain NVRAM. It would be really nice
>if the array NVRAM would be available for ZIL storage. It would also
>be nice for extra hardware (PCI-X, PCIe card) that added NVRAM storage
>to various sun low/mid-range servers that are currently acting as
>ZFS/NFS servers. Or maybe someone knows of cheap SSD storage that
>could be used for the ZIL? I think several HD''s are available with
>SCSI/ATA interfaces.

Would flash memory be fast enough (current flash memory has reasonable
sequential write throughput but horrible "I/O" ops)

Casper

Albert Chin wrote:
> PSARC 2007/171 will be available in b68. Any documentation anywhere on
> how to take advantage of it?
Should be part of b68.
> Some of the Sun storage arrays contain NVRAM. It would be really nice
> if the array NVRAM would be available for ZIL storage. It would also
> be nice for extra hardware (PCI-X, PCIe card) that added NVRAM storage
> to various sun low/mid-range servers that are currently acting as
> ZFS/NFS servers. Or maybe someone knows of cheap SSD storage that
> could be used for the ZIL? I think several HD''s are available with
> SCSI/ATA interfaces.
First, you need a workload where the ZIL has an impact.

The expected improvement comes from separating the ZIL workload, which is
small, sequential iops with a high desire for low latency, from more
random workloads.  If you are using an array with NVRAM, it will hide some
of the issues which lead us to want the separation.  However, if you are
using a bunch of JBODs, then it might be worthwhile to have a device
dedicated to the ZIL.

I expect the workload characterizations to begin soon, once b68 arrives and
people get back from the holiday.  After some testing, we''ll have a
better
idea of how well it works for a variety of workloads.
  -- richard

On Tue, Jul 03, 2007 at 05:31:00PM +0200, Casper.Dik at Sun.COM
wrote:
> 
> >PSARC 2007/171 will be available in b68. Any documentation anywhere on
> >how to take advantage of it?
> >
> >Some of the Sun storage arrays contain NVRAM. It would be really nice
> >if the array NVRAM would be available for ZIL storage. It would also
> >be nice for extra hardware (PCI-X, PCIe card) that added NVRAM storage
> >to various sun low/mid-range servers that are currently acting as
> >ZFS/NFS servers. Or maybe someone knows of cheap SSD storage that
> >could be used for the ZIL? I think several HD''s are available
with
> >SCSI/ATA interfaces.
> 
> Would flash memory be fast enough (current flash memory has reasonable
> sequential write throughput but horrible "I/O" ops)
Good point. The speeds for the following don''t seem very impressive:
  http://www.adtron.com/products/A25fb-SerialATAFlashDisk.html
  http://www.sandisk.com/OEM/ProductCatalog(1321)-SanDisk_SSD_SATA_5000_25.aspx

-- 
albert chin (china at thewrittenword.com)

On Tue, Jul 03, 2007 at 09:01:50AM -0700, Richard Elling
wrote:
> Albert Chin wrote:
> > Some of the Sun storage arrays contain NVRAM. It would be really nice
> > if the array NVRAM would be available for ZIL storage. It would also
> > be nice for extra hardware (PCI-X, PCIe card) that added NVRAM storage
> > to various sun low/mid-range servers that are currently acting as
> > ZFS/NFS servers. Or maybe someone knows of cheap SSD storage that
> > could be used for the ZIL? I think several HD''s are available
with
> > SCSI/ATA interfaces.
> 
> First, you need a workload where the ZIL has an impact.
ZFS/NFS + zil_disable is faster than ZFS/NFS without zil_disable. So,
I presume, ZFS/NFS + an NVRAM-backed ZIL would be noticeably faster
than ZFS/NFS + ZIL.

-- 
albert chin (china at thewrittenword.com)

>Good point. The speeds for the following don''t seem very
impressive:
>  http://www.adtron.com/products/A25fb-SerialATAFlashDisk.html
> 
http://www.sandisk.com/OEM/ProductCatalog(1321)-SanDisk_SSD_SATA_5000_25.aspx

The adton URL leaves out IOops altogether.

Sandisks limit itself to read IOops (claims a whopping 7000).

But from what I''m told, write IOops are a tiny fraction.

Casper

Albert Chin wrote:
> On Tue, Jul 03, 2007 at 09:01:50AM -0700, Richard Elling wrote:
>> Albert Chin wrote:
>>> Some of the Sun storage arrays contain NVRAM. It would be really
nice
>>> if the array NVRAM would be available for ZIL storage. It would
also
>>> be nice for extra hardware (PCI-X, PCIe card) that added NVRAM
storage
>>> to various sun low/mid-range servers that are currently acting as
>>> ZFS/NFS servers. Or maybe someone knows of cheap SSD storage that
>>> could be used for the ZIL? I think several HD''s are
available with
>>> SCSI/ATA interfaces.
>> First, you need a workload where the ZIL has an impact.
> 
> ZFS/NFS + zil_disable is faster than ZFS/NFS without zil_disable. So,
> I presume, ZFS/NFS + an NVRAM-backed ZIL would be noticeably faster
> than ZFS/NFS + ZIL.
... for NFS workloads which are sync-sensitive.
  -- richard

Flash SSDs typically boast a huge number of _read_ IOPS (thousands), but
very few write IOPS (tens). The write throughput numbers quoted are almost
certainly for non-synchronous writes whose latency can easily be in the
milisecond range. STEC makes an interesting device which offers fast
_synchronous_ writes on an SSD, but at a pretty steep cost.

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

On Tue, Jul 03, 2007 at 10:31:28AM -0700, Richard Elling
wrote:
> Albert Chin wrote:
> > On Tue, Jul 03, 2007 at 09:01:50AM -0700, Richard Elling wrote:
> >> Albert Chin wrote:
> >>> Some of the Sun storage arrays contain NVRAM. It would be
really nice
> >>> if the array NVRAM would be available for ZIL storage. It
would also
> >>> be nice for extra hardware (PCI-X, PCIe card) that added NVRAM
storage
> >>> to various sun low/mid-range servers that are currently acting
as
> >>> ZFS/NFS servers. Or maybe someone knows of cheap SSD storage
that
> >>> could be used for the ZIL? I think several HD''s are
available with
> >>> SCSI/ATA interfaces.
> >> First, you need a workload where the ZIL has an impact.
> > 
> > ZFS/NFS + zil_disable is faster than ZFS/NFS without zil_disable. So,
> > I presume, ZFS/NFS + an NVRAM-backed ZIL would be noticeably faster
> > than ZFS/NFS + ZIL.
> 
> ... for NFS workloads which are sync-sensitive.
Well, yes. We''ve made the decision not to set zil_disable in lieu of
the possibility of the ZFS/NFS server crashing and having the clients
out of sync with what''s on the ZFS/NFS server. I think this is the
common case though for a ZFS/NFS server.

-- 
albert chin (china at thewrittenword.com)

On Tue, Jul 03, 2007 at 10:26:20AM -0500, Albert Chin
wrote:
> PSARC 2007/171 will be available in b68. Any documentation anywhere on
> how to take advantage of it?
> 
> Some of the Sun storage arrays contain NVRAM. It would be really nice
> if the array NVRAM would be available for ZIL storage. 
It depends on your array, of course, but in most arrays you can control
the amount of write cache (i.e., NVRAM) dedicated to particular LUNs.
So to use the new separate logging most effectively, you should take
your array, and dedicate all of your NVRAM to a single LUN that you then
use as your separate log device.  Your pool should then use a LUN or LUNs
that do not have any NVRAM dedicated to it.  
> It would also
> be nice for extra hardware (PCI-X, PCIe card) that added NVRAM storage
> to various sun low/mid-range servers that are currently acting as
> ZFS/NFS servers. 
You can do it yourself very easily -- check out the umem cards from Micro
Memory, available at http://www.umem.com.  Reasonable prices ($1000/GB),
they have a Solaris driver, and the performance absolutely rips.
> Or maybe someone knows of cheap SSD storage that
> could be used for the ZIL? I think several HD''s are available with
> SCSI/ATA interfaces.
As Adam mentioned, this is a bit more involved, as most SSDs are biased
very heavily towards reads and away from writes.  So this will be quite
a bit more expensive than NVRAM, at least at the moment...

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc

Adam Leventhal wrote:
> Flash SSDs typically boast a huge number of _read_ IOPS (thousands), but
> very few write IOPS (tens). The write throughput numbers quoted are almost
> certainly for non-synchronous writes whose latency can easily be in the
> milisecond range. STEC makes an interesting device which offers fast
> _synchronous_ writes on an SSD, but at a pretty steep cost.
Yes, and the size of the write iop is crucial.  Interestingly, 128 kBytes
is a good size for flash SSD writes...
  -- richard

On Tue, Jul 03, 2007 at 11:02:24AM -0700, Bryan Cantrill
wrote:
> On Tue, Jul 03, 2007 at 10:26:20AM -0500, Albert Chin wrote:
> > PSARC 2007/171 will be available in b68. Any documentation anywhere on
> > how to take advantage of it?
> > 
> > Some of the Sun storage arrays contain NVRAM. It would be really nice
> > if the array NVRAM would be available for ZIL storage. 
> 
> It depends on your array, of course, but in most arrays you can control
> the amount of write cache (i.e., NVRAM) dedicated to particular LUNs.
> So to use the new separate logging most effectively, you should take
> your array, and dedicate all of your NVRAM to a single LUN that you then
> use as your separate log device.  Your pool should then use a LUN or LUNs
> that do not have any NVRAM dedicated to it.  
Hmm, interesting. We''ll try to find out if the 6140''s can do
this.

-- 
albert chin (china at thewrittenword.com)

On Tue, Jul 03, 2007 at 01:10:25PM -0500, Albert Chin
wrote:
> On Tue, Jul 03, 2007 at 11:02:24AM -0700, Bryan Cantrill wrote:
> > On Tue, Jul 03, 2007 at 10:26:20AM -0500, Albert Chin wrote:
> > > PSARC 2007/171 will be available in b68. Any documentation
anywhere on
> > > how to take advantage of it?
> > > 
> > > Some of the Sun storage arrays contain NVRAM. It would be really
nice
> > > if the array NVRAM would be available for ZIL storage. 
> > 
> > It depends on your array, of course, but in most arrays you can
control
> > the amount of write cache (i.e., NVRAM) dedicated to particular LUNs.
> > So to use the new separate logging most effectively, you should take
> > your array, and dedicate all of your NVRAM to a single LUN that you
then
> > use as your separate log device.  Your pool should then use a LUN or
LUNs
> > that do not have any NVRAM dedicated to it.  
> 
> Hmm, interesting. We''ll try to find out if the 6140''s can
do this.
Yes, they can:  use CAM to set the write cache to be "disabled" on all
but
the LUN(s) that you want to use as the separate ZIL.

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc

On Tue, 3 Jul 2007, Casper.Dik at Sun.COM wrote:
>
>
>> Good point. The speeds for the following don''t seem very
impressive:
>>  http://www.adtron.com/products/A25fb-SerialATAFlashDisk.html
>> 
http://www.sandisk.com/OEM/ProductCatalog(1321)-SanDisk_SSD_SATA_5000_25.aspx
>
>
> The adton URL leaves out IOops altogether.
>
> Sandisks limit itself to read IOops (claims a whopping 7000).
>
> But from what I''m told, write IOops are a tiny fraction.
Agreed.  Here is a cut/paste for a buyer review in ref to a Samsung 
IDE flash memory based 32Gb drive [1]:

Reviewed By: on 6/17/2007
Rating: 5
Tech Level: high - Ownership: 1 month to 1 year

Pros: Performance tests on Linux, UDMA-4, Raw IO. 50MBps sustained 
read. 26MBps sustained write. 2 MB erase block. 3698 4K random 
reads/sec. 24 4K random writes/sec.
Cons: Wish it was SATA and cheaper.
Other Thoughts: As is typical of all Flash drives, terrible random 
write performance, so don''t expect to use this in a high-end 
Read/Write database application. You will be disappointed.

------

I can''t vouch for the accuracy of this information - so ... consider 
it as but one data point. The Sandisk products (when they ship) will 
probably offer better performance than this product, but I expect to 
see the same general operational characteristics.

[1] http://www.newegg.com/Product/Product.aspx?Item=N82E16820147015

Regards,

Al Hopper  Logical Approach Inc, Plano, TX.  al at logical-approach.com
            Voice: 972.379.2133 Fax: 972.379.2134  Timezone: US CDT
OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/

On Jul 3, 2007, at 11:26, Albert Chin wrote:
> PSARC 2007/171 will be available in b68. Any documentation anywhere on
> how to take advantage of it?
For those not in the know, PSARC 2007/171 is a separate intent log  
for ZFS:

	http://cz.opensolaris.org/os/community/arc/caselog/2007/171/

On Tue, 3 Jul 2007 10:26:20 -0500
Albert Chin <opensolaris-zfs-discuss at mlists.thewrittenword.com> wrote:
> Or maybe someone knows of cheap SSD storage that
> could be used for the ZIL?
If 4G is enough for you, take a look at Gigabyte iRam:
http://www.gigabyte.com.tw/Products/Storage/Products_Overview.aspx?ProductID=2180

Linux folks say it''s a bit quirky (it sets ''device
failure'' bit or something like that, so you have to comment out some
checking code in the kernel to have it working), but works extremly well in my
home box and in few production mail servers.

-- 

Jure Pe?ar
http://jure.pecar.org

On Wed, 4 Jul 2007, Jure Pe?~Mar wrote:

... reformatted ... 
> On Tue, 3 Jul 2007 10:26:20 -0500
> Albert Chin <opensolaris-zfs-discuss at mlists.thewrittenword.com>
wrote:
>
>> Or maybe someone knows of cheap SSD storage that
>> could be used for the ZIL?
>
> If 4G is enough for you, take a look at Gigabyte iRam:
>
http://www.gigabyte.com.tw/Products/Storage/Products_Overview.aspx?ProductID=2180
>
> Linux folks say it''s a bit quirky (it sets ''device
failure'' bit or
> something like that, so you have to comment out some checking code 
> in the kernel to have it working), but works extremly well in my 
> home box and in few production mail servers.
The Gigabyte iRam is a very "strange" product - in that, it looks to 
me like Gigabyte can''t make up its mind, whether or not it wants to 
get serious about this product (IMHO).  Let me explain:

a) it''s built on a custom gate array - if they were serious, they 
would "convert" the gate array into a ASIC (application specific IC) 
and gain lower unit cost in return for the upfront NRE (non recurring 
engineering) costs.

b) it''s not particularly fast.  On a scale of 1 (slow) to 10 (fast), 
I''d rate it a 5.  They run the RAM parts at 100MHz.  The SATA 
interface is 1.5Mb/Sec - not 3Mb/Sec.  Overall - its disappointing - 
it could have been a great product.

c) it''s hard to find/purchase.  You have to be determined to own one.

d) the DDR memory parts are limited to 1Gb/socket.  Why not 2 or 4Gb 
per socket.  Again - it "feels" like a Rev 1 product....

e) Having experience with two of these boards, I see the occasional 
glitch.  I''ve used one as a single ZFS device and run a test zone on 
it.  Occasionally I get a checksum error, which can usually be cleaned 
up by running a zfs scrub.  The zfs scrub runs *very* quickly! :)  So 
I know it''s not 100% solid - and I''d have a problem
recommending it.
OTOH they both work well enough to where I find them useful....

e1) Its a physically large board and will probably make the adjacent 
PCI slot inaccessible (unless you have a small, low-profile board in 
mind for the adajacent slot).  See next point for physical form factor 
changes.

f) Recently it was re-branded and re-packaged as the "GO-RAMDiSK-BOX":
http://www.dailytech.com/gigabyte+showcases+goramdiskbox/article7563.htm
But try and buy one of these and you''ll find it does not exist!

Personal conclusion: This product may well solve a problem for you, as 
a swap device or for building test/throw-away zones (or whatever) - or 
it may not.  If you buy one - test it thoroughly before you deploy it. 
Expect it to be fast - but probably not as fast as you would imagine.

I hope this cautionary tale helps someone before they get excited and 
rush out to buy one of these boards.

PS: the Gigabyte recommend Kingston memory parts work well and are 
reasonably priced.

Regards,

Al Hopper  Logical Approach Inc, Plano, TX.  al at logical-approach.com
            Voice: 972.379.2133 Fax: 972.379.2134  Timezone: US CDT
OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/

Bryan Cantrill wrote:
> On Tue, Jul 03, 2007 at 10:26:20AM -0500, Albert Chin wrote:
>   
>> PSARC 2007/171 will be available in b68. Any documentation anywhere on
>> how to take advantage of it?
>>
>> Some of the Sun storage arrays contain NVRAM. It would be really nice
>> if the array NVRAM would be available for ZIL storage. 
>>     
>
> It depends on your array, of course, but in most arrays you can control
> the amount of write cache (i.e., NVRAM) dedicated to particular LUNs.
> So to use the new separate logging most effectively, you should take
> your array, and dedicate all of your NVRAM to a single LUN that you then
> use as your separate log device.  Your pool should then use a LUN or LUNs
> that do not have any NVRAM dedicated to it.  
On some of the new Sun midrange arrays you can disable cache to a LUN 
but I''ve never seen hooks that let you dedicate a certain amount of 
cache to one LUN in particular. (None of the older midrange arrays let 
you do this.) Some of the high end arrays allow you to pin some data in 
cache like the 9990.

On Tue, Jul 03, 2007 at 11:02:24AM -0700, Bryan Cantrill
wrote:
> 
> On Tue, Jul 03, 2007 at 10:26:20AM -0500, Albert Chin wrote:
> > It would also be nice for extra hardware (PCI-X, PCIe card) that
> > added NVRAM storage to various sun low/mid-range servers that are
> > currently acting as ZFS/NFS servers. 
> 
> You can do it yourself very easily -- check out the umem cards from
> Micro Memory, available at http://www.umem.com.  Reasonable prices
> ($1000/GB), they have a Solaris driver, and the performance
> absolutely rips.
The PCIe card is in beta, they don''t sell to individual customers, and
the person I spoke with didn''t even know a vendor (Tier 1/2 OEMs) that
had a Solaris driver. They do have a number of PCI-X cards though.

So, I guess we''ll be testing the "dedicate all NVRAM to LUN"
solution
once b68 is released.

-- 
albert chin (china at thewrittenword.com)

On Mon, 9 Jul 2007, Albert Chin wrote:
> On Tue, Jul 03, 2007 at 11:02:24AM -0700, Bryan Cantrill wrote:
>>
>> On Tue, Jul 03, 2007 at 10:26:20AM -0500, Albert Chin wrote:
>>> It would also be nice for extra hardware (PCI-X, PCIe card) that
>>> added NVRAM storage to various sun low/mid-range servers that are
>>> currently acting as ZFS/NFS servers.
>>
>> You can do it yourself very easily -- check out the umem cards from
>> Micro Memory, available at http://www.umem.com.  Reasonable prices
>> ($1000/GB), they have a Solaris driver, and the performance
>> absolutely rips.
>
> The PCIe card is in beta, they don''t sell to individual customers,
and
> the person I spoke with didn''t even know a vendor (Tier 1/2 OEMs)
that
> had a Solaris driver. They do have a number of PCI-X cards though.
>
> So, I guess we''ll be testing the "dedicate all NVRAM to
LUN" solution
> once b68 is released.
or ramdiskadm(1M) might be interesting...

Regards,

Al Hopper  Logical Approach Inc, Plano, TX.  al at logical-approach.com
            Voice: 972.379.2133 Fax: 972.379.2134  Timezone: US CDT
OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/

On Tue, Jul 10, 2007 at 07:12:35AM -0500, Al Hopper
wrote:
> On Mon, 9 Jul 2007, Albert Chin wrote:
> 
> > On Tue, Jul 03, 2007 at 11:02:24AM -0700, Bryan Cantrill wrote:
> >>
> >> On Tue, Jul 03, 2007 at 10:26:20AM -0500, Albert Chin wrote:
> >>> It would also be nice for extra hardware (PCI-X, PCIe card)
that
> >>> added NVRAM storage to various sun low/mid-range servers that
are
> >>> currently acting as ZFS/NFS servers.
> >>
> >> You can do it yourself very easily -- check out the umem cards
from
> >> Micro Memory, available at http://www.umem.com.  Reasonable prices
> >> ($1000/GB), they have a Solaris driver, and the performance
> >> absolutely rips.
> >
> > The PCIe card is in beta, they don''t sell to individual
customers, and
> > the person I spoke with didn''t even know a vendor (Tier 1/2
OEMs) that
> > had a Solaris driver. They do have a number of PCI-X cards though.
> >
> > So, I guess we''ll be testing the "dedicate all NVRAM to
LUN" solution
> > once b68 is released.
> 
> or ramdiskadm(1M) might be interesting...
Well, that''s not really an option as a panic of the server would not
be good. While the on-disk data would be consistent, data the clients
wrote to the server might not have been committed.

-- 
albert chin (china at thewrittenword.com)

Albert Chin wrote:
> On Tue, Jul 10, 2007 at 07:12:35AM -0500, Al Hopper wrote:
>> On Mon, 9 Jul 2007, Albert Chin wrote:
>>
>>> On Tue, Jul 03, 2007 at 11:02:24AM -0700, Bryan Cantrill wrote:
>>>> On Tue, Jul 03, 2007 at 10:26:20AM -0500, Albert Chin wrote:
>>>>> It would also be nice for extra hardware (PCI-X, PCIe card)
that
>>>>> added NVRAM storage to various sun low/mid-range servers
that are
>>>>> currently acting as ZFS/NFS servers.
>>>> You can do it yourself very easily -- check out the umem cards
from
>>>> Micro Memory, available at http://www.umem.com.  Reasonable
prices
>>>> ($1000/GB), they have a Solaris driver, and the performance
>>>> absolutely rips.
>>> The PCIe card is in beta, they don''t sell to individual
customers, and
>>> the person I spoke with didn''t even know a vendor (Tier
1/2 OEMs) that
>>> had a Solaris driver. They do have a number of PCI-X cards though.
>>>
>>> So, I guess we''ll be testing the "dedicate all NVRAM
to LUN" solution
>>> once b68 is released.
>> or ramdiskadm(1M) might be interesting...
> 
> Well, that''s not really an option as a panic of the server would
not
> be good. While the on-disk data would be consistent, data the clients
> wrote to the server might not have been committed.
But if you want to quickly see _what_ advantage you may have this is a 
good option I think.

Victor