zfs discuss - Sep 2008 - ZIL & NVRAM partitioning?

I understand that if you want to use ZIL, then the requirement is one or more
ZILs per pool.
With an SSD you can partition the disk to allow usage of a single disk for
multiple ZILs.  Can we do the same thing with an PCIe-based NVRAM card (like
http://www.vmetro.com/category4304.html)?

Thanks

Narayan
--
This message posted from opensolaris.org

On 09/05/08 14:42, Narayan Venkat wrote:
> I understand that if you want to use ZIL, then the requirement is one or
more ZILs per pool.
A little clarification of ZFS terms may help here. The term ZIL is somewhat
overloaded. I think what you mean here is a separate log device (slog), because
intent
logs are always present in ZFS. Without a slog, the logs are present in the
"main pool".
There is one log per file system and it allocates blocks in the main pool to
form a chain.
When a slog is defined, then it can be made up of multiple devices (in which
case the
writes are striped across the devices) or it can be in the form on a N way
mirror - to provide redundancy.
 
> With an SSD you can partition the disk to allow usage of a single disk for
multiple ZILs
> Can we do the same thing with an PCIe-based NVRAM card
> (like http://www.vmetro.com/category4304.html)?
I don''t think there''s a Solaris supported driver for that
device.
However, any Solaris device, whether a partition or not, will work
with ZFS provided it''s at least 64MB. It''s performance is
another matter.
> 
> Thanks 
> Narayan

Thanks Neil for the clarification.  

Regards,

Narayan
--
This message posted from opensolaris.org

There is an unsupported driver available though, I posted it to the forums a few
months ago:
http://opensolaris.org/jive/thread.jspa?messageID=260281

The big problem appears to be getting your hands on these cards.  Although I
have the drivers now my first supplier let me down, and while the second insists
they have shipped the cards it''s been three weeks now and
there''s no sign of them.
--
This message posted from opensolaris.org

> The big problem appears to be getting your hands on these cards.   
> Although I have the drivers now my first supplier let me down, and  
> while the second insists they have shipped the cards it''s been
three
> weeks now and there''s no sign of them.
Thanks to Google Shopping I was able to order two of these cards from:
http://www.printsavings.com/01390371OP-discount-MICRO+MEMORY-MM5425--512MB-NVRAM-battery.aspx

They appear to be in good working order, but unfortunately I am unable  
to verify the driver.
"pkgadd -d umem_Sol_Drv_Cust_i386_v01_11.pkg" hangs on "##
Installing
part 1 of 3." on snv_95.  I do not have other Solaris versions to  
experiment with; this is really just a hobby for me.

Cheers,
Kaya Bekiro?lu

On Sat, Sep 06, 2008 at 11:16:15AM -0700, Kaya Bekiro?lu
wrote:
> > The big problem appears to be getting your hands on these cards.   
> > Although I have the drivers now my first supplier let me down, and  
> > while the second insists they have shipped the cards it''s
been three
> > weeks now and there''s no sign of them.
> 
> Thanks to Google Shopping I was able to order two of these cards from:
>
http://www.printsavings.com/01390371OP-discount-MICRO+MEMORY-MM5425--512MB-NVRAM-battery.aspx
> 
> They appear to be in good working order, but unfortunately I am unable
> to verify the driver. "pkgadd -d
umem_Sol_Drv_Cust_i386_v01_11.pkg"
> hangs on "## Installing  part 1 of 3." on snv_95.  I do not have
other
> Solaris versions to  experiment with; this is really just a hobby for
> me.
Does the card come with any programming specs to help debug the driver?

-- 
albert chin (china at thewrittenword.com)

On Sat, Sep 6, 2008 at 5:55 PM, Albert Chin <
opensolaris-zfs-discuss at mlists.thewrittenword.com> wrote:
> On Sat, Sep 06, 2008 at 11:16:15AM -0700, Kaya Bekiro?lu wrote:
> > > The big problem appears to be getting your hands on these cards.
> > > Although I have the drivers now my first supplier let me down,
and
> > > while the second insists they have shipped the cards
it''s been three
> > > weeks now and there''s no sign of them.
> >
> > Thanks to Google Shopping I was able to order two of these cards from:
> >
>
http://www.printsavings.com/01390371OP-discount-MICRO+MEMORY-MM5425--512MB-NVRAM-battery.aspx
> >
> > They appear to be in good working order, but unfortunately I am unable
> > to verify the driver. "pkgadd -d
umem_Sol_Drv_Cust_i386_v01_11.pkg"
> > hangs on "## Installing  part 1 of 3." on snv_95.  I do not
have other
> > Solaris versions to  experiment with; this is really just a hobby for
> > me.
>
> Does the card come with any programming specs to help debug the driver?
>
> --
> albert chin (china at thewrittenword.com)
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

The driver is a binary only with no support that was passed on behind the
scene''s as a favor.  I don''t know what debugging is going ot
be possible.

--Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080906/cd3eda6a/attachment.html>

On Sep 6, 2008, at 11:16 AM, Kaya Bekiro?lu wrote:
> Thanks to Google Shopping I was able to order two of these cards from:
>
http://www.printsavings.com/01390371OP-discount-MICRO+MEMORY-MM5425--512MB-NVRAM-battery.aspx
>
> They appear to be in good working order, but unfortunately I am  
> unable to verify the driver.
> "pkgadd -d umem_Sol_Drv_Cust_i386_v01_11.pkg" hangs on "##  
> Installing part 1 of 3." on snv_95.  I do not have other Solaris  
> versions to experiment with; this is really just a hobby for me.
I spoke too soon.  I was able to complete the driver installation using:

# pkgtrans umem_Sol_Drv_Cust_i386_v01_11.pkg . MMnvdisk
# pkgadd -d . MMnvdisk

At least on x86_64 snv-95, the driver works.  The block devices don''t  
show up as disks in certain utilities, but that''s not surprising.  The
card, as expected, is about 3.5x faster than the Gigabyte i-RAM and  
allows me to saturate a gigabit Ethernet link with single stream NFSv3  
TCP writes.

Well, that''s not quite true.  My NFS write throughput alternates  
between wire speed and a paltry fraction of wire speed.  I don''t  
suspect the driver to be the cause, however.  During my investigation  
I noticed that according to ''zpool iostat'' ZFS suspends use of
the
slog for new writes while the disks busy writing old data, which is  
always seven seconds after the drives finish the previous write  
batch.  This may be the ZFS flusher waking up, or it may be that that  
my slog is full; it''s hard to tell.  ''zpool iostat''
doesn''t show free
space on the slog decreasing during my test, probably because it''s not
using a proper disk driver.

This behavior might work OK when the ZIL is allocated from the same  
disks as the main pool, but is suboptimal when it resides on a fast  
slog device.  Even though NFS write throughput will be constrained to  
platter speeds once the ZIL fills up anyway, ZFS shouldn''t make  
matters worse by unnecessarily increasing NFS latency for small writes  
by bypassing the slog.

Furthermore, I suspect my observed "fast/slow" duty cycle is a  
consequence of this behavior.  If all NFS writes are sync''d to disk  
during "flush mode," then all dirty buffers have been written to disk
once the flush completes, and the disks sit idle until the next  
flush.  This wastes disk bandwidth.  By the same token, the (empty)  
slog isn''t able to accelerate NFS writes during a flush, wasting the  
same amount of bytes in network bandwidth and boosting per-write  
latency over the average.  Because I have enough slog to allow for  
seek-efficient write reordering, I should be able to get platter  
speeds if this behavior were fixed; the disks would always be writing,  
and the slog would always be active.  At least, that''s my theory,  
anyway.  I really don''t know how ZFS works.

I wonder if my problem is, in fact, that my slog is full.  I imagine  
that the ZIL can become fragmented as writes are retired out of  
order.  Inserting new writes into the holes would be tricky, but I  
wonder if simply using a circular ZIL format and initiating a flush  
when the ZIL is half full would work well enough.

Kaya Bekiro?lu

Great to hear the driver works, I''ll cross my fingers that my cards
eventually turn up.

I''m pretty sure I read something about work being done on ZFS to smooth
out those lumpy writes, I''m not 100% sure it''s the same
problem but it did sound very similar.  I can''t remember the RFE / bug
number or when it''s due out though I''m afraid.
--
This message posted from opensolaris.org