zfs discuss - Oct 2009 - primarycache and secondarycache properties on Solaris 10 u8

Hello,

I''ve seen in the "what''s new" of Solaris 10 update 8
just released that
ZFS now includes the "primarycache" and "secondarycache"
properties.

Is this the "equivalent" of the UFS directio? Does it have a similar 
behavior?

I''m thinking about having a database on ZFS with this option, and
Oracle
recommends to have directio when working on top of a file system.

Thanks in advance and best regards,

Javi

Suggest you start with the man page

http://docs.sun.com/app/docs/doc/819-2240/zfs-1m

On 10/15/2009 4:19 PM, Javier Conde wrote:
>
> Hello,
>
> I''ve seen in the "what''s new" of Solaris 10
update 8 just released
> that ZFS now includes the "primarycache" and
"secondarycache" properties.
>
> Is this the "equivalent" of the UFS directio? Does it have a
similar
> behavior?
>
> I''m thinking about having a database on ZFS with this option, and 
> Oracle recommends to have directio when working on top of a file system.
>
> Thanks in advance and best regards,
>
> Javi
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

Other than how to turn these features on and off, only so much
performance related info can be shoe horned into a man page.

You might check out these blogs:

http://blogs.sun.com/roch/entry/people_ask_where_are_we
See the direct I/O section

http://blogs.sun.com/bobn/entry/what_s_new_in_solaris

These are just brief reviews though.

I would like to see more information about how to use these
parameters.

Cindy



On 10/15/09 14:28, Torrey McMahon wrote:
> Suggest you start with the man page
> 
> http://docs.sun.com/app/docs/doc/819-2240/zfs-1m
> 
> On 10/15/2009 4:19 PM, Javier Conde wrote:
>>
>> Hello,
>>
>> I''ve seen in the "what''s new" of Solaris 10
update 8 just released
>> that ZFS now includes the "primarycache" and
"secondarycache" properties.
>>
>> Is this the "equivalent" of the UFS directio? Does it have a
similar
>> behavior?
>>
>> I''m thinking about having a database on ZFS with this option,
and
>> Oracle recommends to have directio when working on top of a file
system.
>>
>> Thanks in advance and best regards,
>>
>> Javi
>>
>> _______________________________________________
>> zfs-discuss mailing list
>> zfs-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On Oct 15, 2009, at 1:19 PM, Javier Conde wrote:
> Hello,
>
> I''ve seen in the "what''s new" of Solaris 10
update 8 just released
> that ZFS now includes the "primarycache" and
"secondarycache"
> properties.
>
> Is this the "equivalent" of the UFS directio?
No.  UFS directio does 3 things:
	1. unbuffered I/O
	2. allow concurrent writers (no single-writer lock)
	3. provide an improved async I/O code path

For #1, the ZFS equivalent is primarycache=none, but you might find that
primarycache=metadata is more appropriate, depending on your workload.

For #2 & #3, ZFS did not have these limitations from the start.
> Does it have a similar behavior?
As you can see, this question doesn''t really make sense because the
definition of "directio" is not consistent across the many  
implementations
calling themselves "*directio*" :-)
> I''m thinking about having a database on ZFS with this option, and
> Oracle recommends to have directio when working on top of a file  
> system.
As always, YMMV. But in general, I think primarycache=metadata is a
reasonable default for Oracle workloads on ZFS file systems or volumes
when there is no L2ARC (cache) device. Tests clearly show that the
L2ARC can benefit Oracle database workloads and represent a new way
to look at the price/performance analysis for databases.
http://blogs.sun.com/fishworks/resource/oow09_ss7000_oracle.pdf

  -- richard

On Thu, 15 Oct 2009, Richard Elling wrote:
> No.  UFS directio does 3 things:
> 	1. unbuffered I/O
> 	2. allow concurrent writers (no single-writer lock)
> 	3. provide an improved async I/O code path
>
> For #1, the ZFS equivalent is primarycache=none, but you might find that
> primarycache=metadata is more appropriate, depending on your workload.
I don''t think that #1 gets you all the way there for unbuffered I/O. 
It would have to bypass most of the I/O scheduling logic for that to 
work, and bypassing the I/O scheduling logic would likely invalidate 
zfs''s ordering assurances, screw up the data layout, and lead to poor 
write performance.

The primarycache=none option seems to simply disable the data cache, 
which means that written data also does not remain in the ARC.  That 
does not mean that written data is not buffered before it is written.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

IIRC the trigger for this thread was the suggestion that
primarycache=none be set on datasets used for swap.  

Presumably swap only gets used when memory is low or
exhausted, so it would it be correct to say that it wouldn''t
make any sense for swap to be in /any/ cache? If this isn''t
what primarycache=none means, shouldn''t there be a
disable-cache-entirely flag for datasets used for swap?

I guess reads from swap must be buffered somewhere, so
it would be an optimization to have such reads buffered
in a read cache. But wouldn''t the read cache be real small
at this point? It''s enough to make your head spin :-)

-- Frank

On Thu, 15 Oct 2009, Frank Middleton wrote:
> IIRC the trigger for this thread was the suggestion that
> primarycache=none be set on datasets used for swap. 
> Presumably swap only gets used when memory is low or
> exhausted, so it would it be correct to say that it wouldn''t
> make any sense for swap to be in /any/ cache? If this isn''t
The MMU page cache (memory) is the common interface to "swap" so swap 
is often cached.  It does not make sense to cache it twice though.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Oct 15, 2009, at 3:13 PM, Bob Friesenhahn wrote:
> On Thu, 15 Oct 2009, Richard Elling wrote:
>> No.  UFS directio does 3 things:
>> 	1. unbuffered I/O
>> 	2. allow concurrent writers (no single-writer lock)
>> 	3. provide an improved async I/O code path
>>
>> For #1, the ZFS equivalent is primarycache=none, but you might find  
>> that
>> primarycache=metadata is more appropriate, depending on your  
>> workload.
>
> I don''t think that #1 gets you all the way there for unbuffered
I/O.
> It would have to bypass most of the I/O scheduling logic for that to  
> work, and bypassing the I/O scheduling logic would likely invalidate  
> zfs''s ordering assurances, screw up the data layout, and lead to  
> poor write performance.
Another overloaded term?  Oracle uses O_DSYNC, but with directio for  
UFS you
basically get that, too.  In other words, UFS directio is not a  
solution for general
purpose workloads. I think primarycache=none will garner similar  
disfavor as it
effectively makes all writes sync and disables prefetching.

Historically, buffered I/O is a bigger problem when the amount of memory
for buffering is limited. Consider that UFS directio arrived back in  
the day when
the average server held perhaps hundreds of MB, every byte mattered.   
Today
RAM is not so constrained, though you do need to be wary of large page  
exhaustion
for Oracle.
  -- richard
> The primarycache=none option seems to simply disable the data cache,  
> which means that written data also does not remain in the ARC.  That  
> does not mean that written data is not buffered before it is written.
>
> Bob
> --
> Bob Friesenhahn
> bfriesen at simple.dallas.tx.us,
http://www.simplesystems.org/users/bfriesen/
> GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

Richard Elling wrote:
> On Oct 15, 2009, at 1:19 PM, Javier Conde wrote:
>> Hello,
>>
>> I''ve seen in the "what''s new" of Solaris 10
update 8 just released
>> that ZFS now includes the "primarycache" and
"secondarycache"
>> properties.
>>
>> Is this the "equivalent" of the UFS directio?
>
> No.  UFS directio does 3 things:
>     1. unbuffered I/O
>     2. allow concurrent writers (no single-writer lock)
>     3. provide an improved async I/O code path
>
for the record - iirc UFS will also disable read-aheads with directio.

-- 
Robert Milkowski
http://milek.blogspot.com