zfs discuss - Jun 2008 - zfs primarycache and secondarycache properties

Moved from PSARC to zfs-code...this discussion is seperate from the case.

Eric kustarz wrote:
>
> On Jun 23, 2008, at 1:20 PM, Darren Reed wrote:
>
>> eric kustarz wrote:
>>>
>>> On Jun 23, 2008, at 1:07 PM, Darren Reed wrote:
>>>
>>>> Tim Haley wrote:
>>>>> ....
>>>>> primarycache=all | none | metadata
>>>>>
>>>>>     Controls what is cached in the primary cache (ARC).  If
set to
>>>>>     "all", then both user data and metadata is
cached.  If set to
>>>>>     "none", then neither user data nor metadata
is cached.  If set to
>>>>>     "metadata", then only metadata is cached. 
The default
>>>>> behavior is
>>>>>     "all".
>>>>>
>>>>
>>>> The description above kind of implies that user data is somehow
>>>> separate to metadata
>>>> but it isn''t possible to say cache only user data
(with the text
>>>> given.)  Is this just an
>>>> oversight or is this really saying you cannot cache only the
user
>>>> data?
>>>
>>> We couldn''t come up with any realistic workload that would
want to
>>> cache user data but not metadata, so we''re not allowing
it.
>>>
>>> We can always add the option later, but if someone has a realistic 
>>> use case for it, i''d be happy to add it now.
>>
>> It''s not so much the "why", but maybe I''d
like to say the primarycache
>> gets metadata and the secondary cache gets user data (or vice versa.)
>> If that make sense?  Or would that require linkage between metadata
>> and user data (across cache boundaries) in order to maintain sanity?
>
> It is the "why".  If there''s no reason to do it, then we
shouldn''t
> allow it (adds more complexity, more confusion, more ways for a 
> customer to shoot themselves in the foot).
>
> However, if there is a legitimate use case, let''s discuss that.
In considering the "why", being aware of some implementation details
seems necessary, such as:
- is there a difference in size between the primary and secondary cache
- how big is the meta data relative to user data
- how many meta data items are there relative to user data

So if I''m constantly accessing just a few files, I may prefer to have
all of
the metadata cache''d by a smaller (primary?) cache and for user data to
not be able to cause that any problems (that capability is there now, ok.)

So the question becomes why does one not want the metadata in the same
cache as the user data?

So I spent some time thinking about different directions you could build
on this in the future, for example:
1) controlling the size of the ARC/L2ARC by controlling the cache size
2) specifying different backing storage for primary/secondary cache
3) having more than two levels of cache
...none of which is precluded by current efforts.

With (2), if the backing storage for each cache is different and it is 
slower
to access the secondary cache than the primary, then you may not want
metadata to be stored in the secondary cache for performance reasons.

As an example, you might be using NVRAM (be it flash or otherwise)
for the primary cache and ordinary RAM for the secondary.  In this case
you probably don''t want any metadata to be stored in the secondary
cache (power failure issues) but  the same may not hold for user data.
But I''m probably wrong about that.

Darren

Darren Reed wrote:
> So I spent some time thinking about different directions you could build
> on this in the future, for example:
> 1) controlling the size of the ARC/L2ARC by controlling the cache size
> 2) specifying different backing storage for primary/secondary cache
> 3) having more than two levels of cache
> ...none of which is precluded by current efforts.
> 
> With (2), if the backing storage for each cache is different and it is 
> slower
> to access the secondary cache than the primary, then you may not want
> metadata to be stored in the secondary cache for performance reasons.
> 
> As an example, you might be using NVRAM (be it flash or otherwise)
> for the primary cache and ordinary RAM for the secondary.  In this case
> you probably don''t want any metadata to be stored in the secondary
> cache (power failure issues) but  the same may not hold for user data.
> But I''m probably wrong about that.
I doubt you would be, the primarycache is system memory not a cache 
device.  The secondarycache is the L2ARC devices specified with the 
"cache" vdev type to zpool so your examle would be the otherway
around.

-- 
Darren J Moffat