zfs discuss - May 2006 - ZFS compression API (Was Re: [osol-discuss] Re: Re: where to start?)

Casper.Dik at sun.com wrote:
>> Robert Milkowski wrote:
>>
>>> But only if compression is turned on for a filesystem.
>> Of course, and the default is off.
>>
>>> However I think it would be good to have an API so application can
>>> decide what to compress and what not.
>> I agree that an API would be good.  However I don''t think
using the API
>> should allow an application to write compressed data if the file system
>> has that functionality turned off.  Its a policy thing if the admin has
>> compression off it is off for a reason. Or maybe what we need is
another
>> property value for compression that allows the app to request it but by
>> default we don''t do it.
> 
> *Only* if we fail to come up with a mechanism to do this properly,
> efficiently automagically.
Which bit did you mean ? The API itself or the policy part  ?

-- 
Darren J Moffat

>Casper.Dik at sun.com wrote:
>>> Robert Milkowski wrote:
>>>
>>>> But only if compression is turned on for a filesystem.
>>> Of course, and the default is off.
>>>
>>>> However I think it would be good to have an API so application
can
>>>> decide what to compress and what not.
>>> I agree that an API would be good.  However I don''t think
using the API
>>> should allow an application to write compressed data if the file
system
>>> has that functionality turned off.  Its a policy thing if the admin
has
>>> compression off it is off for a reason. Or maybe what we need is
another
>>> property value for compression that allows the app to request it
but by
>>> default we don''t do it.
>> 
>> *Only* if we fail to come up with a mechanism to do this properly,
>> efficiently automagically.
>
>Which bit did you mean ? The API itself or the policy part  ?
The API (and therefor also the policy).

If we can make it work without, then that is much better.

Casper

Hello Darren,

Tuesday, May 23, 2006, 11:12:15 AM, you wrote:

DJM> Casper.Dik at sun.com wrote:
>>> Robert Milkowski wrote:
>>>
>>>> But only if compression is turned on for a filesystem.
>>> Of course, and the default is off.
>>>
>>>> However I think it would be good to have an API so application
can
>>>> decide what to compress and what not.
>>> I agree that an API would be good.  However I don''t think
using the API
>>> should allow an application to write compressed data if the file
system
>>> has that functionality turned off.  Its a policy thing if the admin
has
>>> compression off it is off for a reason. Or maybe what we need is
another
>>> property value for compression that allows the app to request it
but by
>>> default we don''t do it.
>> 
>> *Only* if we fail to come up with a mechanism to do this properly,
>> efficiently automagically.
It''s not a good idea IMHO.
The problem is that with stronger compression algorithms due to
performance reasons I want to decide which algorithms and to what
files ZFS should try to compress. For some files I write a lot of data
even if they are compressing quite good i don''t want it - too much CPU
would be consumed. However for other files in the same data pool I
know we do not write them that often so compression for them would be
useful (and cheap in terms of CPU).


-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

Robert Milkowski wrote:
> The problem is that with stronger compression algorithms due to
> performance reasons I want to decide which algorithms and to what
> files ZFS should try to compress. For some files I write a lot of data
> even if they are compressing quite good i don''t want it - too much
CPU
> would be consumed. However for other files in the same data pool I
> know we do not write them that often so compression for them would be
> useful (and cheap in terms of CPU).
This is exactly the reason I don''t want the applications running as a 
normal user to be able to turn on more expensive compression algorithms 
if the administrator of the data set has explicitly not turned on 
compression.

I think there are three possible modes here:

1) No compression ever.
2) Current behaviour
3) New: Best effort from ZFS (ie current behaviour) with hints from 
application on the data eg, don''t bother trying this is an MP3.


-- 
Darren J Moffat

Hello Darren,

Tuesday, May 23, 2006, 4:19:05 PM, you wrote:

DJM> Robert Milkowski wrote:
>> The problem is that with stronger compression algorithms due to
>> performance reasons I want to decide which algorithms and to what
>> files ZFS should try to compress. For some files I write a lot of data
>> even if they are compressing quite good i don''t want it - too
much CPU
>> would be consumed. However for other files in the same data pool I
>> know we do not write them that often so compression for them would be
>> useful (and cheap in terms of CPU).
DJM> This is exactly the reason I don''t want the applications
running as a
DJM> normal user to be able to turn on more expensive compression algorithms
DJM> if the administrator of the data set has explicitly not turned on 
DJM> compression.

DJM> I think there are three possible modes here:

DJM> 1) No compression ever.
DJM> 2) Current behaviour
DJM> 3) New: Best effort from ZFS (ie current behaviour) with hints from 
DJM> application on the data eg, don''t bother trying this is an MP3.

Definitely that''s something I had in mind.
To be more precise case #3 should be something like either ZFS tries
to determine itself which compression to use for each file (block?) or
administrator can setup default compression for a dataset. However in
both these cases application is able either hint ZFS or even force
specific compression algorithm for a file (block? given write? what
about mmap()?).

In some cases algorithm per block (write()) would be preferred - like
if our application writes an email and it already knows what kind of
attachments there are so it doesn''t make sense to even try to compress
this block (however rest of the mail could be compressed). Something
like this:

    ioctl(fd, strong compression)
    write(fd, buf, s1) - headers and body
    ioctl(fd, no compression)
    write(fd, buf, s1) - jpeg attachement

However writev() or sendfilev() would be preferred - I''m not sure how
to cope with that situation.

To make API more universal applications should have an option not only
to request specific compression but to request some kind of
compression level without actually specifying algorithm - like I want
light compression or I don''t care about CPU and want heavy compression
(or something in a middle).

ps. and of course if admin chooses option #1 application can''t force
ZFS to use compression. And in #2 only compression selected by sys
admin is used.

-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com