zfs discuss - Jan 2010 - ZFS default compression and file size limit?

I have a 13GB text file.  I turned ZFS compression on with "zfs set
compression=on mypool".  When i copy the 13GB file into another file, it
does not get compressed (checking via du -sh).  However if i set
compression=gzip, then the file gets compressed.

Is there a limit on file size with the default compression algorithm?  I
did experiment with a much smaller file of 0.5GB with the default
compression and it did get compressed.

I am using S10 U8.

Regards,


--
Wajih Ahmed
Principal Field Technologist
877.274.6589 / x40572
Skype: wajih_ahmed

I have a 13GB text file.  I turned ZFS compression on with "zfs set
compression=on mypool".  When i copy the 13GB file into another file, it
does not get compressed (checking via du -sh).  However if i set
compression=gzip, then the file gets compressed.

Is there a limit on file size with the default compression algorithm?  I
did experiment with a much smaller file of 0.5GB with the default
compression and it did get compressed.

I am using S10 U8 x86/64.

Regards,


--
Wajih Ahmed
Principal Field Technologist
877.274.6589 / x40572
Skype: wajih_ahmed

On 20/01/2010 13:39, Wajih Ahmed wrote:
> I have a 13GB text file.  I turned ZFS compression on with "zfs set
> compression=on mypool".  When i copy the 13GB file into another file,
it
> does not get compressed (checking via du -sh).  However if i set
> compression=gzip, then the file gets compressed.
>
> Is there a limit on file size with the default compression algorithm?  I
> did experiment with a much smaller file of 0.5GB with the default
> compression and it did get compressed.
>
if a given block is not gaining more than 12.5% from a compression then 
it will not be stored as compressed.
It might be that with a default compression algorithm (lzjb) you are 
gaining less than 12.5% while when using gzip you are getting more 
therefore blocks end up being compressed.

-- 
Robert Milkowski
http://milek.blogspot.com

Mike,

Thank you for your quick response...

Is there a way for me to test the compression from the command line to 
see if lzjb is giving me more or less than the 12.5% mark?  I guess it 
will depend if there is a lzjb command line utility.

I am just a little surprised because gzip-6 is able to compress it to 
4.4GB from 14GB (and gzip-1 4.8GB) and from what i read lzjb should be 
giving me better an 12.5% compression.  For example the *compress* 
command (which i think uses LZO, a slight different variant of 
Lempel-Ziv) manges to reduce it to 8.0GB.  That is a 57% ratio. 

Regards,

--
Wajih Ahmed
Principal Field Technologist
877.274.6589 / x40572
Skype: wajih_ahmed



Robert Milkowski wrote:
> On 20/01/2010 13:39, Wajih Ahmed wrote:
>> I have a 13GB text file.  I turned ZFS compression on with "zfs
set
>> compression=on mypool".  When i copy the 13GB file into another
file, it
>> does not get compressed (checking via du -sh).  However if i set
>> compression=gzip, then the file gets compressed.
>>
>> Is there a limit on file size with the default compression algorithm? 
I
>> did experiment with a much smaller file of 0.5GB with the default
>> compression and it did get compressed.
>>
>
> if a given block is not gaining more than 12.5% from a compression 
> then it will not be stored as compressed.
> It might be that with a default compression algorithm (lzjb) you are 
> gaining less than 12.5% while when using gzip you are getting more 
> therefore blocks end up being compressed.
>

On Wed, Jan 20, 2010 at 12:42:35PM -0500, Wajih Ahmed
wrote:
> Mike,
>
> Thank you for your quick response...
>
> Is there a way for me to test the compression from the command line to  
> see if lzjb is giving me more or less than the 12.5% mark?  I guess it  
> will depend if there is a lzjb command line utility.
>
> I am just a little surprised because gzip-6 is able to compress it to  
> 4.4GB from 14GB (and gzip-1 4.8GB) and from what i read lzjb should be  
> giving me better an 12.5% compression.  For example the *compress*  
> command (which i think uses LZO, a slight different variant of  
> Lempel-Ziv) manges to reduce it to 8.0GB.  That is a 57% ratio. 
That''s over the whole file as a single compression stream.  ZFS has to
compress each block (128k or maybe less) independently.  This can''t do
as well.

--
Dan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100121/812ca5cf/attachment.bin>