Hello !
I searched the net and the forum for this, but couldn`t find anything about
this.
can someone tell, how effective is zfs compression and space-efficiency
(regarding small files) ?
since compression works at the block level, i assume compression may not come
into effect as some may expect. (maybe i`m wrong here)
can someone point me to some comparison table if it exists, or could someone be
so nice to post some real world data? i have no solaris around to play/test for
now.
maybe compare it like this:
example data:
linux-kernel source tree
big oracle data file (/oradata directory)
contents of /usr filesystem
compare:
1. uncompressed as single datastream - e.g. .tar
2. compressed with gz
3. uncompressed effective size on zfs fs (df -k before/after storing files)
4. compressed effective size on zfs fs ( don`t know how to determine)
4. uncompressed effective size on other fs (e.g. reiser3 cause storing small
files efficiently)
5. ???
TIA
roland k.
sysadmin
This message posted from opensolaris.org
On Tue, Aug 22, 2006 at 08:43:32AM -0700, roland wrote:> can someone tell, how effective is zfs compression and > space-efficiency (regarding small files) ? > > since compression works at the block level, i assume compression may > not come into effect as some may expect. (maybe i`m wrong here)It''s true that since we are compressing a block at a time, there are some efficiencies of whole-large-file compression that will be lost. However, since ZFS uses 128k blocks on large files, the difference should be neglegable. For smaller files, ZFS uses a single block that exactly fits the file (compressed or not) (rounded up to the nearest sector size (512 bytes)). So I believe that ZFS''s compression infrastructure permits good efficiency. However, at this point, we have only implemented one compression algorithm, which is much faster than, but does not compress as much as, gzip. We plan to implement a broader range of compression algorithms in the future. --matt
>-----Original Message----- >From: zfs-discuss-bounces at opensolaris.org[mailto:zfs-discuss->bounces at opensolaris.org] On Behalf Of roland>can someone tell, how effective is zfs compression and space-efficiency >(regarding small files) ?>linux-kernel source tree% ls -l linux-2.6.17.tar.gz -rw------- 1 nliu staff 51700445 Aug 22 12:05 linux-2.6.17.tar.gz % ls -l /export/home/zfstest total 8 drwxrwxrwx 19 root root 27 Jun 17 21:49 linux-2.6.17 % zfs get all export/home/zfstest NAME PROPERTY VALUE SOURCE export/home/zfstest type filesystem - export/home/zfstest creation Tue Aug 22 12:05 2006 - export/home/zfstest used 181M - export/home/zfstest available 530G - export/home/zfstest referenced 181M - export/home/zfstest compressratio 1.80x ->big oracle data file (/oradata directory)We use Intersystem''s Cache database. IO performance during copy of noncompress mirror to compress raidz zfs pool capacity operations bandwidth pool used avail read write read write ----------- ----- ----- ----- ----- ----- ----- data 69.7G 338G 6.98K 0 55.8M 0 mirror 23.2G 113G 2.35K 0 18.8M 0 c1t2d0 - - 207 0 18.9M 0 c2t3d0 - - 213 0 18.8M 0 mirror 23.3G 113G 2.34K 0 18.7M 0 c1t4d0 - - 215 0 18.9M 0 c2t5d0 - - 215 0 18.9M 0 mirror 23.2G 113G 2.28K 0 18.2M 0 c1t8d0 - - 212 0 18.5M 0 c2t9d0 - - 211 0 18.3M 0 ----------- ----- ----- ----- ----- ----- ----- export 6.39G 538G 0 498 0 21.2M raidz 6.39G 538G 0 498 0 21.2M c1t10d0 - - 0 398 0 7.11M c2t11d0 - - 0 394 0 7.11M c1t12d0 - - 0 405 0 7.11M c2t13d0 - - 0 378 0 7.03M ----------- ----- ----- ----- ----- ----- ----- Copied 45 GB in approx 15 min. Orginal FS % zfs get all data/csmdb/wsrc/wsrcmsolive NAME PROPERTY VALUE SOURCE data/csmdb/wsrc/wsrcmsolive type filesystem - data/csmdb/wsrc/wsrcmsolive creation Wed Jul 26 16:51 2006 - data/csmdb/wsrc/wsrcmsolive used 44.7G - data/csmdb/wsrc/wsrcmsolive available 332G - data/csmdb/wsrc/wsrcmsolive referenced 44.7G - data/csmdb/wsrc/wsrcmsolive compressratio 1.00x - Compress FS % ls -l total 32611837 -rw-r--r-- 1 root root 47353692160 Aug 22 12:36 CACHE.DAT % du -k 16305921 . % zfs get all export/home/zfstest NAME PROPERTY VALUE SOURCE export/home/zfstest type filesystem - export/home/zfstest creation Tue Aug 22 12:05 2006 - export/home/zfstest used 15.6G - export/home/zfstest available 515G - export/home/zfstest referenced 15.6G - export/home/zfstest compressratio 3.81x - export/home/zfstest mounted yes - Disclaimer: When you check the database it''s only 57% full on the file. So actual compress ratio is not that high (only 1.63). Thanks, Nan Liu This electronic mail transmission from the Richland Newhope Center, including any attached files, may contain confidential and/or privileged information for the sole use of the intended recipient(s). It is not intended for transmission to, or receipt by, any unauthorized parties. Any review, use, distribution, dissemination, downloading, copying or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), you are hereby notified that you have received this transmission in error. If you have received this transmission in error, please delete it, as well as any copies, from your system without copying it, and notify the sender of the error by reply e-mail, so that our address record can be corrected. Thank you.
On Tue, 22 Aug 2006, Matthew Ahrens wrote:> gzip. We plan to implement a broader range of compression algorithms in > the future.Cool. Presumably, the algorithm used will be a user-settable property? -- Rich Teer, SCNA, SCSA, OpenSolaris CAB member President, Rite Online Inc. Voice: +1 (250) 979-1638 URL: http://www.rite-group.com/rich
On Tue, Aug 22, 2006 at 10:09:19AM -0700, Rich Teer wrote:> On Tue, 22 Aug 2006, Matthew Ahrens wrote: > > > gzip. We plan to implement a broader range of compression algorithms in > > the future. > > Cool. Presumably, the algorithm used will be a user-settable property?That''s correct, you will be able to do ''zfs set compression=<whatever> <fs>''. --matt
For what it''s worth, an unbuilt Solaris workspace (containing only source files and SCCS files) stored on ZFS has a reported compression ratio of about 1.87. A large filesystem on which I keep primarily compilers (including several versions of the full Sun Studio install) has a ratio of 1.57. It''s not extraordinary compression, but it''s pretty fast.... This message posted from opensolaris.org
Anantha N. Srirama
2006-Aug-22 19:25 UTC
[zfs-discuss] Re: ZFS compression / space efficiency
We''re running ZFS with compress=ON on a E2900. I''m hosting SAS/SPDS datasets (files) on these filesystems and am achieving 1:3.87 (as reported by zfs) compression. Your mileage will vary depending on the data you are writing. If your data is already compressed (zip files) then don''t expect any payback. This message posted from opensolaris.org