Running SXDE (snv_70) for a file server, and I must admit I''m new to Solaris and zfs. zfs does not appear to do any compression at all, here is what I did to set it up: I created a four drive raidz array: zpool create pool raidz c0d0 c0d1 c1d0 c1d1 then an extra file system on that: zfs create -o mountpoint=/export/data pool/data After adding a bit of data (200GB or so), i figured that compression would in fact be a good idea: zfs set compression=on pool/data So far so good, "zfs get all" reports compression to be active. Now for the problem: After adding another 300GB of uncompressed .tif and .bin/.cue (audio CD) files, compression ratio is still at 1.00, indicating that no compression has taken place. Where did I go wrong? Anything I can do to get compression back after the cause for the problem has been fixed? All the data has been copied across a network, through Samba - set up according to instructions found in various Solaris forums. -- Joachim Hvorfor bruke fremmedord, n?r det finnes adekvate norske substitutter?
On Sat, 26 Jan 2008 15:20:01 +0100, Rob Logan <Rob at Logan.com> wrote:> it won''t go back and compress old data, but if you copy > the files, perhaps into a new folder, they will be > compressed on that copy or move.I can''t see why that should help, since the 300GB I added after enabling compression hasn''t been compressed at all. The compression is only seemingly enabled, is this a bug with build 70, or is there some misconfiguration at my place? -- Joachim Hvorfor bruke fremmedord, n?r det finnes adekvate norske substitutter?
On 26-Jan-08, at 2:24 AM, Joachim Pihl wrote:> Running SXDE (snv_70) for a file server, and I must admit I''m new to > Solaris and zfs. zfs does not appear to do any compression at all, > here is > what I did to set it up: > > I created a four drive raidz array: > > zpool create pool raidz c0d0 c0d1 c1d0 c1d1 > > then an extra file system on that: > > zfs create -o mountpoint=/export/data pool/data > > After adding a bit of data (200GB or so), i figured that > compression would > in fact be a good idea: > > zfs set compression=on pool/data > > So far so good, "zfs get all" reports compression to be active. Now > for > the problem: After adding another 300GB of uncompressed .tif > and .bin/.cue > (audio CD) files,I wouldn''t expect very much compression on that kind of data. --Toby> compression ratio is still at 1.00, indicating that no > compression has taken place. Where did I go wrong? Anything I can > do to > get compression back after the cause for the problem has been > fixed? All > the data has been copied across a network, through Samba - set up > according to instructions found in various Solaris forums. > > -- > Joachim > > Hvorfor bruke fremmedord, n?r det finnes adekvate norske substitutter? > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On Jan 26, 2008, at 3:24 AM, Joachim Pihl wrote:> So far so good, "zfs get all" reports compression to be active. Now > for > the problem: After adding another 300GB of uncompressed .tif > and .bin/.cue > (audio CD) files, compression ratio is still at 1.00, indicating > that no > compression has taken place.TIFF files can how their own compression ("compressed TIFF") and many image editors have this on by default, so you wouldn''t know about it unless you specifically looked for it. I think the TIFF spec specifies LZW compression for this... but either way, if this is indeed the case, zfs compression won''t help with those Now, the bin/cue file format specifies no optional compression, so those bin files should be nothing but a raw image of 16bit PCM audio... which you should see some (but not great) compression with. The default lzjb compression scheme in zfs might not be terribly effective on this type of file data being that it''s optimized for speed rather than compression efficiency. Try turning on gzip compression in zfs instead and see if things improve. To make that simple I''d just make a new fs (eg; pool/data/audio) and then ''zfs set compression=gzip-4 pool/data/audio'' and then mv your bin/ cue files there. /dale
On Sat, 26 Jan 2008 23:33:16 +0100, Toby Thain <toby at smartgames.ca> wrote:> On 26-Jan-08, at 2:24 AM, Joachim Pihl wrote:>> So far so good, "zfs get all" reports compression to be active. Now for >> the problem: After adding another 300GB of uncompressed .tif and >> .bin/.cue >> (audio CD) files, > > I wouldn''t expect very much compression on that kind of data.I didn''t expect miracles, but since WinRAR gave 13% compression on a representative data set with ZIP compression at the fastest setting, I was expecting to see a compression ratio > 1.05 at least, not == 1.00. Getting 5-10% more space on a drive never hurts. Could it be that I should specify compression algorithm? Documentation says it will use lzjb by default, so I did not set it explicitly. -- Joachim Hvorfor bruke fremmedord, n?r det finnes adekvate norske substitutter?
> I didn''t expect miracles, but since WinRAR gave 13% compressionZFS doesn''t compress a block if it can''t get a certain amount of return on it. Since the default compression is less effective than RAR, you can bet ZFS is seeing much less than 13% return. I expect everything is working properly, and you are just dealing with well-compressed files :) You could try setting the zfs compression to a stronger algorithm to see if anything gets compressed that way. This message posted from opensolaris.org
Why not create a sample dataset and throw some large .txt files out there and see what happens? That way you''ll know for certain if there''s some bug you''re hitting, or if it''s just not applicable to your current dataset. On 1/27/08, Joachim Pihl <jpihl at broadpark.no> wrote:> > On Sat, 26 Jan 2008 23:33:16 +0100, Toby Thain <toby at smartgames.ca> wrote: > > > On 26-Jan-08, at 2:24 AM, Joachim Pihl wrote: > > >> So far so good, "zfs get all" reports compression to be active. Now for > >> the problem: After adding another 300GB of uncompressed .tif and > >> .bin/.cue > >> (audio CD) files, > > > > I wouldn''t expect very much compression on that kind of data. > > I didn''t expect miracles, but since WinRAR gave 13% compression on a > representative data set with ZIP compression at the fastest setting, I was > expecting to see a compression ratio > 1.05 at least, not == 1.00. Getting > 5-10% more space on a drive never hurts. > > Could it be that I should specify compression algorithm? Documentation > says it will use lzjb by default, so I did not set it explicitly. > > > -- > Joachim > > Hvorfor bruke fremmedord, n?r det finnes adekvate norske substitutter? > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080127/a0160b1f/attachment.html>
On Sun, 27 Jan 2008 19:39:40 +0100, Tim <tim at tcsac.net> wrote:> Why not create a sample dataset and throw some large .txt files out there > and see what happens? That way you''ll know for certain if there''s some > bug > you''re hitting, or if it''s just not applicable to your current dataset.Good idea, but not tonight... It try two things: creating a few dozen GBs worth of "The quick brown fox...", and then create a temporary file system, enable the compression _before_ adding anything to it, then copy the same data to the new file system to see if I can spot any difference. Is there any way to check the compression of a single file in zfs, btw? Would come in handy for this task, and sifting through tons of man files is _not_ my favorite hobby. :-) -- Joachim Hvorfor bruke fremmedord, n?r det finnes adekvate norske substitutter?
On Sun, 27 Jan 2008 19:11:33 +0100, MC <rac at eastlink.ca> wrote:>> I didn''t expect miracles, but since WinRAR gave 13% compression > > ZFS doesn''t compress a block if it can''t get a certain amount of return > on it. > > Since the default compression is less effective than RAR, you can bet > ZFS is seeing much less than 13% return.What is the threshold, does anyone know that? It sound like a fair decision, though. Try to compress on write, increase perrformance on read by not compressing if there is little space to be gained. Wouldn''t slow down writes compared to not having such a threshold either. Clever!> I expect everything is working properly, and you are just dealing with > well-compressed files :)Raw audio and uncompressed 16bit TIFF files (35mm negative scans) have high content entropy, so they are difficult to compress, that could be the culprit.> You could try setting the zfs compression to a stronger algorithm to see > if anything gets compressed that way.-- Joachim Hvorfor bruke fremmedord, n?r det finnes adekvate norske substitutter?
Joachim Pihl wrote:> On Sun, 27 Jan 2008 19:11:33 +0100, MC <rac at eastlink.ca> wrote: > >>> I didn''t expect miracles, but since WinRAR gave 13% compression >> ZFS doesn''t compress a block if it can''t get a certain amount of return >> on it. >> >> Since the default compression is less effective than RAR, you can bet >> ZFS is seeing much less than 13% return. > > What is the threshold, does anyone know that? It sound like a fair > decision, though. Try to compress on write, increase perrformance on read > by not compressing if there is little space to be gained. Wouldn''t slow > down writes compared to not having such a threshold either. Clever!See line 108 of this file: http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/fs/zfs/zio_compress.c#73 -- Darren J Moffat