Hi Some developers here said a long time ago that someone should show the code for LZO compression support for ZFS before talking about the next step. I made that code with a friend and we also made a little benchmark to give a first impression: http://denisy.dyndns.org/lzo_vs_lzjb/ I hope we made no technical error, but if you find something not accurate, we will correct it. Denis -- This message posted from opensolaris.org
Great read. On Fri, Aug 14, 2009 at 3:05 AM, Denis Ahrens <no-reply at opensolaris.org>wrote:> Hi > > Some developers here said a long time ago that someone should show > the code for LZO compression support for ZFS before talking about the > next step. I made that code with a friend and we also made a little > benchmark to give a first impression: > > http://denisy.dyndns.org/lzo_vs_lzjb/ > > I hope we made no technical error, but if you find something > not accurate, we will correct it. > > Denis > -- > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090814/776b6d12/attachment.html>
On Fri, Aug 14, 2009 at 2:05 PM, Denis Ahrens<no-reply at opensolaris.org> wrote:> Some developers here said a long time ago that someone should show > the code for LZO compression support for ZFS before talking about the > next step.Isn''t the main problem license, LZO being GPL, while zfs CDDL? -- Fajar
Thanks for the informative analysis! Just wondering - are there better candidates out there than even LZO for this purpose? Alex. On Fri, Aug 14, 2009 at 8:05 AM, Denis Ahrens<no-reply at opensolaris.org> wrote:> Hi > > Some developers here said a long time ago that someone should show > the code for LZO compression support for ZFS before talking about the > next step. I made that code with a friend and we also made a little > benchmark to give a first impression: > > http://denisy.dyndns.org/lzo_vs_lzjb/ > > I hope we made no technical error, but if you find something > not accurate, we will correct it. > > Denis > -- > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-- Marie von Ebner-Eschenbach - "Even a stopped clock is right twice a day." - http://www.brainyquote.com/quotes/authors/m/marie_von_ebnereschenbac.html
Just to answer my own question - this one might be interesting: http://www.quicklz.com/ Alex. On Fri, Aug 14, 2009 at 3:15 PM, Alex Lam S.L.<alexlamsl at gmail.com> wrote:> Thanks for the informative analysis! > > Just wondering - are there better candidates out there than even LZO > for this purpose? > > Alex. > > > On Fri, Aug 14, 2009 at 8:05 AM, Denis Ahrens<no-reply at opensolaris.org> wrote: >> Hi >> >> Some developers here said a long time ago that someone should show >> the code for LZO compression support for ZFS before talking about the >> next step. I made that code with a friend and we also made a little >> benchmark to give a first impression: >> >> http://denisy.dyndns.org/lzo_vs_lzjb/ >> >> I hope we made no technical error, but if you find something >> not accurate, we will correct it. >> >> Denis >> -- >> This message posted from opensolaris.org >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> > > > > -- > > Marie von Ebner-Eschenbach ?- "Even a stopped clock is right twice a > day." - http://www.brainyquote.com/quotes/authors/m/marie_von_ebnereschenbac.html >-- Mike Ditka - "If God had wanted man to play soccer, he wouldn''t have given us arms." - http://www.brainyquote.com/quotes/authors/m/mike_ditka.html
The only problem I see, is data set size. Let me explain (and please correct if I''m wrong). ZFS basically compresses 2 things -- metadata and data. And data is at most 128k chunk. Each chunk is individually compressed, not the whole file. This should affect dictionary size for the lzo compressor, which would change compression ratio and speed. I think a more valid test (if you don''t add lzo to your test kernel), would be to chunk each file into 128k blocks, then run the compressor on that. Likewise, you would need to decompress each 128k block. It would be best to read/write the chunks from one file, but you could use some sort of tempfs as a first run with multiple files. Just some thoughts, and thanks for the interesting work! Jeb -- This message posted from opensolaris.org
On Aug 14, 2009, at 7:15 AM, Alex Lam S.L. wrote:> Thanks for the informative analysis! > > Just wondering - are there better candidates out there than even LZO > for this purpose?tamp is perhaps another interesting candidate. But Tim is in a better position to say where it is heading. http://blogs.sun.com/timc/entry/tamp_a_lightweight_multi_threaded Personally, I don''t think any of the current LZ candidates will be worth the effort to integrate. To see why, I put some data in my blog and show why gzip was worth the effort. http://richardelling.blogspot.com/2009/08/justifying-new-compression-algorithms.html -- richard