Hi again.... ... so the story goes on.... I''m running my storage array form a host (T5220 Sparc) running Solaris 8/07 with the latest patches. ZFS is still at version 4. So I enable the lzjb compression feature and start poring in syslog files. However the problem appears that the files are not compressing. A zfs get all command shows compression is "on". My compressratio reports 1.00 and the files are the same size coming from the previous host to the new pool. The files are basic ascii text. IS zfs compressing in the background and I just am not able to see it? Have I pulled another "user error" mistake? How can I prove I''m going to be able to compress any data?? Help!! Thanks in advance --Kenny -- This message posted from opensolaris.org
On Sun, 7 Sep 2008, Kenny wrote:> > So I enable the lzjb compression feature and start poring in syslog > files. However the problem appears that the files are not > compressing. A zfs get all command shows compression is "on". My > compressratio reports 1.00 and the files are the same size coming > from the previous host to the new pool. The files are basic ascii > text. > > IS zfs compressing in the background and I just am not able to see > it? Have I pulled another "user error" mistake?I think that you may be the victim of bad advice from myself and others. Every so often (something like seven seconds) zfs commits pending writes to disk. This ensures that old writes don''t remain in memory forever so that they never make it to disk (and are lost of the system crashes or loses power). The problem is that if your log files are written really slowly, then the amount of data which accumulates within 7 seconds is very small (might be a single line of text), and ends up being uncompressable. The block compressor will compress based on data in the block rather than whatever happens to be in the existing file already. If the problem is due to the "trickle factor" then you should see that if you copy a large log file that the filesystem now shows that some data is compressed. Another factor is if you chose to use NFS mounts to access your log server (bad idea). If you did this then the write is committed right away without buffering for 7 seconds. I believe that ZFS normally extends short tail blocks up to the filesystem block size, but this behavior may not apply for compression since compression writes variable sized blocks up to the filesystem block size. Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
On Sun, 7 Sep 2008, Bob Friesenhahn wrote:> If the problem is due to the "trickle factor" then you should see that > if you copy a large log file that the filesystem now shows that some > data is compressed.I should mention that if copying the file causes it to be nicely compressed, then you can use this to your advantage. Your log-file rotator can copy the file and delete the original rather than just renaming it. You would not want to take any snapshots until after the logs have been rotated since the snapshots would capture the sub-sized poorly-compressed blocks. Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
On Sun, Sep 7, 2008 at 15:08, Bob Friesenhahn <bfriesen at simple.dallas.tx.us> wrote:> I should mention that if copying the file causes it to be nicely > compressed, then you can use this to your advantage. Your log-file > rotator can copy the file and delete the original rather than just > renaming it. You would not want to take any snapshots until after the > logs have been rotated since the snapshots would capture the sub-sized > poorly-compressed blocks.I agree. Another option is to leave compression off on the "current" set of log files (from today, or this week, depending on how often you rotate) and create a child filesystem with compression on to store the archived logs. This is what I do at work, and it does pretty well: NAME PROPERTY VALUE local-space/logdisk/oldlogs compressratio 3.27x NAME USED local-space/logdisk/oldlogs 80.2G # tar cf - . | wc -c 257,002,106,368 So I''ve got 250 gigs compressed into 80. A separate filesystem with different properties is my suggestion. Will