On 3/15/16 10:13 AM, Sven Hartge wrote:> I don't have a script, but I can provide some numbers. I did a test with > a server for about 10.000 users and 2TB worth of mail, converting from > Maildir++ to mdbox with zlib (level = 6) and had a final size of 1TB, so > 2:1 reduction.These numbers roughly match my results. About 6 TB of mail compresses down to about 3 TB. The difference in ongoing CPU use for compression "on the fly" of newly arriving mail, and decompression of mail from the disk, is unnoticeable on my servers. Also keep in mind that even if it does increase CPU usage, it reduces disk usage. This is probably an excellent tradeoff for most people, since most servers are limited by disk throughput/latency more than CPU power. -- Robert L Mathews, Tiger Technologies, http://www.tigertech.net/
Quoting Robert L Mathews <lists at tigertech.com>:> On 3/15/16 10:13 AM, Sven Hartge wrote: > >> I don't have a script, but I can provide some numbers. I did a test with >> a server for about 10.000 users and 2TB worth of mail, converting from >> Maildir++ to mdbox with zlib (level = 6) and had a final size of 1TB, so >> 2:1 reduction. > > These numbers roughly match my results. About 6 TB of mail compresses > down to about 3 TB. > > Also keep in mind that even if it does increase CPU usage, it reduces > disk usage. This is probably an excellent tradeoff for most people, > since most servers are limited by disk throughput/latency more than CPU > power.Just thought I'd add, because it frustrated me and it's an amusing anecdote to this - The new 4k ashift doubled my disk usage of Maildir++ mail. So logically, if you're migrating from maildir++ to mdbox on 4k sector system, you may see a sizable decrease in disk usage without a compression change. Rick
Quoting Rick Romero <rick at havokmon.com>:> Quoting Robert L Mathews <lists at tigertech.com>: > >> On 3/15/16 10:13 AM, Sven Hartge wrote: >> >>> I don't have a script, but I can provide some numbers. I did a testwith>>> a server for about 10.000 users and 2TB worth of mail, converting from >>> Maildir++ to mdbox with zlib (level = 6) and had a final size of 1TB,so>>> 2:1 reduction. >> >> These numbers roughly match my results. About 6 TB of mail compresses >> down to about 3 TB. >> >> Also keep in mind that even if it does increase CPU usage, it reduces >> disk usage. This is probably an excellent tradeoff for most people, >> since most servers are limited by disk throughput/latency more than CPU >> power. > > Just thought I'd add, because it frustrated me and it's an amusing > anecdote > to this - The new 4k ashift doubled my disk usage of Maildir++ mail. > > So logically, if you're migrating from maildir++ to mdbox on 4k sector > system, you may see a sizable decrease in disk usage without acompression> change. > RickWith ZFS - Sorry - I dropped that in my edit.
Robert L Mathews <lists at tigertech.com> wrote:> Also keep in mind that even if it does increase CPU usage, it reduces > disk usage. This is probably an excellent tradeoff for most people, > since most servers are limited by disk throughput/latency more than > CPU power.IOPS are harder to scale (meaning: cost more to scale) than CPU power. And gzip (or lz4 of implemented someday) (or even blosc: http://www.blosc.org/. They claim "Designed to transmit data to the processor cache faster than a memcpy() OS call.") is effectively free with todays CPUs. Gr??e, Sven. -- Sigmentation fault. Core dumped.
On 15.03.2016 21:45, Sven Hartge wrote:> Robert L Mathews <lists at tigertech.com> wrote: > >> Also keep in mind that even if it does increase CPU usage, it reduces >> disk usage. This is probably an excellent tradeoff for most people, >> since most servers are limited by disk throughput/latency more than >> CPU power. > > IOPS are harder to scale (meaning: cost more to scale) than CPU power. > > And gzip (or lz4 of implemented someday) (or even blosc:liblz4 has been supported since 2.2.11+ http://wiki2.dovecot.org/Plugins/Zlib> http://www.blosc.org/. They claim "Designed to transmit data to the > processor cache faster than a memcpy() OS call.") is effectively free > with todays CPUs. > > Gr??e, > Sven. >