Hi, may be someone has already done that: Do you have a script(?) tool which shows the efficiency of the mail compression if zlib is used? Something that shows the uncompressed size vrs. the compressed. Thanks for hints! /G?tz -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5571 bytes Desc: S/MIME Cryptographic Signature URL: <http://dovecot.org/pipermail/dovecot/attachments/20160315/3e656081/attachment-0001.p7s>
Em 15/03/16 12:01, G?tz Reinicke - IT Koordinator escreveu:> Hi, > > may be someone has already done that: Do you have a script(?) tool which > shows the efficiency of the mail compression if zlib is used? > > Something that shows the uncompressed size vrs. the compressed. >While i dont have the data you're looking for, i do have lots of servers running with zlib enabled and, if someone makes the script, i can run on some servers and provide the results ! -- Atenciosamente / Sincerily, Leonardo Rodrigues Solutti Tecnologia http://www.solutti.com.br Minha armadilha de SPAM, N?O mandem email gertrudes at solutti.com.br My SPAMTRAP, do not email it
G?tz Reinicke - IT Koordinator <goetz.reinicke at filmakademie.de> wrote:> may be someone has already done that: Do you have a script(?) tool > which shows the efficiency of the mail compression if zlib is used?> Something that shows the uncompressed size vrs. the compressed.I don't have a script, but I can provide some numbers. I did a test with a server for about 10.000 users and 2TB worth of mail, converting from Maildir++ to mdbox with zlib (level = 6) and had a final size of 1TB, so 2:1 reduction. Gr??e, Sven. -- Sigmentation fault. Core dumped.
On 16/03/2016 2:01 AM, G?tz Reinicke - IT Koordinator wrote:> Hi, > > may be someone has already done that: Do you have a script(?) tool which > shows the efficiency of the mail compression if zlib is used? > > Something that shows the uncompressed size vrs. the compressed.Remember one thing; emails are stored in plain text, the same text that they are normally transmitted b/w servers. With that in mind, text, particularly with repeating and common things like headers (and other things), then you should get significant reduction in size. The exception of the size benefits (storage), is when you have emails that are less than the file system block size (4k ext4 perhaps). So many emails are smaller than a block size and for those, zipping is not much benefit as it won't make a scrap of difference to storage. However, when you have users that send attachments and sometimes very large attachments, well, it will save loads of storage on those emails. Next, if you have a CPU bottleneck, then the extra overhead of compression may also be an issue; but unless your server is working hard, compression isn't likely to tax the CPU a great deal. Cheers Andrewm -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: OpenPGP digital signature URL: <http://dovecot.org/pipermail/dovecot/attachments/20160316/fb64e385/attachment.sig>
On 3/15/16 10:13 AM, Sven Hartge wrote:> I don't have a script, but I can provide some numbers. I did a test with > a server for about 10.000 users and 2TB worth of mail, converting from > Maildir++ to mdbox with zlib (level = 6) and had a final size of 1TB, so > 2:1 reduction.These numbers roughly match my results. About 6 TB of mail compresses down to about 3 TB. The difference in ongoing CPU use for compression "on the fly" of newly arriving mail, and decompression of mail from the disk, is unnoticeable on my servers. Also keep in mind that even if it does increase CPU usage, it reduces disk usage. This is probably an excellent tradeoff for most people, since most servers are limited by disk throughput/latency more than CPU power. -- Robert L Mathews, Tiger Technologies, http://www.tigertech.net/
Andrew McGlashan <andrew.mcglashan at affinityvision.com.au> writes:> On 16/03/2016 2:01 AM, G?tz Reinicke - IT Koordinator wrote: >> Hi, >> >> may be someone has already done that: Do you have a script(?) tool which >> shows the efficiency of the mail compression if zlib is used? >> >> Something that shows the uncompressed size vrs. the compressed. > > Remember one thing; emails are stored in plain text, the same text that > they are normally transmitted b/w servers.Emails are not stored in plaintext if you are using dbox/mdbox.
Götz Reinicke - IT Koordinator
2016-Mar-16 07:50 UTC
overview zlib efficiency? Summary and added note
Am 15.03.16 um 16:01 schrieb G?tz Reinicke - IT Koordinator:> Hi, > > may be someone has already done that: Do you have a script(?) tool which > shows the efficiency of the mail compression if zlib is used? > > Something that shows the uncompressed size vrs. the compressed.Hi, maybe my question was a bit misleading. But anyway thanks for your feedback regarding your experiences and compression rates. We already thought about the benefit of less IO and more CPU power, which is no concern. The mailboxes I checked also go with 40-60% compression rate. But what I was looking for was a tool or way to see what volume would be used if we where not using compression. e.g. "du -hs --without-zlib" Our management would like to see a graph one day which shows the volume uncompressed and compressed ... Adding zlib with mdbox or maildir - as we do it currently - is from my POV if you have the CPU power a MUST :) happy dovecoting - G?tz -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5571 bytes Desc: S/MIME Cryptographic Signature URL: <http://dovecot.org/pipermail/dovecot/attachments/20160316/5313e7cd/attachment.p7s>
Hi,
use "doveadm" to get all real message
doveadm -f table fetch -A "size.physical" ALL | awk
'{s+=$2}END{printf("%.2fMB\n", s/1024/1024);}'
189247.67MB .. 185G
use "du" to get size on disc:
In my case
with deduplication:
/srv/stroage/# du -s -h *
53G vmail
75G vmail_sis
without deduplication
/srv/stroage/# du -s -h -l *
53G vmail
209G vmail_sis
j4i, SIS can't use the zlib plugin so the 75G in my case are not
compressed (I haven't a filesystem that I trust and has a compression
feature). Anyway it has a 3:1 ratio in my case.
Maybe I interpret the SIS wrong and SIS couldn't be counted with du -l
(count links).
But if someone doesn't have SIS this values should be point you into the
right direction.
bye
Harald
Am 16.03.2016 um 08:50 schrieb G?tz Reinicke - IT
Koordinator:> Am 15.03.16 um 16:01 schrieb G?tz Reinicke - IT Koordinator:
>> Hi,
>>
>> may be someone has already done that: Do you have a script(?) tool
which
>> shows the efficiency of the mail compression if zlib is used?
>>
>> Something that shows the uncompressed size vrs. the compressed.
>
> Hi,
>
> maybe my question was a bit misleading. But anyway thanks for your
> feedback regarding your experiences and compression rates.
>
> We already thought about the benefit of less IO and more CPU power,
> which is no concern.
>
> The mailboxes I checked also go with 40-60% compression rate.
>
> But what I was looking for was a tool or way to see what volume would be
> used if we where not using compression.
>
> e.g. "du -hs --without-zlib"
>
> Our management would like to see a graph one day which shows the volume
> uncompressed and compressed ...
>
> Adding zlib with mdbox or maildir - as we do it currently - is from my
> POV if you have the CPU power a MUST :)
>
> happy dovecoting - G?tz
>
>
>
--
Harald Leithner
ITronic
Wiedner Hauptstra?e 120/5.1, 1050 Wien, Austria
Tel: +43-1-545 0 604
Mobil: +43-699-123 78 4 78
Mail: leithner at itronic.at | itronic.at
During migration from 1.x with maildir to 2.x with dbox a did the following trick: time dsync -R -u test at example.com backup maildir:/var/spool/imap/tmp/Maildir And got the following results: |orig |366|28s | |gz.6 |260|5min| |bz2.6|202|5min| |xz.1 |211|1min| |xz.2 |213|1min50sec| |xz.3 |201|3min| |xz.6 |198|5min| |xz.9 |198|10min| |lz4 |281|18s | 1-9 - is compression level. I choose lz4 finally. -- Anton Chevychalov G?tz Reinicke - IT Koordinator ????? 2016-03-15 18:01:> Hi, > > may be someone has already done that: Do you have a script(?) tool > which > shows the efficiency of the mail compression if zlib is used? > > Something that shows the uncompressed size vrs. the compressed. > > Thanks for hints! /G?tz