dovecot - Jul 2015 - Questions about hardlinks, alternate storage and compression

Hi Javier,
thanks for your reply.

I already checked SIS and, while interesting, is not what I want, because:
1) it can be difficult to restore a single message/attachment from a backup
2) only the attachments, and not the entire messages, are deduped.

Message-based hardlinks really exists for a reason. The good news is 
that I found _why_ they are not working: it depends from how dovecot and 
its sieve plugin (pigenhole) interact. Basically, if I define anything 
for the before_sieve and after_sieve variables, dovecot stops creating 
hardlinks for multiple copies of the same message.

On the other hand, private (per-user) sieve file works without 
interfering with hardlinks. In a similar manner, disabling sieve also 
permits dovecot to create multiple hardlinks for a single message.

Does someone know if newer dovecot versions change anything in this regard?

Thank you all.

On 13/07/15 11:10, Javier Miguel Rodr?guez wrote:
> Search about "single instance storage dovecot". This is what you
need.
>
> Regards
>
> Javier
>
> On 27/06/2015 18:18, Gionatan Danti wrote:
>> Hi all,
>> I have some questions about hardlinks, alternate storage and
>> compression. I already scanned the list for related information and I
>> have an idea of how things works, but I would like to have a definite
>> answer.
>>
>> System spec:
>> - CentOS 6.6 x64
>> - dovecot-2.0.9-8.el6_6.4.x86_64 RPM package/version
>> - sdbox mail store
>>
>> 1) About hardlinks: when sending the same message to two different
>> recipients, I see that the two u.x files are created as two different
>> files. Diffing them, I see that the only difference is a single char
>> (see [1] for an example). My questions are:
>>   a) it is possible to tell dovecot to create a single file + a single
>> hardlink (linkref=2)? As other IMAP servers support that features (eg:
>> Cyrus, CommunigatePro, etc) I am wondering if I missed something in
>> configuring dovecot...
>>   b) If it is not possible, can I run a script that compare the
>> various files and substitute equal ones (minus the changing line) with
>> hardlinks, or it will confuse dovecot? As a side note, why the
>> changing line ever exists?
>>
>> 2) About alternate storage and compression: actually, I use a single
>> mail_location without compression. I would like to have an alternate
>> storage and to enable compression on it only, leaving the main
>> location without compression. I if understand it correctly, it
>> _should_ be done using a command similar to "doveadm -Dv -o
>> "plugin/zlib_save=gz" altmove -uu testuser sentbefore
8d". I'm right
>> thinking that it should work? I will really end with a primary
>> uncompressed mail store and an alternate, zlib-compressed one?
>>
>> Thank you all and sorry if I did some naive questions.
>>
>> [1]
>> 63c63
>> < G2fd0811c64be8e553d970000eaa8309f
>> ---
>>> G2ed0811c64be8e553d970000eaa8309f
>>
>>
>>
>
-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti at assyoma.it - info at assyoma.it
GPG public key ID: FF5F32A8

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, 13 Jul 2015, Gionatan Danti wrote:
> On the other hand, private (per-user) sieve file works without interfering 
> with hardlinks. In a similar manner, disabling sieve also permits dovecot
to
> create multiple hardlinks for a single message.
>
> Does someone know if newer dovecot versions change anything in this regard?
LMTP adds Delivered-To header, so all delivered messages are unique and 
you cannot hardlink messages regardless of Sieve.

If Dovecot LDA adds headers, too, I do not know.

- -- 
Steffen Kaiser
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEVAwUBVaSpX3z1H7kL/d9rAQKuPwf/e4GddZvm/qj9sfAnVgV3H5iC62fnS6Ny
/TPaXcuLcN5Tx9slhLTwIx8/GRROUVwLVqKYjaXwQciV2yytBu5vkC0+lowIZGq9
kJAAKPp4h3Ia6SDGhI8E5Im9VGGSpbXyLKR+V3rf1G/sOyvJTITliVe4ckf76xrI
c1LGYumW0BGZeNZAAA0lYHZGrgy5meCrL20CMupmahoHsOFw5cA3HhJ/dEBRPlOJ
y886BScRh7dWJXyS+PUzPFlbFOeULKvh6fVwCK7b4+aFkfjLedDLew5TThWiblK5
c5+rx0pAh8xVdXGZyQXzPjUl22KbQmGfzv78XWlN2WksCnMVaFXe2g==3iPP
-----END PGP SIGNATURE-----

On 14/07/15 08:17, Steffen Kaiser wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Mon, 13 Jul 2015, Gionatan Danti wrote:
>
>> On the other hand, private (per-user) sieve file works without
>> interfering with hardlinks. In a similar manner, disabling sieve also
>> permits dovecot to create multiple hardlinks for a single message.
>>
>> Does someone know if newer dovecot versions change anything in this
>> regard?
>
> LMTP adds Delivered-To header, so all delivered messages are unique and
> you cannot hardlink messages regardless of Sieve.
>
> If Dovecot LDA adds headers, too, I do not know.
>
Mmm... I'm using LMTP, but I can't find the "Delivered-To"
header.
Below you can see an example of successfully hard-linked email [1]

I am missing something?

[1]
Return-Path: <g.danti at assyoma.it>
Received: from mail.gruppocrimi.it
         by mail.gruppocrimi.it (Dovecot) with LMTP id 
VFA8Fj3OmlUStwAA6qgwnw
         ; Mon, 06 Jul 2015 20:51:41 +0200
Received: from localhost (localhost.localdomain [127.0.0.1])
         by mail.gruppocrimi.it (Postfix) with ESMTP id 22AB4A1A85;
         Mon,  6 Jul 2015 20:51:41 +0200 (CEST)
X-Virus-Scanned: amavisd-new at gruppocrimi.it
Received: from mail.gruppocrimi.it ([127.0.0.1])
         by localhost (mail.gruppocrimi.it [127.0.0.1]) (amavisd-new, 
port 10024)
         with LMTP id NC3YcizeDFPO; Mon,  6 Jul 2015 20:51:40 +0200 (CEST)
Received: from mr003msb.fastweb.it (mr003msb.fastweb.it [85.18.95.87])
         by mail.gruppocrimi.it (Postfix) with ESMTP id 4380DA1A7C;
         Mon,  6 Jul 2015 20:51:40 +0200 (CEST)
Received: from ceres.assyoma.it (93.63.55.57) by mr003msb.fastweb.it 
(8.5.140.03)
         id 55501C9F0432631D; Mon, 6 Jul 2015 20:51:40 +0200
Received: by ceres.assyoma.it (Postfix, from userid 48)
         id B7B912643B4; Mon,  6 Jul 2015 20:51:39 +0200 (CEST)
To: gionatan.danti at gruppocrimi.it, g.danti at gruppocrimi.it
Subject: test invio
X-PHP-Originating-Script: 0:rcube.php
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII;
  format=flowed
Content-Transfer-Encoding: 7bit
Date: Mon, 06 Jul 2015 20:51:39 +0200
From: Gionatan Danti <g.danti at assyoma.it>
Organization: Assyoma s.r.l.
Message-ID: <ef3149d82ef4350c1c9b6da9c1f03fae at assyoma.it>
X-Sender: g.danti at assyoma.it
User-Agent: Roundcube Webmail/1.0.5


-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti at assyoma.it - info at assyoma.it
GPG public key ID: FF5F32A8