dovecot - Sep 2015 - Problems setting up SIS (duplicate files with same hash)

Hi everyone,

I am currently trying to setup our next dovecot server (2.2.18) with SIS
enabled, but I have enconutered 1 problem and 1 question:


*1) Same attachment, different filenames*

I have a feeling a misconfigured something (or forgot to do something)
because SIS almost works, but not quite since 3 identical attachment take
3x the space (see end of email for my SIS config).

For 3 the emails I sent with the same attachment, I got 3 seperate files
with the same hash name but a different string following the dash:

-rw------- 1 info info 2.6M Sep  5 11:14
e35083e3280a21c6aaabbae8cb23d54493f514a5-6a22711a7b05eb5509480000045c9bf4
-rw------- 1 info info 2.6M Sep  5 11:16
e35083e3280a21c6aaabbae8cb23d54493f514a5-6c22711a7b05eb5509480000045c9bf4
-rw------- 2 info info 2.6M Sep  5 11:18
e35083e3280a21c6aaabbae8cb23d54493f514a5-6e22711a7b05eb5509480000045c9bf4
drwx------ 2 info info 4.0K Sep  5 11:18 hashes

For the record, the "hashes" subfolder has only 1 hash
(e35083e3280a21c6aaabbae8cb23d54493f514a5) which perfectly matches the
first part of the 3 filenames.



*2) Saving attachments under a generic name or more flexible permissions?*

It seems that dovecot saves the attachment with the username and group of
the user who has received the email.

Is it possible to have dovecot save the attachments as a generic user (like
dovecot or dovenull) so that all my different users can share the same pool
of attachments?

Alternatively, is it possible to have dovecot save the attachment under
more flexible permissions so that they can be shared between users?

This would allow us to save alot of disk space.


*Here is the SIS config I am using:*

mail_attachment_dir = /data/emails/attachments
#mail_attachment_min_size = 128k
#mail_attachment_fs = sis posix
#mail_attachment_hash = %{sha1}



Any help would be very much appreciated.

Thank you,

Joe B

I'm starting to think there is something I don't understand about
running
dovecot with SIS.

After I deleted the 3 emails that have the same attachment, the 3
attachments are still on disk (saved under the name of the hash).

After running doveadm purge & doveadm deduplicate, the 3 attachments are
still on disk (the 3 emails were the only emails on server, so I can
guarantee that no other emails are pointing to the attachments).

*What am I not understanding?* It makes no sense that the attachments would
never be cleaned, so I am sure I am missing something.

Best regards,

Joe B

On Sat, Sep 5, 2015 at 11:40 AM, Joe Beaubien <joe.beaubien at gmail.com>
wrote:
> Hi everyone,
>
> I am currently trying to setup our next dovecot server (2.2.18) with SIS
> enabled, but I have enconutered 1 problem and 1 question:
>
>
> *1) Same attachment, different filenames*
>
> I have a feeling a misconfigured something (or forgot to do something)
> because SIS almost works, but not quite since 3 identical attachment take
> 3x the space (see end of email for my SIS config).
>
> For 3 the emails I sent with the same attachment, I got 3 seperate files
> with the same hash name but a different string following the dash:
>
> -rw------- 1 info info 2.6M Sep  5 11:14
> e35083e3280a21c6aaabbae8cb23d54493f514a5-6a22711a7b05eb5509480000045c9bf4
> -rw------- 1 info info 2.6M Sep  5 11:16
> e35083e3280a21c6aaabbae8cb23d54493f514a5-6c22711a7b05eb5509480000045c9bf4
> -rw------- 2 info info 2.6M Sep  5 11:18
> e35083e3280a21c6aaabbae8cb23d54493f514a5-6e22711a7b05eb5509480000045c9bf4
> drwx------ 2 info info 4.0K Sep  5 11:18 hashes
>
> For the record, the "hashes" subfolder has only 1 hash
> (e35083e3280a21c6aaabbae8cb23d54493f514a5) which perfectly matches the
> first part of the 3 filenames.
>
>
>
> *2) Saving attachments under a generic name or more flexible permissions?*
>
> It seems that dovecot saves the attachment with the username and group of
> the user who has received the email.
>
> Is it possible to have dovecot save the attachments as a generic user
> (like dovecot or dovenull) so that all my different users can share the
> same pool of attachments?
>
> Alternatively, is it possible to have dovecot save the attachment under
> more flexible permissions so that they can be shared between users?
>
> This would allow us to save alot of disk space.
>
>
> *Here is the SIS config I am using:*
>
> mail_attachment_dir = /data/emails/attachments
> #mail_attachment_min_size = 128k
> #mail_attachment_fs = sis posix
> #mail_attachment_hash = %{sha1}
>
>
>
> Any help would be very much appreciated.
>
> Thank you,
>
> Joe B
>

Can anyone help with SIS?



On Sat, Sep 5, 2015 at 11:40 AM, Joe Beaubien <joe.beaubien at gmail.com>
wrote:
> Hi everyone,
>
> I am currently trying to setup our next dovecot server (2.2.18) with SIS
> enabled, but I have enconutered 1 problem and 1 question:
>
>
> *1) Same attachment, different filenames*
>
> I have a feeling a misconfigured something (or forgot to do something)
> because SIS almost works, but not quite since 3 identical attachment take
> 3x the space (see end of email for my SIS config).
>
> For 3 the emails I sent with the same attachment, I got 3 seperate files
> with the same hash name but a different string following the dash:
>
> -rw------- 1 info info 2.6M Sep  5 11:14
> e35083e3280a21c6aaabbae8cb23d54493f514a5-6a22711a7b05eb5509480000045c9bf4
> -rw------- 1 info info 2.6M Sep  5 11:16
> e35083e3280a21c6aaabbae8cb23d54493f514a5-6c22711a7b05eb5509480000045c9bf4
> -rw------- 2 info info 2.6M Sep  5 11:18
> e35083e3280a21c6aaabbae8cb23d54493f514a5-6e22711a7b05eb5509480000045c9bf4
> drwx------ 2 info info 4.0K Sep  5 11:18 hashes
>
> For the record, the "hashes" subfolder has only 1 hash
> (e35083e3280a21c6aaabbae8cb23d54493f514a5) which perfectly matches the
> first part of the 3 filenames.
>
>
>
> *2) Saving attachments under a generic name or more flexible permissions?*
>
> It seems that dovecot saves the attachment with the username and group of
> the user who has received the email.
>
> Is it possible to have dovecot save the attachments as a generic user
> (like dovecot or dovenull) so that all my different users can share the
> same pool of attachments?
>
> Alternatively, is it possible to have dovecot save the attachment under
> more flexible permissions so that they can be shared between users?
>
> This would allow us to save alot of disk space.
>
>
> *Here is the SIS config I am using:*
>
> mail_attachment_dir = /data/emails/attachments
> #mail_attachment_min_size = 128k
> #mail_attachment_fs = sis posix
> #mail_attachment_hash = %{sha1}
>
>
>
> Any help would be very much appreciated.
>
> Thank you,
>
> Joe B
>

I figured out item #1, however, #2 is still problematic.

How can I setup dovecot to handle multiple mailboxes with SIS?

If I use "sis posix", the folders /var/attachments/[0-9a-f][0-9a-f]
get
created under the name of the first mailbox. Whenevver the second mailbox
tries to read/write to those folders, it fails. Also, it seems that the
attachments never get deduplicated, but i'll deal with that later.

If I use "sis-queue", then the files get deduplicated, but I cannot
fix the
multiple mailbox issues. I tried adding %u to the mail_attachment_dir,
whcih seems to work at first. But when I try to deduplicate the attachments
with "doveadm sis deduplicate" it seems that mail_attachment_fs does
not
support %u and the path does not resolve properly.

Any ideas?

Thanks,

Joe B

I'm trying to run SIS with queueing enabled

I tried running SIS with "sis posix", however when the
/var/attachments/[0-9a-f][0-9a-f] folders get created under the name of the
first mailbox, the second mailbox cannot write/read to them.

I then tried adding %u to mail_attachment_dir, but then I had errors when
running doveadm sis deduplicate because mail_attachment_fs doesn't seem to
support %u in the queue path.

How can I support multiple mailboxes with SIS???

On Sat, Sep 5, 2015 at 11:40 AM, Joe Beaubien <joe.beaubien at gmail.com>
wrote:
> Hi everyone,
>
> I am currently trying to setup our next dovecot server (2.2.18) with SIS
> enabled, but I have enconutered 1 problem and 1 question:
>
>
> *1) Same attachment, different filenames*
>
> I have a feeling a misconfigured something (or forgot to do something)
> because SIS almost works, but not quite since 3 identical attachment take
> 3x the space (see end of email for my SIS config).
>
> For 3 the emails I sent with the same attachment, I got 3 seperate files
> with the same hash name but a different string following the dash:
>
> -rw------- 1 info info 2.6M Sep  5 11:14
> e35083e3280a21c6aaabbae8cb23d54493f514a5-6a22711a7b05eb5509480000045c9bf4
> -rw------- 1 info info 2.6M Sep  5 11:16
> e35083e3280a21c6aaabbae8cb23d54493f514a5-6c22711a7b05eb5509480000045c9bf4
> -rw------- 2 info info 2.6M Sep  5 11:18
> e35083e3280a21c6aaabbae8cb23d54493f514a5-6e22711a7b05eb5509480000045c9bf4
> drwx------ 2 info info 4.0K Sep  5 11:18 hashes
>
> For the record, the "hashes" subfolder has only 1 hash
> (e35083e3280a21c6aaabbae8cb23d54493f514a5) which perfectly matches the
> first part of the 3 filenames.
>
>
>
> *2) Saving attachments under a generic name or more flexible permissions?*
>
> It seems that dovecot saves the attachment with the username and group of
> the user who has received the email.
>
> Is it possible to have dovecot save the attachments as a generic user
> (like dovecot or dovenull) so that all my different users can share the
> same pool of attachments?
>
> Alternatively, is it possible to have dovecot save the attachment under
> more flexible permissions so that they can be shared between users?
>
> This would allow us to save alot of disk space.
>
>
> *Here is the SIS config I am using:*
>
> mail_attachment_dir = /data/emails/attachments
> #mail_attachment_min_size = 128k
> #mail_attachment_fs = sis posix
> #mail_attachment_hash = %{sha1}
>
>
>
> Any help would be very much appreciated.
>
> Thank you,
>
> Joe B
>