dovecot - Sep 2010 - (Single instance) attachment storage with single-dbox

Just a note to myself and whoever else cares, should be added to wiki
once it has its own page about this:

With single-dbox messages can be copied with hard linking. This means
that there can be multiple files that point to the same attachment file.
The attachment is now deleted only once the mail file's link count drops
to zero, so this works fine..

..until someone goes and starts manually copying files or maybe
restoring from backups or whatever, causing the hard links to be
replaced with separate files having link count=1, even when there are
other files pointing to the same attachment file. Now deleting this one
mail will delete the attachment too early.

I can't think of any other reasonable way to handle this though, so
unless someon has some great ideas, I think the solution is to simply
add enough warnings that message store shouldn't be accessed directly.
Maybe add some import/export commands to doveadm which can be used to
add a bunch of mails to storage without doing it directly on filesystem.

On 2010-09-24 2:57 PM, Timo Sirainen wrote:
> Maybe add some import/export commands to doveadm which can be used to
> add a bunch of mails to storage without doing it directly on filesystem.
+1

I've always just used cp -rp to restore mail, so yeah, if that would
break SiS, I agree there should be a doveadm command for restoring mails.

-- 

Best regards,

Charles

On 9/24/2010 11:57 AM, Timo Sirainen wrote:
> Just a note to myself and whoever else cares, should be added to wiki
> once it has its own page about this:
>
> With single-dbox messages can be copied with hard linking. This means
> that there can be multiple files that point to the same attachment file.
> The attachment is now deleted only once the mail file's link count
drops
> to zero, so this works fine..
Why is this specific to sdbox?  Does mdbox SIS work differently?
-- 
Daniel

On 24/09/2010 19:57, Timo Sirainen wrote:
> Just a note to myself and whoever else cares, should be added to wiki once
it has its own page about this:
>    
I have been meaning to say we should have a wiki page about this.
> With single-dbox messages can be copied with hard linking. This means that
there can be multiple files that point to the same attachment file. The
attachment is now deleted only once the mail file's link count drops to
zero, so this works fine..
>
> ..until someone goes and starts manually copying files or maybe restoring
from backups or whatever, causing the hard links to be replaced with separate
files having link count=1, even when there are other files pointing to the same
attachment file. Now deleting this one mail will delete the attachment too
early.
>
> I can't think of any other reasonable way to handle this though, so
unless someon has some great ideas, I think the solution is to simply
> add enough warnings that message store shouldn't be accessed directly.
Maybe add some import/export commands to doveadm which can be used to add a
bunch of mails to storage without doing it directly on filesystem.
Conceptually there is an attachment table with a primary key, and a 
message table which has foreign keys referring into the attachment 
table. In database theory, you could set up a foreign key constraint, 
and then it would not be possible to remove attachments which were still 
referenced by messages.

I'm not saying all this data should be under *SQL; just thinking aloud.

Can we do something /like/ that but only using a filesystem?

Perhaps each message could have its own hard links to the attachment 
file? That way a message's attachments would be kept in existence by the 
filesystem itself.

When you store a message with attachments, you could store the message file:

u.123

and have its attachments stored as hard links under names which appear 
adjacent to the message file's name when the directory listing is sorted.

u.123.a.1
u.123.a.2

Because the attachment files are adjacently-named to the message file, 
people accessing the sdbox mailstore as a filesystem should realise that 
they must backup/restore the message file and its associated attachment 
files.

Bill