dovecot - Jun 2008 - deliver saving mails with hard linking

http://dovecot.org/tmp/deliver-multiple.diff for Dovecot v1.1 implements
-p <path> parameter for deliver, which reads the input mail from the
specified path instead of stdin. With maildir and hardlink copying
enabled, it also tries to hard link the file to destination instead of
copying it.

So to get the same mail delivered to multiple recipients using hard
links, you'd have to write a small wrapper script that stored the mail
temporarily somewhere and then called deliver for each recipient using
the -p parameter.

Any thoughts on if this should go to v1.1 like this? Some problems:

 - If you're not using a single UID/GID you probably have to make the
file world-readable.
 - A wrapper script is kind of ugly, so it would be nice if deliver took
a list of all recipients as input. But the deliver code is currently
ugly enough that this would require larger changes.
 - Am I forgetting something?..

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20080601/1adff394/attachment-0002.bin>

On Sun, 2008-06-01 at 22:32 +0300, Timo Sirainen wrote:
> http://dovecot.org/tmp/deliver-multiple.diff for Dovecot v1.1 implements
> -p <path> parameter for deliver, which reads the input mail from the
> specified path instead of stdin. With maildir and hardlink copying
> enabled, it also tries to hard link the file to destination instead of
> copying it.
Wow. ;)

>  - Am I forgetting something?..
Cleaning up?  Quoting the part from option (2) of your previous mail,
which this seems to implement:
> ?All messages could be then stored in some global
> directory and hard linked from there to users' mailboxes.
When reading that I already wondered about cleaning up and freeing disk
space. If every recipient deleted their own "copy of the mail", the
inodes link count will go down to 1 due to the still existing global
copy. But it will survive despite being "deleted" (from the collective
users POV), occupying disk space -- and possibly keeping data around
that is assumed to be removed.

Of course, one could periodically check the link counts in yet another
external script, carefully ensuring deliver is not currently doing its
job, and get rid of the stale copies...

  guenther


-- 
char *t="\10pse\0r\0dtu\0. at
ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8?
c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){
putchar(t[s]);h=m;s=0; }}}

Timo Sirainen wrote:
> http://dovecot.org/tmp/deliver-multiple.diff for Dovecot v1.1 implements
> -p <path> parameter for deliver, which reads the input mail from the
> specified path instead of stdin. With maildir and hardlink copying
> enabled, it also tries to hard link the file to destination instead of
> copying it.
Excellent, this could save lots of disk space.

Does this also work through the Sieve plugin, or is Sieve doing its own
writing?


Anders

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sun, 1 Jun 2008, Timo Sirainen wrote:

The SHA1 idea took me:

If deliver calculates the hash for each mail, then hardlinks each mail 
into, say, $HOME{target user}/../.inspool/{sha1_hash} (to support if you 
have your users on different physical disks).
If the hash already exists and the size is equal, hardlink the new 
recpient's file to it.

This cache would be cleaned by a cron script, e.g. once a day or something 
like that. Deliver could update the atime to reflect its last hard link or 
so.

The calculation overhead might be heavy.

But the "-p" approach seems to cry for to support LMTP.

BTW: For some accounts I run a nightly cron script that hardlinks any 
equal files, because they archive everything. I do first collect all mails 
sorted by size, then really compare them byte-by-byte if one mail is newer 
than one day.

Bye,

- -- 
Steffen Kaiser
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFIQ8ETVJMDrex4hCIRAv/8AKCrBNsxHvpfAk0Zi17N+t+SgZy3BgCfTOl3
irK1VrncoMf874FywCrb6/o=ZI+x
-----END PGP SIGNATURE-----