What would be involved in implementing SIS within Dovecot? A new or modified mailbox format? -- Daniel
On Fri, 2009-08-14 at 11:28 -0700, Daniel L. Miller wrote:> What would be involved in implementing SIS within Dovecot? A new or > modified mailbox format?It could be added to dbox without too much trouble. I already kind of planned for it: /* Pointer to external message data. Format is: 1*(<start offset> <byte count> <ref>) */ DBOX_METADATA_EXT_REF = 'X', Nothing uses that yet though. So what you'd need is: 1) When writing the data, extract the attachments and write them to different files. Add pointers to those files to the EXT_REF metadata. Dovecot's message parsers should make this not-too-difficult to implement. 2) When reading messages, get the ext references and then you should be able to create concat-istream by piecing together the main dbox istream with those external istreams. Maybe implement some istream that lazily opens the external files only when it's actually needed. 3) Once that works, figure out a way to implement SIS for those externally written files. Creating hard links to files a global attachment storage would probably be a workable solution. Then scan through the storage once in a while and delete files with link count=1. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: <http://dovecot.org/pipermail/dovecot/attachments/20090814/77f3d458/attachment-0002.bin>
Timo Sirainen wrote:> On Fri, 2009-08-14 at 11:28 -0700, Daniel L. Miller wrote: > >> What would be involved in implementing SIS within Dovecot? A new or >> modified mailbox format? >> > > It could be added to dbox without too much trouble. I already kind of > planned for it: > > /* Pointer to external message data. Format is: > 1*(<start offset> <byte count> <ref>) */ > DBOX_METADATA_EXT_REF = 'X', > > Nothing uses that yet though. So what you'd need is: >Now do we need to implement some kind of external database for tracking the attachments between mailboxes? Any thoughts on what that should look like? -- Daniel
Quoting Timo Sirainen <tss at iki.fi>:> 1) When writing the data, extract the attachments and write them to > different files. Add pointers to those files to the EXT_REF metadata. > Dovecot's message parsers should make this not-too-difficult to > implement.I'd rather it did mime parts, rather than attachments. In my use case, we don't get attachments distrubuted as widely as we get messages distributed. If the local mailbox had the headers, but the SIS area had the mime parts, this would save tons of space. Since attachments are mime-parts, this works for both cases...> 3) Once that works, figure out a way to implement SIS for those > externally written files. Creating hard links to files a global > attachment storage would probably be a workable solution. Then scan > through the storage once in a while and delete files with link count=1.Hardlinks is one way, for filesystems that support it. But it does have limits (can't span volumes, etc). But any kind of setup that can maintain the file and a usage count should work (and the two don't have to be kept together, though they can). If you add a management interface, all the better. BTW, PMDF implemented all this eons ago (in their popstore I think it was) system added around PMDF 5 or 6.... Was actually pretty nice, in particular for the times (this was the 1990's). Anyway, my $0.02 worth, not that I'm waiting on this feature, but it sure would save me tons of disk space if I had it... -- Eric Rostetter The Department of Physics The University of Texas at Austin This message is provided "AS IS" without warranty of any kind, either expressed or implied. Use this message at your own risk.