dovecot - Sep 2011 - Webmail architecture : questions about cache and storage backend

Hi,

I have some questions about dovecot-2.1 and need some advice about 
software architecture.
My company plans to develop a kind of Document Managing System (DMS) 
and one of its module is a webmail
that is currently in a pre-alpha stage (if you are curious you can have 
a look here
http://kairos.smartmobili.com/ but don't be surprised if it doesn't 
work and please
do not use IE for now).
 From my personal analysis that I am sure is incomplete we have the 
following options to handle
emails storage and cache:

1) Do not use a cache and rely on dovecot performance to retrieve 
email's header in real time.
    Actually don't know if it's that simple because we might need to 
have a transformed version
    of emails to be able to display them more easily in the webmail. In 
this case this option is
    not really possible.

2) Use new dovecot-2.1 imap backend and develop a kind of dummy imap 
server listening to a custom port
    and storing emails inside a mongoDB database. Advantage is it allow 
replication very easily.

3) Wait for dovecot2.x to implement a NoSQL storage. Will it happen 
soon ?

If you have other suggestions or ideas I would be happy to hear them.

Vincent R.

On Wed, 2011-09-21 at 14:04 +0200, forumer at smartmobili.com wrote:
> 1) Do not use a cache and rely on dovecot performance to retrieve 
> email's header in real time.
>     Actually don't know if it's that simple because we might need
to
> have a transformed version
>     of emails to be able to display them more easily in the webmail. In 
> this case this option is
>     not really possible.
I'm not really sure what you mean by this. Dovecot can cache individual
headers and make fetching them fast.
> 2) Use new dovecot-2.1 imap backend and develop a kind of dummy imap 
> server listening to a custom port
>     and storing emails inside a mongoDB database. Advantage is it allow 
> replication very easily.
I suppose that would be the easiest way to do it currently. But note
that you can't get all of Dovecot's features to work through imapc
backend. Especially the concept of message and mailbox GUIDs doesn't
exist in IMAP protocol (some kind of a new X-DOVECOT extension could be
added for those of course).

Also implementing a bug-free IMAP server may not be as easy as you
think, even though Dovecot uses only a small subset of IMAP commands. Be
sure to stress test Dovecot using imaptest: http://imapwiki.org/ImapTest
(I've noticed this week that Exchange totally screws up its internal
IMAP state when stress testing.)
> 3) Wait for dovecot2.x to implement a NoSQL storage. Will it happen 
> soon ?
I hope so! :) But I can't really say when.. My priorities currently are:

1. Do whatever paying customers want (not that much currently)
2. Finish v2.1.0 (still quite a lot of small things to do)
3. Create v2.2 branch where lib-storage API is changed to support more
asynchronous lookups, and modify imapc backend to use those (= support
handling commands in parallel)
4. Probably change dbox code to use lib-fs, or maybe create a new
backend using lib-fs
5. Implement NoSQL backend for lib-fs

Actually 3 isn't really necessary for 4-5, but it improves the
performance.