dovecot - Mar 2008 - Future enhancement of imap: sorting?

After solving a problem with a sweet little application I discovered (more 
on that below), I wanted to get some feedback about a potential 
enhancement for IMAP: standardized, MUA independant, server-side sorting.

Currently, there are two basic solutions for sorting: handle it during 
delivery or handle it in the MUA. However, if you handle it in the MUA, 
you lose some of the client independancy and portability that imap was 
designed to address, as there is no mechanism to sync sorting rules 
between clients, and many lightweight clients don't support sorting at 
all. On the flip side, in distributed mail envrionments sorting at 
delivery time isn't feasible, as you typically have the entire mailspool 
owned by a single UID and mounted via NFS. My basic point is that the two 
paradigms for sorting have fundemental flaws that severaly limit them.

It occured to me that it would be ideal to enable sorting in the imap 
daemon, enabling users to implement sorting rules which are stored in 
their directory (similar to the other imap metadata), and which get 
invoked whenever mail is checked. Perhaps that is beyond the reasonable 
scope of the imap protocol, but in my mind sorting is a critical task that 
is poorly addressed and needs to be standardized. I'd love to hear 
opinions on this.


See, in our mail cluster environment, it's basically impossible to use 
procmail without forwarding your mail to our shell server (nor are we able 
to use the dovecot LDA). So, the question presented to me by one of my 
business partners was, since Outlook was shortsighted and didn't include 
the ability to sync sorting rules between clients, nor did they implement 
sorting functionality whatsoever on the windows mobile version, what 
options does he have to ensure proper sorting of his mail without having 
to leave outlook running 24x7? He doesn't want to have to wade through 
mailing list email on his mobile outlook, nor does he want to have to 
ensure his filter rules are consistent in both his outlook and our 
webmail.

So, I poked around in the ports tree, and found a gem: imapfilter.

http://imapfilter.hellug.gr/

It's configured via lua, supports a very rich array of actions and 
queries, and has proven to be very robust and lightweight. I helped him 
get started with some sample rules, added it to run every minute as a cron 
job, and boom: robust, fully featured, MUA independant filtering in an 
environment that can't support procmail or the dovecot LDA. 

As nice as it is, it's still a personal solution. In my mind we need to 
have standardized, server-side, MUA-independant sorting rules that get 
invoked when a mailbox is checked. Thoughts?

Andy

---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---

On Thu, 2008-03-06 at 20:59 -0500, Andy Dills wrote:
> As nice as it is, it's still a personal solution. In my mind we need to
> have standardized, server-side, MUA-independant sorting rules that get 
> invoked when a mailbox is checked. Thoughts?
Filtering is a better name for this, took me a few paragraphs before I
understood what you meant by sorting. :)

You could already create a plugin that runs a command when it sees new
messages in a mailbox. I'm not really interested in writing this
feature..

A somewhat related feature that I'm going to implement is virtual
mailboxes where messages are shown according to specified search rules.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20080307/cde68329/attachment-0002.bin>

On Fri, 7 Mar 2008, Timo Sirainen wrote:
> On Thu, 2008-03-06 at 20:59 -0500, Andy Dills wrote:
> 
> > As nice as it is, it's still a personal solution. In my mind we
need to
> > have standardized, server-side, MUA-independant sorting rules that get
> > invoked when a mailbox is checked. Thoughts?
> 
> Filtering is a better name for this, took me a few paragraphs before I
> understood what you meant by sorting. :)
Heh, sorry about that. In my mind, sorting implies directing an object to 
a location based on some arbitrary criteria, whereas filtering (in my 
mind) implies rejecting or accepting based on some arbitrary criteria.
> You could already create a plugin that runs a command when it sees new
> messages in a mailbox. I'm not really interested in writing this
> feature..
I didn't mean to imply this as a request for functionality...it would be 
worthless as a feature in dovecot without support in the IMAP protocol and 
thus the MUAs. I was thinking more along the lines of a future enchanement 
for the actual IMAP protocol, as this seems like something that should 
have been addressed but wasn't, leading to an array of flawed solutions. I 
assume you feel that this is beyond the scope of what IMAP should handle 
natively? On one hand it feels like it is, but on the other hand it seems 
like something in desperate need of standardization.

The idea for virtual folders based on search criteria is quite 
interesting, by the way. How would you enable users to create them from 
their client?

Thanks,
Andy

---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---