I merged all the new features and latest v1.1 changes under one tree: http://hg.dovecot.org/dovecot-1.2/ The changeset order is kind of weird though. The timestamps jump all around. Wonder if there's a way to reorder them so that it looks more sane.. The status of the new features: 1. CONDSTORE extension is probably the largest change. It adds new "modification sequences" for messages that increase whenever the message's metadata changes. I'll probably have to reimplement the way modseqs are calculated, because modseqs will be very useful when implementing replication and the current way just doesn't work with it. If modseq-supporting clients see the current modseqs and later the server gets upgraded to new modseqs, the clients will most likely break. So this change should be done for v1.2. 2. QRESYNC, ESEARCH, SEARCHRES and WITHIN extensions are fully implemented. QRESYNC might need some cleanups and optimizations though. 3. CONTEXT=SEARCH extension is fully implemented, but could use a few optimizations (caching PARTIAL and CONTEXT searches, maybe others too). 4. Virtual mailboxes should work fast after mailbox is opened. The initial opening could use several optimizations though. It could probably share some code with QRESYNC to avoid the full initial search (storing each backend's modseq to index header). Also if search parameters don't contain any dynamically changing data, there's no point in searching the old messages. The current design doesn't allow changing the search parameters or list of mailboxes, otherwise it breaks more or less badly. I guess I could add code to check if the dovecot-virtual file's mtime has changed and if so make it do a full resync. This anyway means that there's no way to support wildcard mailbox names (e.g. "all mailboxes"). But does anyone really want that (yet)? It'll anyway be faster/easier to implement once mailbox list indexes are implemented. I'll still have to add a new X-MAILBOX search parameter which can be used to test what the backend mailbox name is. This will be especially useful with INTHREAD extension. I guess it wouldn't hurt to have FETCH X-MAILBOX if someone wants it. 5. Thread indexes should work, but some optimizations missing and there are some small bugs left. X-REFERENCES2 thread algorithm works. 6. INTHREAD extension isn't started yet, but I'll start it soon. Hopefully won't be too tricky to get it working with virtual mailboxes and CONTEXT=SEARCH.. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: <http://dovecot.org/pipermail/dovecot/attachments/20080609/c701904d/attachment-0002.bin>
Updates: On Mon, 2008-06-09 at 05:51 +0300, Timo Sirainen wrote:> I merged all the new features and latest v1.1 changes under one tree: > > http://hg.dovecot.org/dovecot-1.2/Nightly snapshots are also from v1.2 code tree nowadays.> 1. CONDSTORE extension is probably the largest change. It adds new > "modification sequences" for messages that increase whenever the > message's metadata changes. > > I'll probably have to reimplement the way modseqs are calculated, > because modseqs will be very useful when implementing replication and > the current way just doesn't work with it. If modseq-supporting clients > see the current modseqs and later the server gets upgraded to new > modseqs, the clients will most likely break. So this change should be > done for v1.2.Modseq changes are implemented. The only issue with CONDSTORE is that STORE UNCHANGEDSINCE command doesn't atomically check-and-update. Implementing the atomicity should be pretty easy since there is a similar check already in the code. The largest issue with it is changing APIs enough to support returning back which messages failed the STORE. Still should be pretty easy.> 4. Virtual mailboxes should work fast after mailbox is opened. The > initial opening could use several optimizations though. It could > probably share some code with QRESYNC to avoid the full initial search > (storing each backend's modseq to index header). Also if search > parameters don't contain any dynamically changing data, there's no point > in searching the old messages.Implemented initial opening optimizations. I haven't done much testing though, other than it appears not to crash and appears to work with simple tests. :) So the current implementation should be as fast as it's possible to make it.> The current design doesn't allow changing the search parameters or list > of mailboxes, otherwise it breaks more or less badly. I guess I could > add code to check if the dovecot-virtual file's mtime has changed and if > so make it do a full resync. This anyway means that there's no way to > support wildcard mailbox names (e.g. "all mailboxes"). But does anyone > really want that (yet)? It'll anyway be faster/easier to implement once > mailbox list indexes are implemented.Changing mailbox list is now detected and handled, as well as UIDVALIDITY changing in mailboxes. Mailbox list wildcards wouldn't be all that difficult to implement anymore if someone wants them, but until then I don't think I'll bother. ?Changing search parameters still isn't detected though. Maybe it could store a MD5 sum of the search parameters in the header and if it changes rebuild the entire mailbox.> I'll still have to add a new X-MAILBOX search parameter which can be > used to test what the backend mailbox name is. This will be especially > useful with INTHREAD extension. I guess it wouldn't hurt to have FETCH > X-MAILBOX if someone wants it.Oh, almost forgot about this one.> 6. INTHREAD extension isn't started yet, but I'll start it soon. > Hopefully won't be too tricky to get it working with virtual mailboxes > and CONTEXT=SEARCH..This one is the last major unimplemented? v1.2 feature. After that I'll start finishing, optimizing and stabilizing the features for a v1.2 release (as well as start v2.0/replication coding). I'm hoping for v1.2.0 release by the end of this summer. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: <http://dovecot.org/pipermail/dovecot/attachments/20080618/6799b2e6/attachment-0002.bin>
Hi Timo, First of all, dovecot is great! :) Question on CONDSTORE. I haven't re-read RFC to confirm, isn't CONDSTORE operates under switch mode with command ENABLE? So that IMAP client needs to request such capability. Maybe I mixed up with another IMAP command. Thanks Joseph Timo Sirainen wrote:> Updates: > > On Mon, 2008-06-09 at 05:51 +0300, Timo Sirainen wrote: >> I merged all the new features and latest v1.1 changes under one tree: >> >> http://hg.dovecot.org/dovecot-1.2/ > > Nightly snapshots are also from v1.2 code tree nowadays. > >> 1. CONDSTORE extension is probably the largest change. It adds new >> "modification sequences" for messages that increase whenever the >> message's metadata changes. >> >> I'll probably have to reimplement the way modseqs are calculated, >> because modseqs will be very useful when implementing replication and >> the current way just doesn't work with it. If modseq-supporting clients >> see the current modseqs and later the server gets upgraded to new >> modseqs, the clients will most likely break. So this change should be >> done for v1.2. > > Modseq changes are implemented. The only issue with CONDSTORE is that > STORE UNCHANGEDSINCE command doesn't atomically check-and-update. > Implementing the atomicity should be pretty easy since there is a > similar check already in the code. The largest issue with it is changing > APIs enough to support returning back which messages failed the STORE. > Still should be pretty easy. > >> 4. Virtual mailboxes should work fast after mailbox is opened. The >> initial opening could use several optimizations though. It could >> probably share some code with QRESYNC to avoid the full initial search >> (storing each backend's modseq to index header). Also if search >> parameters don't contain any dynamically changing data, there's no point >> in searching the old messages. > > Implemented initial opening optimizations. I haven't done much testing > though, other than it appears not to crash and appears to work with > simple tests. :) So the current implementation should be as fast as it's > possible to make it. > >> The current design doesn't allow changing the search parameters or list >> of mailboxes, otherwise it breaks more or less badly. I guess I could >> add code to check if the dovecot-virtual file's mtime has changed and if >> so make it do a full resync. This anyway means that there's no way to >> support wildcard mailbox names (e.g. "all mailboxes"). But does anyone >> really want that (yet)? It'll anyway be faster/easier to implement once >> mailbox list indexes are implemented. > > Changing mailbox list is now detected and handled, as well as > UIDVALIDITY changing in mailboxes. Mailbox list wildcards wouldn't be > all that difficult to implement anymore if someone wants them, but until > then I don't think I'll bother. > > ?Changing search parameters still isn't detected though. Maybe it could > store a MD5 sum of the search parameters in the header and if it changes > rebuild the entire mailbox. > >> I'll still have to add a new X-MAILBOX search parameter which can be >> used to test what the backend mailbox name is. This will be especially >> useful with INTHREAD extension. I guess it wouldn't hurt to have FETCH >> X-MAILBOX if someone wants it. > > Oh, almost forgot about this one. > >> 6. INTHREAD extension isn't started yet, but I'll start it soon. >> Hopefully won't be too tricky to get it working with virtual mailboxes >> and CONTEXT=SEARCH.. > > This one is the last major unimplemented? v1.2 feature. After that I'll > start finishing, optimizing and stabilizing the features for a v1.2 > release (as well as start v2.0/replication coding). I'm hoping for > v1.2.0 release by the end of this summer.
Timo Sirainen wrote:> This one is the last major unimplemented? v1.2 feature.Can I make a very weak suggestion to look at that ZLIB compression extension I think you mentioned in the past? The motivation is that I find my "8 mbit broadband" link seems to saturate at quite low numbers of headers per second when Thunderbird is pulling down new mailbox messages. As you know on most of my machines I use our compression proxy application which is very noticably increasing my mailbox access speeds even on cutting edge broadband (for europe). Now whilst probably zero clients implement the compression extension this is also a chicken/egg thing so we could start by having a working implementation on the server end at least Second reason is that this suggests that a typical rented server with a meagre 100mbit connection could be network limited while replicating, rather than being network or CPU bound. A lightly compressed protocol *might* be a win even on fairly fast connections simply because many of the imap command outputs seem to compress extremely well (13:1 is typical based on the rather inefficient way OE accesses IMAP and 4:1 average is very normal even for more efficient implementations - YMMV) Anyway, just a thought - I'm assuming that the probable implementation is going to be fairly simple. I would think that zlib and/or lzo would be good compressors if there is a choice of implementations? Certainly LZO would be a good choice for faster 100mbit connections Ed W