dovecot - Jan 2010 - Mail process plans: multiple connections in a process + idle processes

Maybe for v2.1:

imap and pop3 processes can currently handle multiple connections, but
they're assigned pretty much randomly to processes. It would be better
if it was possible to:

a) Require that a process handles only a specific user's connections

b) Require that a process handles only a specific UNIX UID's connections
(e.g. if each domain has a separate UID)

c) Prefer to put user's connections to same process, but don't require
it.

To do this, there needs to be a process between imap/pop3-login and
imap/pop3. Let's call it imap/pop3-tracker process (or aggregator? or
any suggestions?). This process accepts all connections coming from
login processes when they've finished authenticating a connection.
Internally it figures out if the connection should go to a new or an
existing process. It keeps track of 

username -> { mail_process[], cookie (see idle processes below) }
mail_process -> { unix_socket_fd, connection_count }

So there's a connection to each mail process. This connection is used to
send connection fds for the process to handle. It's also used to receive
notifications when a connection is closed.

The mail process sockets could also be used to communicate to a specific
process. For example doveadm could use this in future to ask what a
specific user or process is currently doing, by connecting to
imap-tracker, which forwards the request to wanted process(es).


The above isn't really anything new, just a more specific plan about
what I had been thinking for a long time. A new and more interesting
idea is to gather IDLEing imap connections under a few imap-idle
processes:

When a connection has been IDLEing for a few minutes (or something), and
there's no special state in the connection (e.g. CONTEXT=SEARCH
searches), the connection could be moved to a separate imap-idle process
that simply waits for something to happen. When something does happen,
it'll move the connection back to imap process, which restores state and
continues. The idea would be to reduce the number of idling imap
processes that are wasting memory.

The state that needs to be saved and restored is:

 - selected mailbox name, GUID, UIDVALIDITY (for reliably selecting back
the original mailbox, or reliably fail if it's gone)
 - flag: mailbox is EXAMINEd, not SELECTed
 - highest seen modseq
 - UIDs of \Recent messages
 - SEARCHRES extension: last saved SEARCH result
 - List of keywords that had been advertised in FLAGS/PERMANENTFLAGS
reply to client. Or probably MD5 sum of it would be enough. If MD5 had
changed, the list would be re-sent to client.
 - connection fd
 - paths that are being monitored with [id]notify/kqueue

Connection is moved back when a) new command is received from connection
fd, b) one of the monitored files changes. The connections are moved
back via imap-tracker process, by sending the saved state along the
request. For security there's a per-user security cookie (128 bits of
randomness), which is sent to imap processes, and when restoring
connection from imap-idle, the request must provide the correct cookie
or imap-tracker fails the login. (Normally new post-login imap/pop3
connections can't be created without having authenticated first via
dovecot-auth. They verify it.)

So when connection comes back, imap process opens the mailbox based on
saved information. If this fails for some reason (e.g. mailbox deleted),
client gets disconnected. Next Dovecot gets a list of changes to mailbox
since the saved modseq, and based on that sends the missing FETCH,
EXPUNGE, EXISTS and RECENT events. Again if it somehow fails, the client
gets disconnected.

At least initially this would only be done for connections that are
running IDLE command, because non-IDLE connections are a lot more
difficult to restore to their original state. This is mainly because if
a message gets expunged, it's still visible to the connection until it
issues a command that allows sending EXPUNGE event. To handle that,
there would have to be code that allows building a view of mailbox where
there are some expunged messages. Also it can be difficult to find out
the expunged messages' flags, unless they're also part of the saved
state.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20100123/dd5a04eb/attachment-0002.bin>

On Sat, 2010-01-23 at 13:42 +0200, Timo Sirainen wrote:
> c) Prefer to put user's connections to same process, but don't
require
> it.
This + async io as described in my other recent "plan" mail should
make
it possible to handle hundreds of connections in a process without I/O
waits. And combined with imap-idle processes, there could be just 10 or
so imap processes always running and handling thousands of connections,
with disk I/O being the only bottleneck :)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20100123/05ff451b/attachment-0002.bin>