dovecot - Jul 2010 - Doveadm protocol for parallel user processing

A lot of doveadm commands process users' mails. They can optionally
process either a single user or multiple users (e.g. -A = all, -u
'*@domain.org'). For example to expunge old mails from everyone's
Trash,
you'd use:

doveadm expunge -A mailbox Trash savedbefore 30d

The problem is, this doesn't work with the new director way of doing
proxying. Only a single server can access a user's mails safely, so this
command can't be run safely if there are multiple mail servers. It's
also a bit problematic for regular proxying, but it's especially
problematic with director: Since users aren't permanently assigned
anywhere, it's not even possible to build a user <-> host mapping and
run the command only for those users in their assigned server.

First I started thinking about adding another lib-storage backend that
supports proxying to another server, maybe via IMAP or maybe via another
protocol. Maybe I should do that some day anyway, but it seems a bit
complex for this task.

Then I also realized that even if all users are on local server, it
would probably finish faster if doveadm could handle users in parallel
in separate processes (due to disk I/O latencies and maybe other
latencies).

So, it seems like it would be a good idea to make doveadm internally a
client/server architecture. When you start doveadm, it would be only the
"brain" part, which connects to the worker doveadms either via local
UNIX socket or to remote servers. Admin could configure how many doveadm
worker processes the brain should connect per server (could be a
per-server setting).

So, what's needed to make this happen?

1. The actual client/server split and the protocol. Probably can be
really simple and won't be anything interesting. Or maybe it becomes
more interesting once I start thinking about it more, we'll see. :)

2. Authentication.

3. TLS-protection? After merging code from dovecot-2.0-sslstream branch
this should be pretty easy. Although Dovecot should finally then add
support for checking that CommonName/etc match the server name.

Authentication
--------------

For local UNIX sockets that are 0600 root, the authentication wouldn't
be necessary. It could internally check that if the listener socket has
these permissions, it would just announce that doveadm is already
authenticated.

Otherwise, and especially for TCP connections, there should be
authentication. I was thinking about using already existing master users
for this. Doveadm client says "I'm <masteruser>, my password is
<pass>,
and I want to access <user>'s mails". Auth process does the
authentication and replies either success or failure, just like for
similar IMAP/POP3/etc logins.

It's possible to restrict what users a master user can log in as, like
admin at domain.org could only log in as *@domain.org. This means that you
could also enable doveadm protocol for external users, and let the
domain administrators use doveadm directly to manage their own domain's
users.

PLAIN authentication mechanism would be easy to support. But this might
be a good time to also introduce client side SASL library to Dovecot to
support other auth mechanisms. But then again, maybe it doesn't really
matter..

Privilege dropping
------------------

I hate it that with multiple UIDs doveadm processes currently have to be
run as root and they only temporarily drop privileges. With
client/server split it would be possible to get rid of this. Each
doveadm worker process would then handle only a single user and then
exit. This would be optional similar to how login processes are now,
e.g.:

service doveadm {
  # 1 = default = one user access per process, 0 = unlimited user accesses
  service_count = 1
}

LMTP privilege dropping
-----------------------

Completely unrelated to doveadm, but due to the above doveadm privilege
dropping thinking I also started thinking about LMTP. It's also annoying
that it runs as root, and only temporarily drops privileges. It could
also avoid this by saving a mail only to a single user per process. The
problem is that there can be multiple RCPT TO:s in a same session, but
this can be handled by treating them similarly than when using LMTP
proxying: Just connect to localhost again and proxy the mail there for
each user separately. This could also be optionally done only when the
process UID differs from the destionation user's UID.

When?
-----

These are too invasive changes to v2.0, so I guess they'll wait until
v2.1. Luckily there won't be any hugely destabilizing changes going into
v2.1, so maybe v2.1.0 release won't be too many months after v2.0.0. :)

On Thu, 2010-07-22 at 15:43 +0100, Timo Sirainen wrote:
> For local UNIX sockets that are 0600 root, the authentication wouldn't
> be necessary. It could internally check that if the listener socket has
> these permissions, it would just announce that doveadm is already
> authenticated.
Or slightly better: It's preauthenticated if the socket is 0600 and its
user matches service doveadm { user }. So if only a single UID is used
for all users, there's no need to start doveadm as root or to
authenticate if both the brain and workers start with same UID.

On Thu, 2010-07-22 at 15:43 +0100, Timo Sirainen wrote:
> These are too invasive changes to v2.0, so I guess they'll wait until
> v2.1. Luckily there won't be any hugely destabilizing changes going
into
> v2.1, so maybe v2.1.0 release won't be too many months after v2.0.0. :)
The changes weren't very invasive after all, so I just added it to v2.0
today: http://hg.dovecot.org/dovecot-2.0/rev/b60e225386bf

Also this is disabled by default, so the potential bugs happen only when
doveadm_worker_count is set to non-zero. I didn't yet write support for
authentication, but it should be pretty easy to add next week.