Patch #10 allows the pop3 and imap mail processes to handle multiple
clients. We know this weakens the security model but it greatly
increases scalability especially when clients are idle.
Here's how it works. When there are no mail processes, or none
serving fewer than mail_max_connections clients, the master creates a
new mail process pretty much like it does now, but with some new
environment variables (PERSISTENT_MAIL_PROCESS and CONNECTION_ID, and
ADVISE_SET[GU]ID instead of RESTRICT_SET[GU]ID) and a unix-domain
socket connection back to the master. For future clients the master
sends the client socket and a full dump of the environment via that
connection. Persistent mail processes multiplex I/O for their
connected clients and switch the whole environment and the effective
user ID, group ID, and supplemental groups each time. The result is
that most of dovecot's code and assumptions remain unchanged: the
mail process executes imap/pop commands with the permissions of the
user and with nearly the same environment so getenv() still works as
expected. Some assumptions do change, for instance global state
variables like last_partial and quota_set are forbidden, and must be
stored per-client or per-user. Also it's no longer OK just to exit or
panic on error, for that disconnects multiple users not just one.
Some key entry points in the patch are create_mail_process() in mail-
process.c and io_env_switch() in ioloop.c.
Notes about this patch:
- The base for this patch is dovecot-1.1.7 + Apple patch 9, not
because this patch needs Open Directory but simply because it adds one
line to the Apple attributions in COPYING.
- This patch depends on Apple patch 7 (hash_table_create/destroy).
- This patch introduces the following new config options for the pop3
and imap protocols:
- mail_process_per_connection = yes
- mail_max_connections = 20
The default value of mail_process_per_connection (yes) preserves the
current secure model. Changing it to no allows multiplexing, with up
to mail_max_connections simultaneous clients per process. See dovecot-
example.conf for more info.
- It restricts the mail_executable config option to minimize
unpleasant surprises. When mail_process_per_connection = no the
communication protocol between the master process and the mail
processes changes as described above. A naive third-party
mail_executable might break. See dovecot-example.conf for more info.
- Some changes are intentionally awkward in order to minimize code
deltas to simplify merges (of new dovecot releases into our source
tree). These are so marked. Feel free to clean them up.
- Changes tagged with "APPLE" are ours, including the whole new
directory mail-common. All the untagged changes, including the entire
contents of the new files mail-user.[ch], are copied straight from
dovecot-1.2 (alpha3, I believe). We needed dovecot-1.2's support for
multiple users in order to handle quotas correctly, and we needed it
in 1.1. Specifically, we copied these changes from dovecot-1.2 (and a
couple others too I think):
8082 http://hg.dovecot.org/dovecot-1.2/rev/db66611fd195
Added struct mail_user and fixed the code to support multiple
users per process.
8084 http://hg.dovecot.org/dovecot-1.2/rev/f12f8c1da0bf
Forgot to add mail-user.* files in previous struct mail_user
commit.
8085 http://hg.dovecot.org/dovecot-1.2/rev/bf83aa9c3f4a
Removed pool parameter from mail_namespaces_init*(). Use
mail_user's pool intead.
8091 http://hg.dovecot.org/dovecot-1.2/rev/ceca59aaae89
quota-fs: compile fix for previous changes.
8096 http://hg.dovecot.org/dovecot-1.2/rev/f35a8a3dc06d
Fixed FS quota compiling and Maildir++ quota with multiple users.
8109 http://hg.dovecot.org/dovecot-1.2/rev/e7929190cd32
fts-solr: Fixed compiling with recent struct mail_user changes.
8137 http://hg.dovecot.org/dovecot-1.2/rev/b2a258213ee0
Created mail_user_[try_]home_expand(). Used them for expanding
mail directories.
8294 http://hg.dovecot.org/dovecot-1.2/rev/8aa69e3d27ef
Trash plugin: Assign storage to all mailboxes at startup so
errors are caught immediately.
Also previous optimization change broke trash plugin when using
multiple
mail_users. This change fixes it to work again.
If a change in the patch is not marked somehow with APPLE then it
comes from dovecot-1.2.
- Some other data structures also needed to be made per-client/user,
for instance last_partial from imap-fetch-body.c, quota_set from quota-
plugin.c, and process_primary_gid etc. from restrict-access.c.
- Sending the master process SIGINFO (on platforms that support that
signal) makes it print its view of all the clients connected to all
the mail processes. Sending SIGINFO to an individual mail process
makes it print its view. These are not necessarily always the same.
For instance, on a config reload (SIGHUP), the master disconnects from
the mail processes but leaves them running. The output is admittedly
kinda geeky.
- The logging tag for a persistent mail process is just "*" instead of
the user name, since a process can serve multiple users. Some
individual messages (such as connect/disconnect) identify the user.
- The expire plugin uses global state variables which are incompatible
with persistent mail processes. The patch detects trouble but does
not fix it because we don't need it. We leave that as an exercise for
you :).
- The dict client interface is blocking which is unfortunate for
persistent mail processes. The patch does not address this, since the
delay is probably small and we don't use it anyway.
- Some of the accounting is gnarly because of the need to honor
mail_max_userip_connections in a world where a single user can have
any number of active connections on any number of mail processes.
Finally, here's the patch.
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: patch10.txt
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20090106/82abaed9/attachment-0002.txt>
-------------- next part --------------