http://hg.dovecot.org/dovecot-2.2/rev/64c73e6bd397 Today I finally committed the "imap-hibernate" feature that I first started developing about a year ago (and had been thinking about for several years before that). The main purpose here is to reduce the number of imap processes and the amount of memory they use by moving IDLEing connections into imap-hibernate processes where they are waiting for something to happen (changes to the mailbox or IMAP client to do something). Mailbox changes are noticed only if inotify/kqueue is enabled, so without them the hibernation feature won't work well. For now only the IMAP connections that are in IDLE can be hibernated. In theory non-IDLEing connections could also be hibernated, but the main problem here is that if EXPUNGEs can't be sent at the time the hibernation is over, the current code can't handle it. They would require recreating a view that contains such expunged messages, which could be possible with some extra work. Some IMAP extensions aren't supported yet: - NOTIFY: This would just need to import/export the NOTIFY state. Also hibernation could be done at any time if VANISHED replies could be sent to the selected mailbox. - SEARCH=CONTEXT: A bit more troublesome extension to support. I'm not sure how big the trouble would be. It's rarely used though. - COMPRESS: The [de]compression code would need to be moved to login process instead of being done in imap-compress plugin. This would be a larger change, although it's also useful for various other purposes. The hibernation can be enabled for now simply by setting imap_hibernate_timeout setting to non-zero, which controls how long to wait after client has issued IDLE (and nothing happens) until the process hibernates. Some future work might include: - Moving the hibernation all the way to the first Dovecot proxy layer to avoid the need for imap-hibernate process at all. (Also moving the COMPRESS handling all the way there.) - Dynamically adjusting when the client should be hibernated instead of just a fixed imap_hibernate_timeout setting. Especially try to avoid such a situation where the hibernation is almost always terminated soon after it started. Although that probably is unlikely, so maybe not worth the trouble to prevent.. - Hibernation could be used to control the load of the server. If it's running too slow or if there are already too many imap processes, some of the connections could be moved to hibernation stage. Or just kept in hibernation for a while longer even after the imap process would normally be recreated (e.g. wait for several minutes before reporting new emails, or several seconds before replying to DONE). - The EXPUNGE/VANISHED and FETCH FLAGS replies aren't sent asynchronously, but instead fully buffered to the output buffer. This works as long as the list of changes after hibernation isn't huge, but otherwise it's wasting a lot of memory. - Maybe support non-inotify/kqueue change notifications from lib-storage. This could be complicated though to do efficiently. Or a simple way would be to just send a UDP or UNIX datagram packet on all changes to all the imap-hibernate processes and see if anybody cares about those. I'm not sure how efficient that would be.
> http://hg.dovecot.org/dovecot-2.2/rev/64c73e6bd397==> /var/log/dovecot/dovecot.log <=Aug 25 09:42:07 nihlus dovecot: imap(tlx at leuxner.net): Error: net_connect_unix(/var/run/dovecot/imap-hibernate) failed: Permission denied Aug 25 09:42:07 nihlus dovecot: imap(tlx at leuxner.net): Error: Couldn't hibernate imap client: Couldn't export state: Virtual mailboxes have no GUIDs Aug 25 09:42:07 nihlus dovecot: imap(tlx at leuxner.net): Error: net_connect_unix(/var/run/dovecot/imap-hibernate) failed: Permission denied $ l /var/run/dovecot/ total 8 srw------- 1 root root 0 Aug 24 18:22 anvil srw------- 1 root root 0 Aug 24 18:22 anvil-auth-penalty srw------- 1 dovecot root 0 Aug 25 09:17 auth-client srw------- 1 dovecot root 0 Aug 25 09:17 auth-login srw------- 1 root root 0 Aug 25 09:17 auth-master -rw------- 1 root root 32 Aug 7 12:03 auth-token-secret.dat srw-rw-rw- 1 dovecot root 0 Aug 25 09:17 auth-userdb srw------- 1 doveauth root 0 Aug 25 09:17 auth-worker srw------- 1 root root 0 Aug 25 09:17 config srw------- 1 root root 0 Aug 25 09:17 dict srw------- 1 root root 0 Aug 25 09:17 director-admin srw------- 1 root root 0 Aug 25 09:17 director-userdb srw-rw-rw- 1 root root 0 Aug 25 09:17 dns-client srw------- 1 root root 0 Aug 25 09:17 doveadm-server lrwxrwxrwx 1 root root 25 Aug 24 18:22 dovecot.conf -> /etc/dovecot/dovecot.conf drwxr-xr-x 2 root root 40 Aug 7 12:03 empty srw------- 1 root root 0 Aug 25 09:17 imap-hibernate srw------- 1 root root 0 Aug 25 09:17 imap-master srw-rw-rw- 1 root root 0 Aug 25 09:17 imap-urlauth srw------- 1 dovecot root 0 Aug 25 09:17 imap-urlauth-worker srw-rw-rw- 1 root root 0 Aug 25 09:17 indexer srw------- 1 dovecot root 0 Aug 25 09:17 indexer-worker srw------- 1 root root 0 Aug 25 09:17 ipc srw-rw-rw- 1 root root 0 Aug 25 09:17 lmtp srw------- 1 root root 0 Aug 25 09:17 log-errors drwxr-x--- 2 root dovenull 120 Aug 25 09:17 login -rw------- 1 root root 6 Aug 24 18:22 master.pid srw------- 1 root root 0 Aug 25 09:17 replication-notify prw------- 1 root root 0 Aug 25 09:17 replication-notify-fifo srw------- 1 dovecot root 0 Aug 25 09:17 replicator srw-rw-rw- 1 root root 0 Aug 25 09:17 ssl-params srw------- 1 root root 0 Aug 25 09:17 stats prw------- 1 vmail root 0 Aug 25 09:43 stats-mail drwxr-x--- 2 root dovenull 80 Aug 25 09:17 token-login The socket is created as ?root? per default. Does it need to run under another user to mitigate the permissions issue? Regards Thomas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 204 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://dovecot.org/pipermail/dovecot/attachments/20150825/c609433a/attachment.sig>
* Thomas Leuxner <tlx at leuxner.net> 2015.08.25 09:45:> > http://hg.dovecot.org/dovecot-2.2/rev/64c73e6bd397 > > ==> /var/log/dovecot/dovecot.log <=> Aug 25 09:42:07 nihlus dovecot: imap(tlx at leuxner.net): Error: net_connect_unix(/var/run/dovecot/imap-hibernate) failed: Permission denied > Aug 25 09:42:07 nihlus dovecot: imap(tlx at leuxner.net): Error: Couldn't hibernate imap client: Couldn't export state: Virtual mailboxes have no GUIDs > Aug 25 09:42:07 nihlus dovecot: imap(tlx at leuxner.net): Error: net_connect_unix(/var/run/dovecot/imap-hibernate) failed: Permission denied'chmod 666' mitigates the permission issue on the socket. However it seems to have other issues then: ==> /var/log/dovecot/dovecot.log <=Aug 25 09:56:49 nihlus dovecot: imap-hibernate: Error: Failed to parse client input: Invalid peer_ip value: Aug 25 09:56:49 nihlus dovecot: imap(tlx at leuxner.net): Error: /var/run/dovecot/imap-hibernate returned failure: Failed to parse client input: Invalid peer_ip value: Aug 25 09:56:49 nihlus dovecot: imap: Error: Aug 25 09:56:49 nihlus dovecot: imap-hibernate: Error: Failed to parse client input: Invalid peer_ip value: Aug 25 09:56:49 nihlus dovecot: imap(tlx at leuxner.net): Error: /var/run/dovecot/imap-hibernate returned failure: Failed to parse client input: Invalid peer_ip value: Aug 25 09:56:49 nihlus dovecot: imap: Error: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: Digital signature URL: <http://dovecot.org/pipermail/dovecot/attachments/20150825/e4fc0fc3/attachment.sig>
> Le 25 ao?t 2015 ? 01:27, Timo Sirainen a ?crit : > > http://hg.dovecot.org/dovecot-2.2/rev/64c73e6bd397 > > Today I finally committed the "imap-hibernate" feature that I first started developing about a year ago (and had been thinking about for several years before that). The main purpose here is to reduce the number of imap processes and the amount of memory they use by moving IDLEing connections into imap-hibernate processes where they are waiting for something to happen (changes to the mailbox or IMAP client to do something). Mailbox changes are noticed only if inotify/kqueue is enabled, so without them the hibernation feature won't work well. > > [?]Wow! I remember you told us about such a development some times ago; many, many thanks for having persevered. Unfortunately, I won?t be able to test that feature in a near future. In the meantime, I?ll be enthusiastically reading any report our list?s fellows could share. :-) Axel