dovecot - Apr 2009 - Dovecot, LVS and the issues I have with it.

We run around 5 dovecot (debian etch 1.0.rc15) POP/IMAP 'nodes' using 
the LVS load balancer and an NFS based SAN. it works pretty well. I love 
the robustness of load balancing POP/IMAP.  We do a reasonable amount of 
throughput through these especially at peak times pushing our SAN to 
around 1.5k IOP/s

We currently have two issues with this setup. One of which is NFS index 
corruption issues we get due to NFS/dovecot locking. Basically the UUID 
list or a .index gets corrupt. This causes a full re-indexing of the 
mailbox / broken mailbox until i delete the indexes. In the UUID lists 
case the symptom tends to effect use who use POP rather than IMAP and 
insist on keeping messages on the server. Because it's corrupt it gets 
rebuilt one way or the other and the users email client proceeds to 
redownload the entire mailbox again until he remarks them to be saved. 
This tends to annoy the user a lot. After a bit of testing we do however 
expect this to be fixed by version 1.1. However if anyone has any 
comments on this I would certainly be interested.

The other issue is a little tricky to describe or even log effectively. 
Occasionally a node basically receives more connections than it is able 
to handle. The connection count goes through the roof for that node. It 
will go up way beyond 150 dovecot authentication threads and 100 or so 
active POP/IMAP dovecot threads. The IO_wait/cpu and memory usage also 
tends to be spiking at this point. The server gets to the tipping point 
where it can no longer serve it's POP/IMAP requests fast enough compared 
to the number of connections it's getting. I'd be fine with this but 
this creates some less than desirable symptoms:

1. We obviously reach the auth thread cap eventually so any new auth 
requests basically get refused by the server. To the user this is a seen 
as an outlook/mail client re-auth pop-up request. This annoys them. 
Ideally if the server stops accepting auth requests it should fall off 
our load balancer until it can consistently accept them again. Since the 
LVS detects a node fail by whether the tcp port is still open this 
doesn't happen, since dovecot keeps the port open. This is obviously 
more an LVS issue and not for this mailing list I expect unless anyone 
has any config tweaking tips here?

2. Now here's my real gripe. Dovecot does not handle running out 
resources very gracefully at all in our setup. It does start killing 
threads after a while. I get multiple *"dovecot: child 17989 (login) 
killed with signal 9". *I'm not exactly sure what's going on here 
because after this all I can see is the machine totally out of memory 
and the kernel starts killing absolutely everything. All services are 
killed (including ssh etc..) and I plug a monitor into the server and 
find the last few lines of the console listing init and other rather 
important things having just been killed. At this point it is a case of 
power cycling the server and all is back to normal again.

I imagine there's not a huge amount of people using dovecot in this way. 
But anyone got any recommendations here? I really like using dovecot in 
this setup it handles it pretty well and the redundancy and 
functionality options it provides have been invaluable.

Neil

O/H neil ??????:
> We run around 5 dovecot (debian etch 1.0.rc15) POP/IMAP 'nodes'
using
> the LVS load balancer and an NFS based SAN. it works pretty well. I 
> love the robustness of load balancing POP/IMAP. We do a reasonable 
> amount of throughput through these especially at peak times pushing 
> our SAN to around 1.5k IOP/s
>
> We currently have two issues with this setup. One of which is NFS 
> index corruption issues we get due to NFS/dovecot locking. Basically 
> the UUID list or a .index gets corrupt. This causes a full re-indexing 
> of the mailbox / broken mailbox until i delete the indexes. In the 
> UUID lists case the symptom tends to effect use who use POP rather 
> than IMAP and insist on keeping messages on the server. Because it's 
> corrupt it gets rebuilt one way or the other and the users email 
> client proceeds to redownload the entire mailbox again until he 
> remarks them to be saved. This tends to annoy the user a lot. After a 
> bit of testing we do however expect this to be fixed by version 1.1. 
> However if anyone has any comments on this I would certainly be 
> interested.
>
> The other issue is a little tricky to describe or even log 
> effectively. Occasionally a node basically receives more connections 
> than it is able to handle. The connection count goes through the roof 
> for that node. It will go up way beyond 150 dovecot authentication 
> threads and 100 or so active POP/IMAP dovecot threads. The IO_wait/cpu 
> and memory usage also tends to be spiking at this point. The server 
> gets to the tipping point where it can no longer serve it's POP/IMAP 
> requests fast enough compared to the number of connections it's 
> getting. I'd be fine with this but this creates some less than 
> desirable symptoms:
>
> 1. We obviously reach the auth thread cap eventually so any new auth 
> requests basically get refused by the server. To the user this is a 
> seen as an outlook/mail client re-auth pop-up request. This annoys 
> them. Ideally if the server stops accepting auth requests it should 
> fall off our load balancer until it can consistently accept them 
> again. Since the LVS detects a node fail by whether the tcp port is 
> still open this doesn't happen, since dovecot keeps the port open. 
> This is obviously more an LVS issue and not for this mailing list I 
> expect unless anyone has any config tweaking tips here?
Perhaps try the (Weighted) Least Connections algorithm ? That way, every 
member of your cluster would get a roughly equal number of pop/imap 
requests/active connections. Also, give the persistent (-p) option a 
look. If you're sill running out, then perhaps you need to throw more 
hardware/disk spindles/nfsd threads at the problem.

 > Since the LVS detects a node fail by whether the tcp port is still open

That's not entirely true: LVS does not detect anything whatsoever 
regarding the health of the nodes.
ldirectord works that way and instructs LVS via ipvsadm to add or remove 
nodes. Unfortunately ldirectord is very http centered and only does 
plain tcp connection stuff for the rest of the services (https and ftp 
have their own check_ functions too).
Your options:
- It is fairly trivial to add something like ``check_imap'' or 
``check_pop3'' along the lines of ``check_http'' and
``check_ftp'' to
ldirectord, and use that to monitor your actual service and not the tcp 
socket status. Perl Modules Mail::POP3Client, Net::POP3, 
Mail::IMAPClient or Net::IMAP will probably come in handy.
(untested) - use mon instead of ldirectord. it seems to have more 
monitoring options (http://mon.wiki.kernel.org/index.php/Monitors)
(untested) - use heartbeat v2 style resource monitoring

we're running a similar setup and the one thing that troubled us were 
the index files. As soon as we set up each server to maintain its own 
index files on local disk (we could spare the disk space and CPU) and 
not on NFS we were fine. Mail still resides on NFS. YMMV of course.

Best Regards,
Thanos Chatziathanassiou
>
> 2. Now here's my real gripe. Dovecot does not handle running out 
> resources very gracefully at all in our setup. It does start killing 
> threads after a while. I get multiple *"dovecot: child 17989 (login) 
> killed with signal 9". *I'm not exactly sure what's going on
here
> because after this all I can see is the machine totally out of memory 
> and the kernel starts killing absolutely everything. All services are 
> killed (including ssh etc..) and I plug a monitor into the server and 
> find the last few lines of the console listing init and other rather 
> important things having just been killed. At this point it is a case 
> of power cycling the server and all is back to normal again.
>
> I imagine there's not a huge amount of people using dovecot in this 
> way. But anyone got any recommendations here? I really like using 
> dovecot in this setup it handles it pretty well and the redundancy and 
> functionality options it provides have been invaluable.
>
> Neil
>
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3242 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20090406/bf4f61ba/attachment-0002.bin>

On 4/6/2009, neil (neil.townsend at switchmedia.co.uk)
wrote:
> We run around 5 dovecot (debian etch 1.0.rc15) POP/IMAP 'nodes'
using
> the LVS load balancer and an NFS based SAN. it works pretty well. I
> love the robustness of load balancing POP/IMAP.  We do a reasonable
> amount of throughput through these especially at peak times pushing
> our SAN to around 1.5k IOP/s
> 
> We currently have two issues with this setup. One of which is NFS
> index corruption issues we get due to NFS/dovecot locking. Basically
> the UUID list or a .index gets corrupt.
One word: upgrade.

1.0rc15 is totally outdated and unsupported. There have been so many
fixed since then - many of which relate directly to NFS problems - they
are too numerous to list.

Personally, I'd go ahead and move to 1.2 rc2 (once you've got it working
on a test server), but you need to go to at least 1.1.13...

-- 

Best regards,

Charles

On Mon, 2009-04-06 at 12:42 +0100, neil wrote:
> We currently have two issues with this setup. One of which is NFS index 
> corruption issues we get due to NFS/dovecot locking. Basically the UUID 
> list or a .index gets corrupt. This causes a full re-indexing of the 
> mailbox / broken mailbox until i delete the indexes. In the UUID lists 
> case the symptom tends to effect use who use POP rather than IMAP and 
> insist on keeping messages on the server. Because it's corrupt it gets 
> rebuilt one way or the other and the users email client proceeds to 
> redownload the entire mailbox again until he remarks them to be saved. 
> This tends to annoy the user a lot. After a bit of testing we do however 
> expect this to be fixed by version 1.1. However if anyone has any 
> comments on this I would certainly be interested.
v1.1 at least handles it much better.
> 1. We obviously reach the auth thread cap eventually so any new auth 
> requests basically get refused by the server. 
Auth thread? Do you mean the max. number of login connections/processes?
Do you have login_process_per_connection=no? That might help.
http://wiki.dovecot.org/LoginProcess
> 2. Now here's my real gripe. Dovecot does not handle running out 
> resources very gracefully at all in our setup. It does start killing 
> threads after a while. I get multiple *"dovecot: child 17989 (login) 
> killed with signal 9". 
Dovecot doesn't kill them, kernel kills them.
> *I'm not exactly sure what's going on here 
> because after this all I can see is the machine totally out of memory 
> and the kernel starts killing absolutely everything. All services are 
> killed (including ssh etc..) and I plug a monitor into the server and 
> find the last few lines of the console listing init and other rather 
> important things having just been killed. At this point it is a case of 
> power cycling the server and all is back to normal again.
Maybe it would help to change max_mail_processes to a lower number,
those are probably the ones eating the memory.

dovecot -n output might have been helpful also in giving some
suggestions.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20090408/80241448/attachment-0002.bin>