dovecot - May 2012 - High level of pop3 popping causing server to become unresponsive

Hello all,

We have put Dovecot 2.1.4 on several of our production servers (CentOS, on
Dell R710, with 20GB memory, dual CPU Quad-core). We have a single instance
of Dovecot running and currently have several instances of Popa3d.  When
there are significant amount of popping from 2 mailboxes that dovecot that
is popping from (500+ msgs in the mailboxes), the popping of the messages
causes the boxes to become unresponsive.  We use another application that
connects to the Dovecot, downloads 2-10 messages, then processes them, then
sends the delete command to Dovecot.

When this issue occurs we are unable to become Root, or login again if we
close our ssh connection.  This only occures when Dovecot is doing the
popping.  If we only run the older Popa3d, this doesn't occur.  We believe
it is caused by the way dovecot is authenticating.

We are using auth_mechanisms = plain

passdb
drive = shadow

usedb
driver = passwd
args = blocking=yes

If anyone could suggest what could be causing the login issue, we would
appreciate any incite to fix it!

Thanks,

Kevin

On 17.5.2012, at 18.22, Root Kev wrote:
> We have put Dovecot 2.1.4 on several of our production servers (CentOS, on
> Dell R710, with 20GB memory, dual CPU Quad-core). We have a single instance
> of Dovecot running and currently have several instances of Popa3d.  When
> there are significant amount of popping from 2 mailboxes that dovecot that
> is popping from (500+ msgs in the mailboxes), the popping of the messages
> causes the boxes to become unresponsive.  We use another application that
> connects to the Dovecot, downloads 2-10 messages, then processes them, then
> sends the delete command to Dovecot.
Unresponsive for a long time?.. What CentOS version?
> When this issue occurs we are unable to become Root, or login again if we
> close our ssh connection.  This only occures when Dovecot is doing the
> popping.  If we only run the older Popa3d, this doesn't occur.  We
believe
> it is caused by the way dovecot is authenticating.
Sounds like PAM is hanging. Is the (CPU) load in general high at this time?
> We are using auth_mechanisms = plain
> 
> passdb
> drive = shadow
> 
> usedb
> driver = passwd
> args = blocking=yes
Using shadow/passwd directly shouldn't affect PAM at all. So this is a
rather strange problem..

Missed CCing on last reply.  See below..

Also, would having Nagios checking the number of messages in a mailbox
cause issues with the popping of messages? And/Or would having a user
accessing the mailbox from two different applications cause issues?  Ie.
from Outlook and mobile device?  I am under the impression that a lock file
is generated which should deal with the issue of contention, deleting mail,
etc.

Thanks for any information.

Kevin


Sorry for the delay in responding, long weekend in
Canada...
>
> When trying to SSH into the server, the server prompts for user name then
> password, then just hangs.  Same if there is already a console connection
> over, when trying to SU to root, it just hangs after entering the password.
>
> Our passwords are in the shadow file, and because this is a server
> dedicated for this task, there is only the default linux users, and maybe 8
> other user accounts in the shadow file.
>
> There shouldn't be high IO as all this box does is postfix, pop3ad and
> dovecot.  This box is seeing less then 3000 emails a day, and only has 5
> mailboxes on it.
>
> Thanks for the continued suggestions...
>
> Kevin
>
>
> On Sat, May 19, 2012 at 3:33 PM, Timo Sirainen <tss at iki.fi> wrote:
>
>> On Fri, 2012-05-18 at 09:21 -0400, Root Kev wrote:
>> > During the last time that the load went up, it became unable to
login /
>> su
>> > to root for the entire period that dovecot was running, we had to
kill
>> > dovecot and go back to Popa3d until the mailq was cleared up.  We
are
>> > running CentOS 5.6 server.  Based on TOP running at the time the
CPU
>> usage
>> > was running under 10%.  Once Dovecot was killed, we were then able
to
>> log
>> > in /su again.
>>
>> Like Kelsey said, a very high disk IO might explain this, although
>> normally the login should still eventually succeed. Another thing
I'm
>> wondering is if some process limit reached. How does the login/su fail,
>> does it just hang or immediately fail with some error?
>>
>> > We were under the impression that checking to shadow directly
should be
>> the
>> > fastest and least amount of overhead, is any of the other ways to
>> connect
>> > have less load on authentication to PAM?
>>
>> Your passwords really are in /etc/shadow file, not LDAP/something else?
>> I don't think the problem is with authentication. Reading
/etc/shadow is
>> pretty fast (unless maybe if it's a huge file) and it anyway
can't block
>> login/su from working.
>>
>>
>>
>

On 24 May 2012, at 16:54, Root Kev <root.kev at gmail.com> wrote:
> Currently cannot use IMAP as our application to access the mailbox
> currently is only setup to access pop3 mailboxes.  We are currently using
> Dovecot 2.1.4.  Below is the majority of our configs, it is mostly basic:
> 
> protocols = pop3
> listen = ***Address here***
> base_dir = /var/run/dovecot1/
> instance_name = Popper
> login_greeting = Popper
> mail_location >
mbox:/var/empty:INBOX=/opt/mailstore/spool/mail/%u:INDEX=MEMORY
> 
> !include conf.d/*.conf
> !include_try local.conf
> 
> disable_plaintext_auth = no
> auth_mechanisms = plain
> log_path = syslog
> syslog_facility = mail
> auth_verbose = yes
> log_timestamp = "%b %d %H:%M:%S "
> namespace inbox {
> inbox = yes
> }
> mail_privileged_group = mail
> lock_method = fcntl
> mbox_read_locks = fcntl
> mbox_write_locks = dotlock fcntl
> pop3_uidl_format = %08Xu%08Xv
> passdb {
>  driver = checkpassword
>  args = /usr/bin/checkpassword
> }
> userdb {
>  driver = prefetch
> }
Hi,

Having a system with a third of our users on POP3 (2300000 of them), no trouble
with dovecot (v2.1.5, on CentOS 5).
But one thing surprises me in your config, the INDEX=MEMORY in the location
parameter. That means that for each POP3 connection, dovecot has to read each
and every mails to create the index in memory. That might be why the machine
becomes unresponsive.
Unless you have a specific reason to use memory index (and I would be curious to
know about it), I would suggest to remove this and keep standard file indexes,
and your performances should improve a lot.

Regards,

	Thierry

So the best way to would be to remove it that part completely, or should it
be stored somewhere on disk?

Thanks again,

Kevin

> Hi,
>
> Having a system with a third of our users on POP3 (2300000 of them), no
> trouble with dovecot (v2.1.5, on CentOS 5).
> But one thing surprises me in your config, the INDEX=MEMORY in the
> location parameter. That means that for each POP3 connection, dovecot has
> to read each and every mails to create the index in memory. That might be
> why the machine becomes unresponsive.
> Unless you have a specific reason to use memory index (and I would be
> curious to know about it), I would suggest to remove this and keep standard
> file indexes, and your performances should improve a lot.
>
> Regards,
>
>        Thierry
>
>

On 25 May 2012, at 20:00, Root Kev <root.kev at gmail.com> wrote:
> So the best way to would be to remove it that part completely, or should it
be stored somewhere on disk?
> 
> Thanks again,
> 
> Kevin
yes, best is to remove ":INDEX=MEMORY" part, and it will store indexes
in your INBOX path, which is fine.

If you want to put it in a different path, you can have something like
INDEX=/path/to/indexes/%u

Regards,

	Thierry
> 
> 
> Hi,
> 
> Having a system with a third of our users on POP3 (2300000 of them), no
trouble with dovecot (v2.1.5, on CentOS 5).
> But one thing surprises me in your config, the INDEX=MEMORY in the location
parameter. That means that for each POP3 connection, dovecot has to read each
and every mails to create the index in memory. That might be why the machine
becomes unresponsive.
> Unless you have a specific reason to use memory index (and I would be
curious to know about it), I would suggest to remove this and keep standard file
indexes, and your performances should improve a lot.
> 
> Regards,
> 
>        Thierry
> 
>

Thierry,

Had a chance to test this change this morning, and in my test environment,
this does drastically improve the ability to ssh and su during heavy pop3
load (in test environment, change of 10-15sec to 1-2sec login).  While this
is better, the popping is still slowing down authentication on the box
during load.  Is there anything else that might be changed to increase
speed?

Just a recap, I basically have between 2-8 mailboxes on each server, and an
application that creates two connections to each mailbox, processes each
message that comes in, and then deleted each message from the pop3
mailbox.  During high load times (500+ messages) with the settings that I
had, I was unable to reconnect to the server, nor su to root.  Ended up
having to stop the dovecot process and revert to popa3d (luckily had a ssh
session still open to the box).

This is the config from my test environment:
root at devsmtp:~# dovecot -c /usr/local/etc/dovecot/dovecot-test.conf -n
# 2.1.4: /usr/local/etc/dovecot/dovecot-test.conf
# OS: Linux 2.6.32-33-generic-pae i686 Ubuntu 10.04.4 LTS ext4
auth_verbose = yes
base_dir = /var/run/dovecot-test/
disable_plaintext_auth = no
instance_name = dovecot-test
listen = <removed>
login_greeting = Dovecot-Test for removing index
mail_location = mbox:/var/empty:INBOX=/var/mail/%u
mail_privileged_group = mail
namespace inbox {
  inbox = yes
  location   mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix }
passdb {
  driver = shadow
}
protocols = pop3
service pop3-login {
  inet_listener pop3 {
    port = 130
  }
  service_count = 0
}
ssl = no
userdb {
  driver = passwd
}
protocol pop3 {
  pop3_uidl_format = %08Xu%08Xv
}


And from one of my production servers that has the issue:
# 2.1.4: /usr/local/etc/dovecot/dovecot.conf-ml
# OS: Linux 2.6.18-194.17.1.el5 x86_64 Red Hat Enterprise Linux Server
release 5.6 (Tikanga) ext3
auth_verbose = yes
base_dir = /var/run/dovecotml/
disable_plaintext_auth = no
instance_name = Popper
listen = <removed>
login_greeting = Popper
mail_location mbox:/var/empty:INBOX=/opt/mailstore/spool/mail/%u:INDEX=MEMORY
mail_privileged_group = mail
namespace inbox {
  inbox = yes
  location   mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix }
passdb {
  driver = shadow
}
protocols = pop3
ssl = no
userdb {
  args = blocking=yes
  driver = passwd
}
protocol pop3 {
  pop3_uidl_format = %08Xu%08Xv
}


On Fri, May 25, 2012 at 2:24 PM, Thierry de Montaudry <thierry at
odry.net>wrote:
> On 25 May 2012, at 20:00, Root Kev <root.kev at gmail.com> wrote:
>
> So the best way to would be to remove it that part completely, or should
> it be stored somewhere on disk?
>
> Thanks again,
>
> Kevin
>
> yes, best is to remove ":INDEX=MEMORY" part, and it will store
indexes in
> your INBOX path, which is fine.
>
> If you want to put it in a different path, you can have something like
> INDEX=/path/to/indexes/%u
>
> Regards,
>
> Thierry
>
>
>
>> Hi,
>>
>> Having a system with a third of our users on POP3 (2300000 of them), no
>> trouble with dovecot (v2.1.5, on CentOS 5).
>> But one thing surprises me in your config, the INDEX=MEMORY in the
>> location parameter. That means that for each POP3 connection, dovecot
has
>> to read each and every mails to create the index in memory. That might
be
>> why the machine becomes unresponsive.
>> Unless you have a specific reason to use memory index (and I would be
>> curious to know about it), I would suggest to remove this and keep
standard
>> file indexes, and your performances should improve a lot.
>>
>> Regards,
>>
>>        Thierry
>>
>>
>
>