dovecot - Sep 2008 - dovecot imap auth fails to reconnect pgsql backend

Hi,
    We've observed frequent auth failure recently from the SquirrelMail
frontend. The imap server is the dovecot-1.0.7-2 from Centos 5.2. The
auth backend is a pgsql database on another server. The only way to make
it work is to restart the dovecot, though sometimes it'll recover
automatically, but you don't know when. At first we thought there might
be a network problem, but we can always run pgsql client from the
dovecot server to connect and query the backend database during the auth
failure. We then upgraded to dovecot-1.1.3 from atrpms.net yesterday,
and still the same. It looks like the dovecot fails to reconnect/retry
pgsql backend.
    The related maillog in 1.0.7 version:
Sep 16 02:31:22 mail dovecot: imap-login: Disconnected: Inactivity:
method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured
Sep 16 02:33:20 mail dovecot: imap-login: Disconnected: Inactivity:
method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured
Sep 16 02:38:40 mail dovecot: imap-login: Disconnected: Inactivity:
method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured

    The related maillog in 1.1.3:
Sep 21 14:11:24 mail dovecot: imap-login: Disconnected: Inactivity (auth
failed, 1 attempts): method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured
Sep 21 14:14:39 mail dovecot: imap-login: Disconnected: Inactivity (auth
failed, 1 attempts): method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured
Sep 21 14:15:15 mail dovecot: imap-login: Disconnected: Inactivity (no
auth attempts): rip=127.0.0.1, lip=127.0.0.1, secured
Sep 21 14:23:49 mail dovecot: auth(default):
sql(user at our.domain,127.0.0.1): Password query failed: 
Sep 21 14:23:49 mail dovecot: child 15241 (auth) killed with signal 11
Sep 21 14:23:50 mail dovecot: auth(default): pgsql: Connected to
internal	

# dovecot -n
# 1.1.3: /etc/dovecot.conf
protocols: imap pop3
listen(default): *:143
listen(imap): *:143
listen(pop3): *:110
login_dir: /var/run/dovecot/login
login_executable(default): /usr/libexec/dovecot/imap-login
login_executable(imap): /usr/libexec/dovecot/imap-login
login_executable(pop3): /usr/libexec/dovecot/pop3-login
mail_location: maildir:/var/vmail/%d/%n
mail_executable(default): /usr/libexec/dovecot/imap
mail_executable(imap): /usr/libexec/dovecot/imap
mail_executable(pop3): /usr/libexec/dovecot/pop3
mail_plugins(default): quota imap_quota
mail_plugins(imap): quota imap_quota
mail_plugins(pop3): quota
mail_plugin_dir(default): /usr/lib/dovecot/imap
mail_plugin_dir(imap): /usr/lib/dovecot/imap
mail_plugin_dir(pop3): /usr/lib/dovecot/pop3
imap_client_workarounds(default): delay-newmail tb-extra-mailbox-sep
imap_client_workarounds(imap): delay-newmail tb-extra-mailbox-sep
imap_client_workarounds(pop3): 
pop3_client_workarounds(default): 
pop3_client_workarounds(imap): 
pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh
auth default:
  mechanisms: plain login
  realms: our.domain1 our.domain2
  default_realm: our.domain1
  user: mail
  passdb:
    driver: sql
    args: /etc/dovecot-sql.conf
  userdb:
    driver: static
    args: uid=508 gid=509 home=/var/vmail/%d/%n/
quota=maildir:storage=512000
  socket:
    type: listen
    client:
      path: /var/spool/postfix/private/auth
      mode: 432
      user: postfix
      group: postfix
plugin:
  quota: maildir

grep -v '^ *\(#.*\)\?$' /etc/dovecot-sql.conf
driver = pgsql
connect = host=<pg_server_ip> dbname=<pg_dbname>
user=<pg_user>
password=<pg_password>
default_pass_scheme = PLAIN
password_query = SELECT jm || '@%d' as user, password FROM usera where
jm = '%n' and forbid = 'N' and ( '%d' =
'our.domain1' or '%d' 'our.domain2' )

    Can someone shed a light on this?
    Thanks!

Regards,
Frank Wang

> Hi,
>     We've observed frequent auth failure recently from the SquirrelMail
> frontend. The imap server is the dovecot-1.0.7-2 from Centos 5.2. The
> auth backend is a pgsql database on another server. The only way to make
> it work is to restart the dovecot, though sometimes it'll recover
> automatically, but you don't know when. At first we thought there might
> be a network problem, but we can always run pgsql client from the
> dovecot server to connect and query the backend database during the auth
> failure. We then upgraded to dovecot-1.1.3 from atrpms.net yesterday,
> and still the same. It looks like the dovecot fails to reconnect/retry
> pgsql backend.
It's the router, a H3C Quidway AR28-31 with the latest firmware VRP3.4
F0306p06, caused the problem. We upgraded it last week and found part of
the database connection from the mail server were blocked by the
firewall in the DB server syslog because of the bad tcp state. After
revert the firmware, mail server auth act normal again. 

Regards,
Frank Wang

On Sun, 2008-09-21 at 15:44 +0800, Frank Wang wrote:
> Sep 21 14:23:49 mail dovecot: child 15241 (auth) killed with signal 11
Can you still reproduce this crash? It would be nice to get its
backtrace to get that bug fixed. The core file should be in Dovecot's
base_dir (/var/run/dovecot probably). http://dovecot.org/bugreport.html

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20080922/2d8008b0/attachment-0002.bin>

> On Sun, 2008-09-21 at 15:44 +0800, Frank Wang wrote:
> > Sep 21 14:23:49 mail dovecot: child 15241 (auth) killed with signal 11
> 
> Can you still reproduce this crash? It would be nice to get its
> backtrace to get that bug fixed. The core file should be in Dovecot's
> base_dir (/var/run/dovecot probably). http://dovecot.org/bugreport.html
> 
I tried the core dump method from the above link, but without luck. Here
is what I did:
1. vi /etc/dovecot.conf
	mail_drop_priv_before_exec = yes
	mail_debug = yes
	# the rest is untouched
2. ulimit -c unlimited
3. service dovecot restart
4. echo 'core.%p' > /proc/sys/kernel/core_pattern

I noticed this in the maillog this morning:
Sep 24 07:34:20 mail dovecot: child 492 (auth) killed with signal 11

But there isn't any core dump found in the whole file system. Then I
doubted there's maybe some setting problems and did the following:
echo '/tmp/core.%p" > /proc/sys/kernel/core_pattern
ulimit -c unlimited
/etc/init.d/dovecot restart
ps aux | grep dovecot-auth
kill -s 11 <pid_of_dovecot_auth_found>

And there still isn't any /tmp/core.* found. Is there any other way I
can try?

Regards,
Frank Wang