Hi, we are getting some core dumps with signal 6 in ltmp on a dovecot
director proxy server,
like this:
Dec 5 14:31:51 sproxy1 dovecot: lmtp(29999): Panic: file lmtp-proxy.c:
line 376 (lmtp_proxy_output_timeout): assertion failed:
(proxy->data_input->eof)
Dec 5 14:31:51 sproxy1 dovecot: lmtp(29999): Error: Raw backtrace:
/usr/lib64/dovecot/libdovecot.so.0() [0x363323d99a] ->
/usr/lib64/dovecot/libdovecot.so.0() [0x363323d9e6] ->
/usr/lib64/dovecot/libdovecot.so.0(i_error+0) [0x3633216f8f] ->
dovecot/lmtp() [0x406e57] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0xd4)
[0x3633248ff4] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x5b)
[0x3633249bdb] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x28)
[0x3633248c58] ->
/usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13)
[0x3633236fc3] -> dovecot/lmtp(main+0x154) [0x403f84] ->
/lib64/libc.so.6(__libc_start_main+0xfd) [0x373cc1ec5d] ->
dovecot/lmtp() [0x403d69]
Dec 5 14:31:51 sproxy1 abrt[30067]: saved core dump of pid 29999
(/usr/libexec/dovecot/lmtp) to
/var/spool/abrt/ccpp-1323091911-29999.new/coredump (1368064 bytes)
Dec 5 14:31:51 sproxy1 dovecot: master: Error: service(lmtp): child
29999 killed with signal 6 (core dumped)
we have something misconfigured or it's a bug on this dovecot version?
our doveconf -n is:
# 2.0.15: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-71.29.1.el6.x86_64 x86_64 CentOS Linux release 6.0
(Final)
auth_cache_size = 3 k
auth_cache_ttl = 15 mins
auth_verbose = yes
base_dir = /var/run/dovecot/
default_client_limit = 30000
default_process_limit = 5000
director_doveadm_port = 990
director_mail_servers = 10.80.82.21 10.80.82.22
director_servers = 10.80.82.11 10.80.82.12
doveadm_proxy_port = 24245
lmtp_proxy = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date ihave
passdb {
args = proxy=y nopassword=y starttls=any-cert
driver = static
}
postmaster_address = xavier.pons at uib.es
protocols = imap pop3 lmtp sieve
service auth {
client_limit = 27048
unix_listener /var/spool/postfix/private/auth {
mode = 0666
}
unix_listener auth-userdb {
group = dovecot
mode = 0660
}
}
service director {
fifo_listener login/proxy-notify {
mode = 0666
}
inet_listener {
port = 991
}
inet_listener director-doveadm {
port = 990
}
unix_listener director-userdb {
mode = 0660
}
unix_listener login/director {
mode = 0666
}
}
service doveadm {
inet_listener {
port = 24245
}
}
service imap-login {
executable = imap-login director
inet_listener imap {
port = 143
}
inet_listener imaps {
port = 993
ssl = yes
}
}
service lmtp {
inet_listener lmtp {
port = 30025
}
}
service managesieve-login {
executable = managesieve-login director
}
service pop3-login {
executable = pop3-login director
inet_listener pop3 {
port = 110
}
inet_listener pop3s {
port = 995
ssl = yes
}
}
ssl = required
ssl_cert = </etc/pki/tls/certs/imap.crt
ssl_key = </etc/pki/tls/private/imap.uib.es_privatekey.pem
syslog_facility = local1
verbose_proctitle = yes
protocol lmtp {
auth_socket_path = director-userdb
passdb {
args = /etc/dovecot/dovecot-ldap-pass.conf.lmtp
driver = ldap
}
}
protocol doveadm {
auth_socket_path = director-userdb
}
protocol imap {
mail_max_userip_connections = 20
}
protocol pop3 {
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
}
Xavier
--
xavier.pons at uib.es
Centre de Tecnologies de la Informaci?
Universitat Illes Balears
On Mon, Dec 05, 2011 at 03:26:54PM +0100, Xavier Pons wrote:> Hi, we are getting some core dumps with signal 6 in ltmp on a > dovecot director proxy server, like this:http://dovecot.org/list/dovecot/2011-September/060830.html http://www.dovecot.org/list/dovecot/2011-September/061395.html> we have something misconfigured or it's a bug on this dovecot version?I believe there's too short LMTP_PROXY_DATA_INPUT_TIMEOUT_MSECS in lmtp-proxy.c. -jf
This happens when the LMTP proxy doesn't receive feedback from the back-end LMTP process for a certain amount of time. This typically happens either because of very low I/O performance or NFS locks. Do an strace on your back-end LMTP processes. If you see that these processes are waiting on NFS locks ande you are using NFSv3, you should move over to NFSv4. On Dec 5, 2011, at 6:26 AM, Xavier Pons wrote:> Hi, we are getting some core dumps with signal 6 in ltmp on a dovecot director proxy server, > like this: > > Dec 5 14:31:51 sproxy1 dovecot: lmtp(29999): Panic: file lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: (proxy->data_input->eof) > Dec 5 14:31:51 sproxy1 dovecot: lmtp(29999): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0() [0x363323d99a] -> /usr/lib64/dovecot/libdovecot.so.0() [0x363323d9e6] -> /usr/lib64/dovecot/libdovecot.so.0(i_error+0) [0x3633216f8f] -> dovecot/lmtp() [0x406e57] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0xd4) [0x3633248ff4] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x5b) [0x3633249bdb] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x3633248c58] -> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x3633236fc3] -> dovecot/lmtp(main+0x154) [0x403f84] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x373cc1ec5d] -> dovecot/lmtp() [0x403d69] > Dec 5 14:31:51 sproxy1 abrt[30067]: saved core dump of pid 29999 (/usr/libexec/dovecot/lmtp) to /var/spool/abrt/ccpp-1323091911-29999.new/coredump (1368064 bytes) > Dec 5 14:31:51 sproxy1 dovecot: master: Error: service(lmtp): child 29999 killed with signal 6 (core dumped) > > we have something misconfigured or it's a bug on this dovecot version? > > our doveconf -n is: > # 2.0.15: /etc/dovecot/dovecot.conf > # OS: Linux 2.6.32-71.29.1.el6.x86_64 x86_64 CentOS Linux release 6.0 (Final) > auth_cache_size = 3 k > auth_cache_ttl = 15 mins > auth_verbose = yes > base_dir = /var/run/dovecot/ > default_client_limit = 30000 > default_process_limit = 5000 > director_doveadm_port = 990 > director_mail_servers = 10.80.82.21 10.80.82.22 > director_servers = 10.80.82.11 10.80.82.12 > doveadm_proxy_port = 24245 > lmtp_proxy = yes > managesieve_notify_capability = mailto > managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave > passdb { > args = proxy=y nopassword=y starttls=any-cert > driver = static > } > postmaster_address = xavier.pons at uib.es > protocols = imap pop3 lmtp sieve > service auth { > client_limit = 27048 > unix_listener /var/spool/postfix/private/auth { > mode = 0666 > } > unix_listener auth-userdb { > group = dovecot > mode = 0660 > } > } > service director { > fifo_listener login/proxy-notify { > mode = 0666 > } > inet_listener { > port = 991 > } > inet_listener director-doveadm { > port = 990 > } > unix_listener director-userdb { > mode = 0660 > } > unix_listener login/director { > mode = 0666 > } > } > service doveadm { > inet_listener { > port = 24245 > } > } > service imap-login { > executable = imap-login director > inet_listener imap { > port = 143 > } > inet_listener imaps { > port = 993 > ssl = yes > } > } > service lmtp { > inet_listener lmtp { > port = 30025 > } > } > service managesieve-login { > executable = managesieve-login director > } > service pop3-login { > executable = pop3-login director > inet_listener pop3 { > port = 110 > } > inet_listener pop3s { > port = 995 > ssl = yes > } > } > ssl = required > ssl_cert = </etc/pki/tls/certs/imap.crt > ssl_key = </etc/pki/tls/private/imap.uib.es_privatekey.pem > syslog_facility = local1 > verbose_proctitle = yes > protocol lmtp { > auth_socket_path = director-userdb > passdb { > args = /etc/dovecot/dovecot-ldap-pass.conf.lmtp > driver = ldap > } > } > protocol doveadm { > auth_socket_path = director-userdb > } > protocol imap { > mail_max_userip_connections = 20 > } > protocol pop3 { > pop3_client_workarounds = outlook-no-nuls oe-ns-eoh > } > > Xavier > > -- > xavier.pons at uib.es > Centre de Tecnologies de la Informaci? > Universitat Illes Balears > >
On Mon, 2011-12-05 at 15:26 +0100, Xavier Pons wrote:> Hi, we are getting some core dumps with signal 6 in ltmp on a dovecot > director proxy server, > like this: > > Dec 5 14:31:51 sproxy1 dovecot: lmtp(29999): Panic: file lmtp-proxy.c: > line 376 (lmtp_proxy_output_timeout): assertion failed: > (proxy->data_input->eof)I think the way I originally planned LMTP proxying to work is simply too complex to work reliably, perhaps even if the code was bug-free. So instead of reading+writing DATA at the same time, this patch changes the DATA to be first read into memory or temp file, and then from there read and sent to the LMTP backends: http://hg.dovecot.org/dovecot-2.1/raw-rev/51d87deb5c26 The patch applies on top of v2.0 also. I tested that it appears to work, but more testing wouldn't hurt to make sure it won't corrupt large mails when sending them to multiple backends (one mail with multiple RCPT TOs sent to multiple different backends)..
Apparently Analagous Threads
- director lmtp -> smtp problem
- Panic: file lmtp-proxy.c: line 370 (lmtp_proxy_output_timeout): assertion failed: (proxy->data_input->eof)
- Crash in director/lmtp-proxy (lmtp_proxy_output_timeout)
- Could not evaluate: Could not retrieve information from environment production source(s) for one module, for other is ok
- Panic with mail_log activated