dovecot - Dec 2015 - LMTP crashes on Debian with dovecot backports from rename-it.nl

Hello dovecot-users,

we're running Debian 7.9 with dovecot from Stephan Bosch's repository:
deb http://xi.rename-it.nl/debian/ oldstable-auto/dovecot-2.2 main

Starting with dovecot 2.2.20-1 build 28, we suddenly discovered that
some mails (to be locally delivered) "got stuck". dovecot's log
revealed
messages like the following one:

Dec 29 09:56:50 mailhost dovecot:
lmtp(MAILRECIPIENT at chemie.uni-hamburg.de): Error: Raw backtrace:
/usr/lib/dovecot/libdovecot.so.0(+0x8360f) [0x7f58fce4f60f] ->
/usr/lib/dovecot/libdovecot.so.0(+0x8366e) [0x7f58fce4f66e] ->
/usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f58fcdf8a3e] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0x3cc75) [0x7f58fd107c75] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0xabeda) [0x7f58fd176eda] ->
/usr/lib/dovecot/libdovecot-storage.so.0(index_mail_get_first_header+0xba)
[0x7f58fd17723a] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mail_get_first_header+0x38)
[0x7f58fd108cd8] ->
/usr/lib/dovecot/modules/lib20_mail_log_plugin.so(+0x1b5c)
[0x7f58fb16cb5c] ->
/usr/lib/dovecot/modules/lib20_mail_log_plugin.so(+0x22e9)
[0x7f58fb16d2e9] ->
/usr/lib/dovecot/modules/lib15_notify_plugin.so(notify_contexts_mail_copy+0x46)
[0x7f58fb371576] ->
/usr/lib/dovecot/modules/lib15_notify_plugin.so(+0x2750)
[0x7f58fb372750] ->
/usr/lib/dovecot/modules/lib10_quota_plugin.so(+0xbc20) [0x7f58fb580c20]
-> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_copy+0x7d)
[0x7f58fd11305d] ->
/usr/lib/dovecot/libdovecot-lda.so.0(mail_deliver_save+0x171)
[0x7f58fd4019c1]->
/usr/lib/dovecot/libdovecot-lda.so.0(mail_deliver+0x11b)
[0x7f58fd401dfb] -> dovecot/lmtp [DATA
MAILRECIPIENT at chemie.uni-hamburg.de](+0x7263) [0x7f58fd831263] ->
dovecot/lmtp [DATA MAILRECIPIENT at chemie.uni-hamburg.de](+0x79d9)
[0x7f58fd8319d9] -> /usr/lib/dovecot
/libdovecot.so.0(io_loop_call_io+0x5b) [0x7f58fce6282b] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xd7)
[0x7f58fce63c07] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x25)
[0x7f58fce628d5] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38)
[0x7f58fce62a78] ->
/usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13)
[0x7f58fcdfe443] -> dovecot/lmtp [DATA
MAILRECIPIENT at chemie.uni-hamburg.de](main+0x19c) [0x7f58fd82f8dc] ->
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7f58fca5fead]
-> dovecot/lmtp [DATA MAILRECIPIENT at chemie.uni-hamburg.de](+0x59ad)
[0x7f58fd82f9ad]
Dec 29 09:56:50 mailhost dovecot:
lmtp(MAILRECIPIENT at chemie.uni-hamburg.de): Fatal: master: service(lmtp):
child 15897 killed with signal 6 (core dumps disabled)
Dec 29 09:56:51 mailhost dovecot:
lmtp(MAILRECIPIENT at chemie.uni-hamburg.de): Panic: file
../../../src/lib/array.h: line 219 (array_idx_i):
	assertion failed: (idx * array->element_size < array->buffer->used)

All dovecot builds newer than 2.2.20-1~auto+26 show the same behaviour.

Is there anything we can do?

# doveconf -n
# 2.2.20 (): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.10
# OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.9 ext4
auth_default_realm = chemie.uni-hamburg.de
auth_master_user_separator = *
auth_mechanisms = plain login
auth_verbose = yes
login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
%k
mail_gid = vmail
mail_location = maildir:/mail/%d/%n/Maildir
mail_plugins = acl mail_log notify quota zlib
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart
imapflags notify
namespace {
  list = children
  location
maildir:/mail/%%d/%%n/Maildir:INDEX=/mail/%d/%n/shared/%%u:CONTROL=/mail/%d/%n/shared/%%u
  prefix = SHARED/%%u/
  separator = /
  type = shared
}
namespace inbox {
  hidden = no
  inbox = yes
  list = yes
  location   mailbox Drafts {
    auto = subscribe
    special_use = \Drafts
  }
  mailbox Sent {
    auto = subscribe
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    auto = no
    special_use = \Sent
  }
  mailbox Trash {
    auto = subscribe
    special_use = \Trash
  }
  prefix = INBOX/
  separator = /
  subscriptions = yes
  type = private
}
passdb {
  args = /etc/dovecot/deny-users
  deny = yes
  driver = passwd-file
}
passdb {
  args = /etc/dovecot/master-users
  driver = passwd-file
  master = yes
}
passdb {
  args = /etc/dovecot/dovecot-ldap.conf.ext
  driver = ldap
}
plugin {
  acl = vfile
  acl_shared_dict = file:/var/lib/dovecot/shared-mailboxes
  mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename
  mail_log_fields = uid box msgid size
  quota = maildir:Mailbox-Quota
  quota_warning = storage=95%% quota-warning 95 %u
  quota_warning2 = storage=80%% quota-warning 80 %u
  sieve = file:~/sieve;active=~/.dovecot.sieve
  sieve_extensions = +notify +imapflags
  zlib_save = gz
  zlib_save_level = 6
}
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
postmaster_address = postmaster at chemie.uni-hamburg.de
protocols = " imap lmtp sieve pop3"
service auth {
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0666
    user = postfix
  }
}
service imap-login {
  inet_listener imap {
    port = 143
  }
  inet_listener imaps {
    port = 993
    ssl = yes
  }
  process_min_avail = 8
  service_count = 0
  vsz_limit = 256 M
}
service imap {
  executable = imap postlogin
  process_limit = 1024
  vsz_limit = 512 M
}
service lmtp {
  inet_listener lmtp {
    address = 127.0.0.1
    port = 24
  }
  unix_listener /var/spool/postfix/private/lmtp-dovecot {
    group = postfix
    mode = 0660
    user = postfix
  }
}
service pop3-login {
  inet_listener pop3 {
    port = 0
  }
  inet_listener pop3s {
    port = 995
    ssl = yes
  }
}
service pop3 {
  executable = pop3 postlogin
}
service postlogin {
  executable = script-login -d rawlog
}
ssl = required
ssl_ca = </ssl/CA.pem
ssl_cert = </ssl/cert.pem
ssl_key = </ssl/cert.key
ssl_protocols = !SSLv2 !SSLv3
syslog_facility = daemon
userdb {
  args = /etc/dovecot/dovecot-ldap.conf.ext
  driver = ldap
}
verbose_proctitle = yes
protocol lmtp {
  mail_plugins = acl mail_log notify quota zlib sieve
}
protocol lda {
  mail_plugins = acl mail_log notify quota zlib sieve
}
protocol imap {
  mail_max_userip_connections = 50
  mail_plugins = acl mail_log notify quota zlib imap_acl imap_quota
imap_zlib
}
protocol sieve {
  mail_max_userip_connections = 100
}
protocol pop3 {
  mail_max_userip_connections = 50
}


Thanks & kind regards,
Christian Schmidt

-- 
No signature available.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5326 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20151229/e4024960/attachment.p7s>

On 29 Dec 2015, at 06:12, Christian Schmidt <christian.schmidt at
chemie.uni-hamburg.de> wrote:
> 
> Hello dovecot-users,
> 
> we're running Debian 7.9 with dovecot from Stephan Bosch's
repository:
> deb http://xi.rename-it.nl/debian/ oldstable-auto/dovecot-2.2 main
> 
> Starting with dovecot 2.2.20-1 build 28, we suddenly discovered that
> some mails (to be locally delivered) "got stuck". dovecot's
log revealed
> messages like the following one:
> Dec 29 09:56:51 mailhost dovecot:
> lmtp(MAILRECIPIENT at chemie.uni-hamburg.de): Panic: file
> ../../../src/lib/array.h: line 219 (array_idx_i):
> 	assertion failed: (idx * array->element_size <
array->buffer->used)
..
> lmtp(MAILRECIPIENT at chemie.uni-hamburg.de): Error: Raw backtrace:
> /usr/lib/dovecot/libdovecot.so.0(+0x8360f) [0x7f58fce4f60f] ->
> /usr/lib/dovecot/libdovecot.so.0(+0x8366e) [0x7f58fce4f66e] ->
> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f58fcdf8a3e] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(+0x3cc75) [0x7f58fd107c75] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(+0xabeda) [0x7f58fd176eda] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_get_first_header+0xba)
> [0x7f58fd17723a] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_first_header+0x38)
> [0x7f58fd108cd8] ->
> /usr/lib/dovecot/modules/lib20_mail_log_plugin.so(+0x1b5c)
> [0x7f58fb16cb5c] ->
So looking up a header causes a crash. Can you send me one such mail that causes
this crash? The mail can probably be put through
http://dovecot.org/tools/maildir-anonymize.pl to preserve the crash but hide all
the actual email contents.
> Dec 29 09:56:50 mailhost dovecot:
> lmtp(MAILRECIPIENT at chemie.uni-hamburg.de): Fatal: master: service(lmtp):
> child 15897 killed with signal 6 (core dumps disabled)
gdb backtrace would also be useful. See http://dovecot.org/bugreport.html
> All dovecot builds newer than 2.2.20-1~auto+26 show the same behaviour.
So either
https://github.com/dovecot/core/commit/a556e29bb9e48968b88c783148f3308bf90c7912
or
https://github.com/dovecot/core/commit/7d800c558b4a40f11748d4ebfc2bf89eb743f03c
broke it. But I don't have any good ideas why they might have broken it.

Hello Timo,

On 29.12.2015 19:25, Timo Sirainen wrote:
> So looking up a header causes a crash. Can you send me one such mail
> that causes this crash?
I'll ask one of the users suffering from the crash if I may
"borrow" one
of the mails. This will probably take some days.

Thank you!

Best wishes for the new year,
Christian

-- 
No signature available.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5326 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20151230/28622544/attachment.p7s>