thr3ads.net - dovecot - Panic: file dsync-mailbox-tree-sync.c: line 576 (node_mailbox_trees_cmp): assertion failed: (ret != 0) [Dec 2016]

If this information is useful, please help other people find it:
Share via:
Gilles Chauvin
2016-Dec-06 12:58 UTC
Panic: file dsync-mailbox-tree-sync.c: line 576 (node_mailbox_trees_cmp): assertion failed: (ret != 0)

Hi,

Here is a crash that's happening using the latest Dovecot version 
(v2.2.27 on CentOS7 x86_64):

We are using replication. Judging by the second server's logs, I believe 
this has something to do with the fact that we're using the lazy_expunge 
plugin.

Every night after midnight, we purge the lazy_expunge namespace by 
running a command similar to following one on the main server (server01 
in the logs below):
# doveadm expunge -A MAILBOX 'EXPUNGED/*' SAVEDBEFORE "$(date
+'%F')"

The lazy_expunge namespace is configured as follows:

namespace expunged {
   disabled = no
   hidden = yes
   ignore_on_failure = no
   inbox = no
   list = no
   location = 
sdbox:/srv/mail/expunged/%1n/%n:SUBSCRIPTIONS=subscriptions-expunged
   order = 0
   prefix = EXPUNGED/
   separator = /
   subscriptions = no
   type = private
}

plugin {
   lazy_expunge = EXPUNGED/
}



First server's logs:

2016-12-06T13:09:31.212620+01:00 server01 dovecot: dsync-server(user01): 
Panic: file dsync-mailbox-tree-sync.c: line 576 
(node_mailbox_trees_cmp): assertion failed: (ret != 0)
2016-12-06T13:09:31.213473+01:00 server01 dovecot: dsync-server(user01): 
Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x92d70) 
[0x7f57cc18ad70] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x92e4e) 
[0x7f57cc18ae4e] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) 
[0x7f57cc1264e0] -> dovecot/doveadm-server [10.0.0.2 user01 
recv_mailbox_tree_deletes]() [0x447584] -> dovecot/doveadm-server 
[10.0.0.2 user01 recv_mailbox_tree_deletes]() [0x447a57] -> 
dovecot/doveadm-server [10.0.0.2 user01 recv_mailbox_tree_deletes]() 
[0x447849] -> dovecot/doveadm-server [10.0.0.2 user01 
recv_mailbox_tree_deletes]() [0x447849] -> dovecot/doveadm-server 
[10.0.0.2 user01 
recv_mailbox_tree_deletes](dsync_mailbox_trees_sync_init+0x1f9) 
[0x447cc9] -> dovecot/doveadm-server [10.0.0.2 user01 
recv_mailbox_tree_deletes](dsync_brain_recv_mailbox_tree_deletes+0x207) 
[0x43a477] -> dovecot/doveadm-server [10.0.0.2 user01 
recv_mailbox_tree_deletes](dsync_brain_run+0x5fe) [0x436e4e] -> 
dovecot/doveadm-server [10.0.0.2 user01 recv_mailbox_tree_deletes]() 
[0x437280] -> dovecot/doveadm-server [10.0.0.2 user01 
recv_mailbox_tree_deletes]() [0x44c5af] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) 
[0x7f57cc19e4d2] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xe7) 
[0x7f57cc19fa07] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) 
[0x7f57cc19e56c] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) 
[0x7f57cc19e728] -> dovecot/doveadm-server [10.0.0.2 user01 
recv_mailbox_tree_deletes]() [0x41faae] -> dovecot/doveadm-server 
[10.0.0.2 user01 recv_mailbox_tree_deletes]() [0x421316] -> 
dovecot/doveadm-server [10.0.0.2 user01 recv_mailbox_tree_deletes]() 
[0x433b64] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) 
[0x7f57cc19e4d2] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xe7) 
[0x7f57cc19fa07] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) 
[0x7f57cc19e56c] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) 
[0x7f57cc19e728] -> 
/usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7f57cc12c513] -> dovecot/doveadm-server [10.0.0.2 user01 
recv_mailbox_tree_deletes](main+0x197) [0x413b57] -> 
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f57cbd57b15] -> 
dovecot/doveadm-server [10.0.0.2 user01 recv_mailbox_tree_deletes]() 
[0x413bf5]
2016-12-06T13:09:31.214076+01:00 server01 dovecot: dsync-server(user01): 
Fatal: master: service(doveadm): child 6837 killed with signal 6 (core 
dumps disabled)
2016-12-06T13:11:42.880516+01:00 server01 dovecot: dsync-local(user01): 
Error: read(server02.localdomain) failed: EOF (last sent=mailbox_delete, 
last recv=mailbox_tree_node)


Second server's logs:

2016-12-06T12:09:36.220591+01:00 server02 dovecot: doveadm(user01): 
Error: Duplicate mailbox GUID eab35824e49b4658840001001776124f for 
mailboxes EXPUNGED/Trash/C2i - 
Coordination-temp-ac1fea3af7e1f2c29ce1c62442a06272/Demissions C2i and 
EXPUNGED/Trash/C2i - Coordination/Demissions C2i - giving a new GUID 
c4ffaf08709c465813ae0000a26dfa6e to EXPUNGED/Trash/C2i - 
Coordination-temp-ac1fea3af7e1f2c29ce1c62442a06272/Demissions C2i
2016-12-06T12:09:36.220859+01:00 server02 dovecot: doveadm(user01): 
Error: Duplicate mailbox GUID 2ab10a2bee9b4658840001001776124f for 
mailboxes EXPUNGED/Trash/C2i - 
Coordination-temp-ac1fea3af7e1f2c29ce1c62442a06272/2014-2015 and 
EXPUNGED/Trash/C2i - Coordination/2014-2015 - giving a new GUID 
c6ffaf08709c465813ae0000a26dfa6e to EXPUNGED/Trash/C2i - 
Coordination-temp-ac1fea3af7e1f2c29ce1c62442a06272/2014-2015
2016-12-06T12:09:36.221091+01:00 server02 dovecot: doveadm(user01): 
Error: Duplicate mailbox GUID eeb35824e49b4658840001001776124f for 
mailboxes EXPUNGED/Trash/C2i - 
Coordination-temp-ac1fea3af7e1f2c29ce1c62442a06272/2012-2013 and 
EXPUNGED/Trash/C2i - Coordination/2012-2013 - giving a new GUID 
c8ffaf08709c465813ae0000a26dfa6e to EXPUNGED/Trash/C2i - 
Coordination-temp-ac1fea3af7e1f2c29ce1c62442a06272/2012-2013
2016-12-06T12:09:36.284224+01:00 server02 dovecot: dsync-local(user01): 
Error: Mailbox EXPUNGED/Trash/C2i - Coordination/2008-2009/Formation FC 
sync: mailbox_rename failed: Can't rename mailboxes to/from expunge 
namespace.
2016-12-06T12:09:45.469706+01:00 server02 dovecot: dsync-server(user01): 
Error: Duplicate mailbox GUID eab35824e49b4658840001001776124f for 
mailboxes EXPUNGED/Trash/C2i - 
Coordination-temp-ac1fea3af7e1f2c29ce1c62442a06272/Demissions C2i and 
EXPUNGED/Trash/C2i - Coordination/Demissions C2i - giving a new GUID 
671ae30f799c465882b00000a26dfa6e to EXPUNGED/Trash/C2i - 
Coordination-temp-ac1fea3af7e1f2c29ce1c62442a06272/Demissions C2i
2016-12-06T12:09:45.470389+01:00 server02 dovecot: dsync-server(user01): 
Error: Duplicate mailbox GUID 2ab10a2bee9b4658840001001776124f for 
mailboxes EXPUNGED/Trash/C2i - 
Coordination-temp-ac1fea3af7e1f2c29ce1c62442a06272/2014-2015 and 
EXPUNGED/Trash/C2i - Coordination/2014-2015 - giving a new GUID 
691ae30f799c465882b00000a26dfa6e to EXPUNGED/Trash/C2i - 
Coordination-temp-ac1fea3af7e1f2c29ce1c62442a06272/2014-2015
2016-12-06T12:09:45.471214+01:00 server02 dovecot: dsync-server(user01): 
Error: Duplicate mailbox GUID eeb35824e49b4658840001001776124f for 
mailboxes EXPUNGED/Trash/C2i - 
Coordination-temp-ac1fea3af7e1f2c29ce1c62442a06272/2012-2013 and 
EXPUNGED/Trash/C2i - Coordination/2012-2013 - giving a new GUID 
6b1ae30f799c465882b00000a26dfa6e to EXPUNGED/Trash/C2i - 
Coordination-temp-ac1fea3af7e1f2c29ce1c62442a06272/2012-2013
2016-12-06T12:09:45.485663+01:00 server02 dovecot: dsync-server(user01): 
Error: Mailbox EXPUNGED/Trash/C2i - Coordination/2008-2009/Formation FC 
sync: mailbox_rename failed: Can't rename mailboxes to/from expunge 
namespace.

[... Previous line repeated numerous times ...]

2016-12-06T12:14:32.583382+01:00 server02 dovecot: dsync-server(user01): 
Error: Mailbox EXPUNGED/Trash/C2i - Coordination/2008-2009/Formation FC 
sync: mailbox_rename failed: Can't rename mailboxes to/from expunge 
namespace.
2016-12-06T12:14:36.778278+01:00 server02 dovecot: dsync-local(user01): 
Error: Mailbox EXPUNGED/Trash/C2i - Coordination/2008-2009/Formation FC 
sync: mailbox_rename failed: Can't rename mailboxes to/from expunge 
namespace.

[... / ...]

2016-12-06T12:19:30.677497+01:00 sauto dovecot: dsync-server(user01): 
Panic: file dsync-mailbox-tree-sync.c: line 576 
(node_mailbox_trees_cmp): assertion failed: (ret != 0)
2016-12-06T12:19:30.678991+01:00 sauto dovecot: dsync-server(user01): 
Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x92d70) 
[0x7f40f3cd1d70] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x92e4e) 
[0x7f40f3cd1e4e] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) 
[0x7f40f3c6d4e0] -> dovecot/doveadm-server [10.0.0.1 user01 
recv_mailbox_tree_deletes]() [0x447584] -> dovecot/doveadm-server 
[10.0.0.1 user01 recv_mailbox_tree_deletes]() [0x447a57] -> 
dovecot/doveadm-server [10.0.0.1 user01 recv_mailbox_tree_deletes]() 
[0x447849] -> dovecot/doveadm-server [10.0.0.1 user01 
recv_mailbox_tree_deletes]() [0x447849] -> dovecot/doveadm-server 
[10.0.0.1 user01 
recv_mailbox_tree_deletes](dsync_mailbox_trees_sync_init+0x1f9) 
[0x447cc9] -> dovecot/doveadm-server [10.0.0.1 user01 
recv_mailbox_tree_deletes](dsync_brain_recv_mailbox_tree_deletes+0x207) 
[0x43a477] -> dovecot/doveadm-server [10.0.0.1 user01 
recv_mailbox_tree_deletes](dsync_brain_run+0x5fe) [0x436e4e] -> 
dovecot/doveadm-server [10.0.0.1 user01 recv_mailbox_tree_deletes]() 
[0x437280] -> dovecot/doveadm-server [10.0.0.1 user01 
recv_mailbox_tree_deletes]() [0x44c5af] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) 
[0x7f40f3ce54d2] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xe7) 
[0x7f40f3ce6a07] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) 
[0x7f40f3ce556c] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) 
[0x7f40f3ce5728] -> dovecot/doveadm-server [10.0.0.1 user01 
recv_mailbox_tree_deletes]() [0x41faae] -> dovecot/doveadm-server 
[10.0.0.1 user01 recv_mailbox_tree_deletes]() [0x421316] -> 
dovecot/doveadm-server [10.0.0.1 user01 recv_mailbox_tree_deletes]() 
[0x433b64] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) 
[0x7f40f3ce54d2] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xe7) 
[0x7f40f3ce6a07] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) 
[0x7f40f3ce556c] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) 
[0x7f40f3ce5728] -> 
/usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7f40f3c73513] -> dovecot/doveadm-server [10.0.0.1 user01 
recv_mailbox_tree_deletes](main+0x197) [0x413b57] -> 
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f40f389eb15] -> 
dovecot/doveadm-server [10.0.0.1 user01 recv_mailbox_tree_deletes]() 
[0x413bf5]
2016-12-06T12:19:30.689038+01:00 sauto dovecot: dsync-server(user01): 
Fatal: master: service(doveadm): child 96318 killed with signal 6 (core 
dumps disabled)
2016-12-06T12:19:30.749074+01:00 sauto dovecot: dsync-local(user01): 
Error: read(server01.localdomain) failed: EOF (last sent=mailbox_delete, 
last recv=mailbox_tree_node)



Thanks,
Kind regards,
Gilles.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3086 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20161206/874e6e8e/attachment-0001.p7s>
Apparently Analagous Threads

Search for more seemingly similar threads
dovecot - Dec 2016 - Panic: file dsync-mailbox-tree-sync.c: line 576 (node_mailbox_trees_cmp): assertion failed: (ret != 0)

Panic: file dsync-mailbox-tree-sync.c: line 576 (node_mailbox_trees_cmp): assertion failed: (ret != 0)

Apparently Analagous Threads

Wisdom of the Ancients
