dovecot - Dec 2013 - reproducible dsync backup panic (dovecot 2.2.7)

Hi all,

I'm receiving a reproducible dsync backup panic for some of my users. The
error message is as follows:

dsync-remote(user@<removed>): Panic: file dsync-brain-mailbox-tree-sync.c:
line 111 (dsync_brain_mailbox_tree_sync_change): assertion failed:
(brain->no_backup_overwrite)
dsync-remote(user@<removed>): Error: Raw backtrace:
/usr/lib/dovecot/libdovecot.so.0(+0x69b70) [0x7f922dad9b70] ->
/usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7f922dad9c4a]
-> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f922da94725] ->
/usr/bin/dsync(dsync_brain_mailbox_tree_sync_change+0x382) [0x42f912] ->
/usr/bin/dsync(dsync_brain_recv_mailbox_tree_deletes+0x10b) [0x42f4db] ->
/usr/bin/dsync(dsync_brain_run+0x305) [0x42c6f5] -> /usr/bin/dsync()
[0x42cbe8] -> /usr/bin/dsync() [0x43e040] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7f922daea066] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd7) [0x7f922daeaf07]
-> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f922dae9b78] ->
/usr/bin/dsync() [0x4297f9] -> /usr/bin/dsync() [0x4133cd] ->
/usr/bin/dsync(doveadm_mail_try_run+0x251) [0x414091] ->
/usr/bin/dsync(main+0x3c8) [0x412f78] ->
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7f922d704ead] ->
/usr/bin/dsync()
  [0x413191]
dsync-local(user@<removed>): Error: read(<remote-mail-server>)
failed: EOF
dsync-local(user@<removed>): Error: Remote command returned error 134

The command I'm using to back up individual user's mail is something
like:

sudo -u vmail dsync -v -u user at server.com -R backup ssh -i sshid-file login
at server.com sudo -u vmail dsync -v -u user at server.com

I use the above command at the backup server. The error started to appear a few
weeks after we started using the new mail system. First it appeared for a single
user, now it appears for some more users as well. It is deterministic and
reproducible.

Any ideas?

I included the relevant part of my config below.

Best regards,

Peter

-------------------------------------------------------------------

# 2.2.7 (5c876ed0c3e0): /etc/dovecot/dovecot.conf
# OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.2 
auth_master_user_separator = *
auth_mechanisms = plain login
hostname = <removed>
listen = *
login_greeting = $hostname IMAP server
mail_attachment_dir = /mail/mailboxes/%d/_attachments_
mail_attachment_fs = sis-queue $mail_attachment_dir/queue:posix
mail_attachment_min_size = 16 k
mail_location = mdbox:~
mail_plugins = " zlib acl"
mdbox_rotate_interval = 5 days
mdbox_rotate_size = 5 M
namespace {
  inbox = yes
  location = 
  prefix = 
  separator = /
  type = private
}
namespace {
  list = children
  location = mdbox:/mail/mailboxes/%%d/%%n:INDEX=/mail/mailboxes/%%d/%%n
  prefix = Accounts/%%n/
  separator = /
  subscriptions = no
  type = shared
}
passdb {
  args = /mail/config/dovecot/passwd
  driver = passwd-file
}
passdb {
  args = /mail/config/dovecot/passwd-master
  driver = passwd-file
  master = yes
  pass = yes
}
plugin {
  acl = vfile
  acl_defaults_from_inbox = yes
  acl_shared_dict = file:/mail/mailboxes/%d/shared-mailboxes.db
  zlib_save = gz
  zlib_save_level = 6
}
postmaster_address = postmaster@<removed>
protocols = imap
service auth {
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0660
    user = postfix
  }
}
service imap-login {
  inet_listener imap {
    port = 0
  }
  inet_listener imaps {
    port = 993
    ssl = yes
  }
}
syslog_facility = local6
userdb {
  args = uid=vmail gid=vmail master_user=%u home=/mail/mailboxes/%d/%n
  driver = static
}
protocol imap {
  mail_plugins = " zlib acl imap_acl"
}

On 17.12.2013, at 11.54, Peter Benko <pbopbo at freemail.hu> wrote:
> I'm receiving a reproducible dsync backup panic for some of my users.
The error message is as follows:
> 
> dsync-remote(user@<removed>): Panic: file
dsync-brain-mailbox-tree-sync.c: line 111
(dsync_brain_mailbox_tree_sync_change): assertion failed:
(brain->no_backup_overwrite)
Can you get it to dump core and use

gdb /usr/bin/doveadm core
bt full
fr 5
p *change
> The command I'm using to back up individual user's mail is
something like:
> 
> sudo -u vmail dsync -v -u user at server.com -R backup ssh -i sshid-file
login at server.com sudo -u vmail dsync -v -u user at server.com
Probably an easier way would be if you could copy the local and remote mdboxes
to e.g. /tmp/mdbox-local and /tmp/mdbox-remote. Then you should be able to do:

gdb --args doveadm -o mail=mdbox:/tmp/mdbox-local backup -R
mdbox:/tmp/mdbox-remote
run
bt full
fr 5
p *change

Oh, and in any case this would be helpful in figuring out the problem. Run these
on both local and remote servers and send the output to me:

doveadm mailbox status -u user at domain 'guid uidvalidity uidnext'
'*'
doveadm dump /mail/mailboxes/domain/user/dovecot.mailbox.log