dovecot - Feb 2012 - dsync deleting too many emails (sdbox)

I'm using dsync to synchronize emails on a laptop where wifi connectivity
sometimes fails in the middle of a sync. I have a shell script that runs
dsync, and here is one line of it including the output of dsync:

+ dsync -f -m realmail mirror /home/paulproteus/projects/ssh-attach/run ssh
rose.makesad.us dsync
dsync-local(paulproteus): Error: dbox
/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails: Unexpectedly lost
uid=337460
dsync-local(paulproteus): Error: msg guid lookup failed: Internal error
occurred. Refer to server log for more information. [2012-02-02 11:02:12]
dsync-local(paulproteus): Warning: sdbox
/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails: Rebuilding index
dsync-local(paulproteus): Warning: sdbox
/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails: Ignoring invalid
filename 130608.broken
dsync-local(paulproteus): Warning: sdbox
/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails: Ignoring invalid
filename 66159.broken
dsync-local(paulproteus): Error: Corrupted dbox file
/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.66159 (around
offset=16): EOF reading msg header (got 0/30 bytes)
dsync-local(paulproteus): Error:
link(/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.66159,
/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.66159.broken)
failed: File exists
dsync-local(paulproteus): Error: Corrupted dbox file
/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.66159 (around
offset=16): EOF reading msg header (got 0/30 bytes)
dsync-local(paulproteus): Warning: sdbox: Skipping unfixable file:
/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.66159
dsync-local(paulproteus): Warning: sdbox
/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails: Ignoring invalid
filename 336269.broken
dsync-local(paulproteus): Error: Corrupted dbox file
/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.130608 (around
offset=16): EOF reading msg header (got 0/30 bytes)
dsync-local(paulproteus): Error:
link(/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.130608,
/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.130608.broken)
failed: File exists
dsync-local(paulproteus): Error: Corrupted dbox file
/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.130608 (around
offset=16): EOF reading msg header (got 0/30 bytes)
dsync-local(paulproteus): Warning: sdbox: Skipping unfixable file:
/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.130608
dsync-remote(paulproteus): Error: proxy server timed out
dsync-local(paulproteus): Error: Corrupted dbox file
/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.336269 (around
offset=16): EOF reading msg header (got 0/30 bytes)
dsync-local(paulproteus): Error:
link(/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.336269,
/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.336269.broken)
failed: File exists
dsync-local(paulproteus): Error: Corrupted dbox file
/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.336269 (around
offset=16): EOF reading msg header (got 0/30 bytes)
dsync-local(paulproteus): Warning: sdbox: Skipping unfixable file:
/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.336269
dsync-local(paulproteus): Warning: Transaction log file
/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/dovecot.index.log
was locked for 1528 seconds

That seemed problematic, but not dangerous.

Then I ran a fresh sync, and found 3000 (of 60,000) messages had been deleted
and expunged
from the "realmail" box.

I'm guessing this is some bad interaction with sdbox and partial file
downloads?

I haven't read the code for this, but I would guess the dsync process
isn't being atomic
about file transfers, so it is leaving half-completed transfers in place, which
results
in corrupt files when they're next examined.

dovecot -n output:

# 2.0.15: /etc/dovecot/dovecot.conf
# OS: Linux 3.0.0-2-amd64 x86_64 Debian wheezy/sid 
mail_location = sdbox:~/Maildir/dbox
passdb {
  driver = pam
}
protocols = " imap"
ssl_cert = </etc/ssl/certs/dovecot.pem
ssl_key = </etc/ssl/private/dovecot.pem
userdb {
  driver = passwd
}

-- Asheesh.

On Thu, 2012-02-02 at 14:59 -0500, Asheesh Laroia wrote:
> I'm guessing this is some bad interaction with sdbox and partial file
downloads?
> 
> I haven't read the code for this, but I would guess the dsync process
isn't being atomic
> about file transfers, so it is leaving half-completed transfers in place,
which results
> in corrupt files when they're next examined.
There were some problems related to this in dbox, although in your case
it seems to be worse than what it should..

I've anyway done several fixes in v2.1. Can you try if these problems
happen with it too? And in any case cleanup the dbox from the *.broken
files, so that "doveadm force-resync" won't give any errors.