dovecot - Oct 2010 - Problems converting to maildir with dsync in 2.0.6

I'm trying to convert from an old Dovecot (1.0.15) and mbox to a new 
Dovecot and maildir.  I'm running into several problems with dsync which 
are causing my mboxes not to convert.

If I use LAYOUT=fs (as I'd prefer) I get an error because I have a 
mailbox called "subscriptions", and none of my messages convert.  Is 
there a way to get dovecot to put its own metadata in dotfiles instead?  
(I guess I can rename my mailbox too :-)

% dsync mirror mbox:~/Mail
dsync(nriley): Error: read() failed with subscription file 
/home/nriley/Maildir/subscriptions: Is a directory
dsync(nriley): Error: Can't update subscription +/INBOX: Internal error 
occurred. Refer to server log for more information. [2010-10-21 20:17:55]
[...]

Also, what "server log" is it referring to above?

So I tried without LAYOUT=fs, and dsync successfully converts some 
mailboxes, then crashes:

% dsync mirror mbox:~/Mail
dsync(nriley): Error: Maildir: 6ef533cd4cdb1fd454b0226bfc851bb3:2, 
changed UID 1846529 -> 1846531
dsync(nriley): Panic: file maildir-save.c: line 892 
(maildir_save_sync_uidlist): assertion failed: (ret > 0)
dsync(nriley): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0 
[0xb7ea21f1] -> 
/usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x47) 
[0xb7ea2317] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0xb7ea2544] 
-> 
/usr/lib/dovecot/libdovecot-storage.so.0(maildir_transaction_save_commit_
pre+0x858) [0xb7f2ab58] -> /usr/lib/dovecot/libdovecot-storage.so.0 
[0xb7f2501f] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(mail_index_transaction_commit_fu
ll+0x85) [0xb7f67eb5] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(index_transaction_commit+0x94) 
[0xb7f24c14] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(mailbox_transaction_commit_get_c
hanges+0x42) [0xb7efeaa2] -> /usr/bin/dsync [0x8056ed2] -> 
/usr/bin/dsync [0x80570f0] -> 
/usr/bin/dsync(dsync_worker_select_mailbox+0x2e) [0x8055b2e] -> 
/usr/bin/dsync [0x80507e6] -> /usr/bin/dsync [0x8050a3b] -> 
/usr/bin/dsync(dsync_brain_msg_sync_new_msgs+0x12) [0x8050c12] -> 
/usr/bin/dsync(dsync_brain_msg_sync_more+0x2e1) [0x8050261] -> 
/usr/bin/dsync(dsync_brain_sync+0x309) [0x804e949] -> /usr/bin/dsync 
[0x804f953] -> /usr/bin/dsync(dsync_brain_sync+0x72e) [0x804ed6e] -> 
/usr/bin/dsync [0x804f7d0] -> /usr/bin/dsync(dsync_brain_sync+0x55d) 
[0x804eb9d] -> /usr/bin/dsync(dsync_brain_sync_all+0x18) [0x804f668] -> 
/usr/bin/dsync(main+0x5a1) [0x804da51] -> 
/lib/libc.so.6(__libc_start_main+0xe5) [0xb7d29455] -> /usr/bin/dsync 
[0x804d2e1]
[1]    30528 abort      dsync mirror mbox:~/Mail

I tried running it in GDB but that crashed GDB, so hopefully the above 
info is OK...

% gdb --args dsync mirror mbox:~/Mail
(gdb) run
Starting program: /usr/bin/dsync mirror mbox:\~/Mail
[Thread debugging using libthread_db enabled]
[New process 30652]
Executing new program: /usr/bin/doveconf
warning: Cannot initialize thread debugging library: generic error
warning: Cannot initialize thread debugging library: generic error
warning: Cannot initialize thread debugging library: generic error
warning: Cannot initialize thread debugging library: generic error
[New process 30652]
[Thread debugging using libthread_db enabled]
[1]    30651 segmentation fault  gdb -q --args dsync mirror mbox:~/Mail

I'm using the Debian packages from rename-it.nl.  Here is doveconf -n 
output:

# 2.0.6 (aaca506ea33b): /etc/dovecot/dovecot.conf
# OS: Linux 2.6.18.8-linode22 i686 Debian 5.0.6 
mail_location = maildir:~/Maildir:INBOX=~/Maildir/INBOX
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date
namespace {
  inbox = yes
  location = 
  prefix = 
  separator = /
}
passdb {
  driver = pam
}
plugin {
  sieve = ~/.dovecot.sieve
  sieve_dir = ~/sieve
}
protocols = imap
ssl = required
ssl_cert = </etc/ssl/certs/dovecot.pem
ssl_key = </etc/ssl/private/dovecot.pem
userdb {
  driver = passwd
}
-- 
Nicholas Riley <njriley at illinois.edu>

In article <njriley-5CB377.20303521102010 at news.gmane.org>,
 Nicholas Riley <njriley at illinois.edu> wrote:
> I'm trying to convert from an old Dovecot (1.0.15) and mbox to a new 
> Dovecot and maildir.  I'm running into several problems with dsync
which
> are causing my mboxes not to convert.
> 
> If I use LAYOUT=fs (as I'd prefer) I get an error because I have a 
> mailbox called "subscriptions", and none of my messages convert. 
Is
> there a way to get dovecot to put its own metadata in dotfiles instead?  
> (I guess I can rename my mailbox too :-)
OK, I found in the source code that I can use SUBSCRIPTIONS=... to fix 
this, although oddly enough I don't see a file being created now.  (This 
might be documented in http://wiki2.dovecot.org/MailLocation or 
http://wiki2.dovecot.org/MailLocation/Maildir...)

So now I get the same crash as before with LAYOUT=fs, and my 
mail_location:

mail_location = 
maildir:~/Maildir:INBOX=~/Maildir/INBOX:LAYOUT=fs:CONTROL=~/Maildir/.dove
cot:SUBSCRIPTIONS=~/Maildir/.subscriptions
-- 
Nicholas Riley <njriley at illinois.edu>

In article <njriley-5CB377.20303521102010 at news.gmane.org>,
 Nicholas Riley <njriley at illinois.edu> wrote:
> So I tried without LAYOUT=fs, and dsync successfully converts some 
> mailboxes, then crashes:
I isolated this to a few mailboxes, one of which I have anonymized and 
posted here:

<http://sabi.net/temp/2010-05.bz2>

Let me know if you need any more info to reproduce the crash...
-- 
Nicholas Riley <njriley at illinois.edu>

On Thu, 2010-10-21 at 20:30 -0500, Nicholas Riley wrote:
> If I use LAYOUT=fs (as I'd prefer) I get an error because I have a 
> mailbox called "subscriptions", and none of my messages convert. 
Is
> there a way to get dovecot to put its own metadata in dotfiles instead?  
> (I guess I can rename my mailbox too :-)
Besides the :SUBSCRIPTIONS=, you could also have put all mailboxes under
another directory, like with ":MAILBOXDIR=mailboxes" the subscriptions
mailbox would be in "mailboxes/subscriptions".
> % dsync mirror mbox:~/Mail
> dsync(nriley): Error: read() failed with subscription file 
> /home/nriley/Maildir/subscriptions: Is a directory
> dsync(nriley): Error: Can't update subscription +/INBOX: Internal error
> occurred. Refer to server log for more information. [2010-10-21 20:17:55]
> [...]
> 
> Also, what "server log" is it referring to above?
Well, it means the error message just above it. Maybe this error message
could be changed when running standalone programs..
> dsync(nriley): Error: Maildir: 6ef533cd4cdb1fd454b0226bfc851bb3:2, 
> changed UID 1846529 -> 1846531
> dsync(nriley): Panic: file maildir-save.c: line 892 
> (maildir_save_sync_uidlist): assertion failed: (ret > 0)
Fixed: http://hg.dovecot.org/dovecot-2.0/rev/c8eb81e1b5d6