dovecot - Sep 2018 - Error using dsync to convert really old mboxes to Maildir and workaround

I ran across what I think may be a bug in dsync when trying to convert from mbox
to Maildir format. My first attempt failed with a runaway dsync process spewing
out thousands of identical lines of the form:

dsync(buus): Error: Duplicate mailbox GUID 00fe691ff60da85b090d0000c8119523 for
mailboxes Archive/Personal and Work/VendorInfo - giving a new GUID
00fe691ff60da85b090d0000c8119523 to Archive/Personal

These two mail folders happen to be among a number of folders I've had for
many years, but also haven't looked at in many years. I used to use UW-IMAP,
but switched to dovecot about two years ago, and I'm sure I've never
used dovecot to look at those folders. So I suspect the root cause of the above
error message was that dovecot had never generated GUIDs for these and some
other really old folders.

After killing dsync with SIGKILL, which couldn't be killed with SIGTERM, I
restored my mbox format Mail folder from backup, restored the mail_location and
then coerced dovecot to generate unique GUIDs for all my mail folders with this
command:

doveadm mailbox status guid `doveadm mailbox list`

After the above command, I was able to convert my mail from mbox to Maildir
format.

Maybe this will help someone else somewhere down the road.

On 9/24/2018 1:50 AM, faginbagin wrote:
> I ran across what I think may be a bug in dsync when trying to convert from
mbox to Maildir format. My first attempt failed with a runaway dsync process
spewing out thousands of identical lines of the form:
> 
> dsync(buus): Error: Duplicate mailbox GUID 00fe691ff60da85b090d0000c8119523
for mailboxes Archive/Personal and Work/VendorInfo - giving a new GUID
00fe691ff60da85b090d0000c8119523 to Archive/Personal
> 
> These two mail folders happen to be among a number of folders I've had
for many years, but also haven't looked at in many years. I used to use
UW-IMAP, but switched to dovecot about two years ago, and I'm sure I've
never used dovecot to look at those folders. So I suspect the root cause of the
above error message was that dovecot had never generated GUIDs for these and
some other really old folders.
> 
> After killing dsync with SIGKILL, which couldn't be killed with
SIGTERM, I restored my mbox format Mail folder from backup, restored the
mail_location and then coerced dovecot to generate unique GUIDs for all my mail
folders with this command:
> 
> doveadm mailbox status guid `doveadm mailbox list`
> 
> After the above command, I was able to convert my mail from mbox to Maildir
format.
> 
> Maybe this will help someone else somewhere down the road.
Forgot to provide dovecot -n output:

# 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.16 (fed8554)
# OS: Linux 4.9.0-8-686-pae i686 Debian 9.5
mail_location = maildir:~/Maildir
# mail_location was the following when error occurred:
# mail_location = mbox:~/Mail:INBOX=/var/mail/%u:UTF-8
# Also separator was not set when error occurred.
mail_privileged_group = mail
namespace inbox {
  inbox = yes
  location   mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix   separator = /
}
passdb {
  driver = pam
}
protocols = " imap"
ssl_cert = </etc/dovecot/dovecot.pem
ssl_key =  # hidden, use -P to show it
userdb {
  driver = passwd
}

On 24.09.2018 09:10, faginbagin wrote:
> On 9/24/2018 1:50 AM, faginbagin wrote:
>> I ran across what I think may be a bug in dsync when trying to convert
from mbox to Maildir format. My first attempt failed with a runaway dsync
process spewing out thousands of identical lines of the form:
>>
>> dsync(buus): Error: Duplicate mailbox GUID
00fe691ff60da85b090d0000c8119523 for mailboxes Archive/Personal and
Work/VendorInfo - giving a new GUID 00fe691ff60da85b090d0000c8119523 to
Archive/Personal
>>
>> These two mail folders happen to be among a number of folders I've
had for many years, but also haven't looked at in many years. I used to use
UW-IMAP, but switched to dovecot about two years ago, and I'm sure I've
never used dovecot to look at those folders. So I suspect the root cause of the
above error message was that dovecot had never generated GUIDs for these and
some other really old folders.
>>
>> After killing dsync with SIGKILL, which couldn't be killed with
SIGTERM, I restored my mbox format Mail folder from backup, restored the
mail_location and then coerced dovecot to generate unique GUIDs for all my mail
folders with this command:
>>
>> doveadm mailbox status guid `doveadm mailbox list`
>>
>> After the above command, I was able to convert my mail from mbox to
Maildir format.
>>
>> Maybe this will help someone else somewhere down the road.
> Forgot to provide dovecot -n output:
>
> # 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.4.16 (fed8554)
> # OS: Linux 4.9.0-8-686-pae i686 Debian 9.5
> mail_location = maildir:~/Maildir
> # mail_location was the following when error occurred:
> # mail_location = mbox:~/Mail:INBOX=/var/mail/%u:UTF-8
> # Also separator was not set when error occurred.
> mail_privileged_group = mail
> namespace inbox {
>   inbox = yes
>   location >   mailbox Drafts {
>     special_use = \Drafts
>   }
>   mailbox Junk {
>     special_use = \Junk
>   }
>   mailbox Sent {
>     special_use = \Sent
>   }
>   mailbox "Sent Messages" {
>     special_use = \Sent
>   }
>   mailbox Trash {
>     special_use = \Trash
>   }
>   prefix >   separator = /
> }
> passdb {
>   driver = pam
> }
> protocols = " imap"
> ssl_cert = </etc/dovecot/dovecot.pem
> ssl_key =  # hidden, use -P to show it
> userdb {
>   driver = passwd
> }
You could try running doveadm index -u username "*" on the old
location
before dsyncing.

Aki