dovecot - Feb 2006 - Problems with POP3 UIDL when migrating from MBOX to Maildir

Hello,

I am in the process of migrating a hosting setup from UW-IMAP to
Dovecot. The protocols available to mail clients are IMAP and POP3
both before and after migration. I also wanted to change the mail
storage format from MBOX to Maildir. However, at this point I hit a
major snag with Dovecot and POP3 UIDL (unique identification listing
for POP3 mailboxes).

Several of the customers connecting to the mailserver in question are
using POP3 clients and leave their e-mails on the server. To avoid
downloading all messages every time the clients connect they use the
UIDL command in the POP3 specification to determine if any new
messages have arrived since last time they checked.

In order for this to work the unique identifier of each message must
be static and can never change. If the identifiers change, every
client will download all messages again, mixing old messages with new,
in the belief that all messages are new since their identifiers
changed.

The first step, migrating from UW-IMAP to Dovecot, did not pose any
problems in this regard. Connecting to the POP3 service manually to
show the UIDL output revealed:

With UW-IMAP:
uidl
+OK Unique-ID listing follows
1 43ef424e00000001
2 43ef424e00000002

With Dovecot:
uidl
+OK
1 43ef424e00000001
2 43ef424e00000002

However, after performing a migration to Maildir, storing all messages
in ~/Maildir for the user in question, I received a different UIDL
output:

uidl
+OK
1 43ef523100000001
2 43ef523100000002

I used the mb2md utility available from
http://batleth.sapienti-sat.org/projects/mb2md/ to do the conversion.
A diff between the new Maildir messages and the old mailbox reveals no
other difference than the leading "From " line of each message missing
from the Maildir messages.

----------- snip ------------------------------
--- /var/mail/testuser  2006-02-12 15:13:01.000000000 +0100
+++ maildir-concat.txt  2006-02-12 15:21:54.000000000 +0100
@@ -1,4 +1,3 @@
-From test at example.com  Sun Feb 12 15:09:49 2006
 Return-Path: <test at example.com>
  X-Original-To: testuser
  Delivered-To: testuser at localhost.localdomain
@@ -16,7 +15,6 @@

 Testing 1

-From test at example.com  Sun Feb 12 15:10:23 2006
 Return-Path: <test at example.com>
  X-Original-To: testuser
  Delivered-To: testuser at localhost.localdomain
----------- snip ------------------------------

When removing the ~/Maildir directory of the user to redo the
conversion I discovered that Dovecot returned another set of UIDL
output this time around:

uidl
+OK
1 43ef528300000001
2 43ef528300000002

This output is for the same messages as in the previous example, with
preserved message filenames. Doing a diff between the old ~/Maildir
and the new reveals the following differences:

----------- snip ------------------------------
$ diff -ur Maildir.example1 Maildir.example2
Binary files Maildir.example1/dovecot.index and
Maildir.example2/dovecot.index differ
Binary files Maildir.example1/dovecot.index.cache and
Maildir.example2/dovecot.index.cache differ
Binary files Maildir.example1/dovecot.index.log and
Maildir.example2/dovecot.index.log differ
diff -ur Maildir.example1/dovecot-uidlist Maildir.example2/dovecot-uidlist
--- Maildir.example1/dovecot-uidlist    2006-02-12 16:20:17.000000000 +0100
+++ Maildir.example2/dovecot-uidlist    2006-02-12 16:21:39.000000000 +0100
@@ -1,3 +1,3 @@
-1 1139757617 3
+1 1139757699 3
 1 1139757559.000001.mbox:2,
 2 1139757559.000000.mbox:2,
----------- snip ------------------------------


Questions:
----------------

How does Dovecot generate the POP3 UIDL?

Is there any workaround that I could use to make Dovecot generate a
consistent POP3 UIDL output, enabling a migration from MBOX to
Maildir, or am I stuck with using MBOX?


My environment
-----------------------

OS: CentOS 4.2
Dovecot version: 1.0 beta3 (built locally to an RPM package, available
on request)

I have tested with earlier versions of Dovecot, getting the same behaviour.

Relevant entries in /etc/dovecot.conf:

----------- snip ------------------------------
protocols = imap imaps pop3 pop3s
listen = *
disable_plaintext_auth = no

protocol pop3 {
  pop3_uidl_format = %08Xv%08Xu
  pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
}
----------- snip ------------------------------

--
/Jonas Olsson

On Sun, 2006-02-12 at 16:20 +0100, Jonas Olsson wrote:
> Questions:
> ----------------
> 
> How does Dovecot generate the POP3 UIDL?
Using pop3_uidl_format. %v means UIDVALIDITY and %u means message UID.
With maildir those are taken from dovecot-uidlist file.
> Is there any workaround that I could use to make Dovecot generate a
> consistent POP3 UIDL output, enabling a migration from MBOX to
> Maildir, or am I stuck with using MBOX?
You can convert your mails in a way that preserves the UIDVALIDITY and
UIDs. In mboxes the UIDVALIDITY is stored in X-IMAPbase or X-IMAP header
in the first mail in the mailbox. UIDs are stored in X-UID header for
each mail.

So what you'd need to do is create some script to build the
dovecot-uidlist file properly. As far as I know no-one's done this yet,
but it shouldn't be too difficult.

dovecot-uidlist file format is:

<uidvalidity> <next-unused-uid>
<uid> <filename>
<next uid> <next filename>
..

You should also set next-unused-uid value properly. It's also in
X-IMAPbase/X-IMAP header as the next number after UIDVALIDITY.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20060212/0f2d6acc/attachment.bin>