Jonas Olsson
2006-Feb-12 15:20 UTC
[Dovecot] Problems with POP3 UIDL when migrating from MBOX to Maildir
Hello, I am in the process of migrating a hosting setup from UW-IMAP to Dovecot. The protocols available to mail clients are IMAP and POP3 both before and after migration. I also wanted to change the mail storage format from MBOX to Maildir. However, at this point I hit a major snag with Dovecot and POP3 UIDL (unique identification listing for POP3 mailboxes). Several of the customers connecting to the mailserver in question are using POP3 clients and leave their e-mails on the server. To avoid downloading all messages every time the clients connect they use the UIDL command in the POP3 specification to determine if any new messages have arrived since last time they checked. In order for this to work the unique identifier of each message must be static and can never change. If the identifiers change, every client will download all messages again, mixing old messages with new, in the belief that all messages are new since their identifiers changed. The first step, migrating from UW-IMAP to Dovecot, did not pose any problems in this regard. Connecting to the POP3 service manually to show the UIDL output revealed: With UW-IMAP: uidl +OK Unique-ID listing follows 1 43ef424e00000001 2 43ef424e00000002 With Dovecot: uidl +OK 1 43ef424e00000001 2 43ef424e00000002 However, after performing a migration to Maildir, storing all messages in ~/Maildir for the user in question, I received a different UIDL output: uidl +OK 1 43ef523100000001 2 43ef523100000002 I used the mb2md utility available from http://batleth.sapienti-sat.org/projects/mb2md/ to do the conversion. A diff between the new Maildir messages and the old mailbox reveals no other difference than the leading "From " line of each message missing from the Maildir messages. ----------- snip ------------------------------ --- /var/mail/testuser 2006-02-12 15:13:01.000000000 +0100 +++ maildir-concat.txt 2006-02-12 15:21:54.000000000 +0100 @@ -1,4 +1,3 @@ -From test at example.com Sun Feb 12 15:09:49 2006 Return-Path: <test at example.com> X-Original-To: testuser Delivered-To: testuser at localhost.localdomain @@ -16,7 +15,6 @@ Testing 1 -From test at example.com Sun Feb 12 15:10:23 2006 Return-Path: <test at example.com> X-Original-To: testuser Delivered-To: testuser at localhost.localdomain ----------- snip ------------------------------ When removing the ~/Maildir directory of the user to redo the conversion I discovered that Dovecot returned another set of UIDL output this time around: uidl +OK 1 43ef528300000001 2 43ef528300000002 This output is for the same messages as in the previous example, with preserved message filenames. Doing a diff between the old ~/Maildir and the new reveals the following differences: ----------- snip ------------------------------ $ diff -ur Maildir.example1 Maildir.example2 Binary files Maildir.example1/dovecot.index and Maildir.example2/dovecot.index differ Binary files Maildir.example1/dovecot.index.cache and Maildir.example2/dovecot.index.cache differ Binary files Maildir.example1/dovecot.index.log and Maildir.example2/dovecot.index.log differ diff -ur Maildir.example1/dovecot-uidlist Maildir.example2/dovecot-uidlist --- Maildir.example1/dovecot-uidlist 2006-02-12 16:20:17.000000000 +0100 +++ Maildir.example2/dovecot-uidlist 2006-02-12 16:21:39.000000000 +0100 @@ -1,3 +1,3 @@ -1 1139757617 3 +1 1139757699 3 1 1139757559.000001.mbox:2, 2 1139757559.000000.mbox:2, ----------- snip ------------------------------ Questions: ---------------- How does Dovecot generate the POP3 UIDL? Is there any workaround that I could use to make Dovecot generate a consistent POP3 UIDL output, enabling a migration from MBOX to Maildir, or am I stuck with using MBOX? My environment ----------------------- OS: CentOS 4.2 Dovecot version: 1.0 beta3 (built locally to an RPM package, available on request) I have tested with earlier versions of Dovecot, getting the same behaviour. Relevant entries in /etc/dovecot.conf: ----------- snip ------------------------------ protocols = imap imaps pop3 pop3s listen = * disable_plaintext_auth = no protocol pop3 { pop3_uidl_format = %08Xv%08Xu pop3_client_workarounds = outlook-no-nuls oe-ns-eoh } ----------- snip ------------------------------ -- /Jonas Olsson
Timo Sirainen
2006-Feb-12 20:44 UTC
[Dovecot] Problems with POP3 UIDL when migrating from MBOX to Maildir
On Sun, 2006-02-12 at 16:20 +0100, Jonas Olsson wrote:> Questions: > ---------------- > > How does Dovecot generate the POP3 UIDL?Using pop3_uidl_format. %v means UIDVALIDITY and %u means message UID. With maildir those are taken from dovecot-uidlist file.> Is there any workaround that I could use to make Dovecot generate a > consistent POP3 UIDL output, enabling a migration from MBOX to > Maildir, or am I stuck with using MBOX?You can convert your mails in a way that preserves the UIDVALIDITY and UIDs. In mboxes the UIDVALIDITY is stored in X-IMAPbase or X-IMAP header in the first mail in the mailbox. UIDs are stored in X-UID header for each mail. So what you'd need to do is create some script to build the dovecot-uidlist file properly. As far as I know no-one's done this yet, but it shouldn't be too difficult. dovecot-uidlist file format is: <uidvalidity> <next-unused-uid> <uid> <filename> <next uid> <next filename> .. You should also set next-unused-uid value properly. It's also in X-IMAPbase/X-IMAP header as the next number after UIDVALIDITY. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: <http://dovecot.org/pipermail/dovecot/attachments/20060212/0f2d6acc/attachment.bin>
Reasonably Related Threads
- Filserver in Domain-trusts
- Perfect LDAP tree
- managesieve script 'redirect' fails @ "Error: sieve: ... aborted due to temporary failure; Error: smtp-server: ... failed: SSL_accept() failed: error:1408F10B:SSL routines:ssl3_get_record:wrong version number"; direct send OK ?
- syncing imap servers with imapsync
- [PATCH 03/13] syntax-check: fix makefile_at_at_check