dovecot - Mar 2015 - Migrating from Cyrus to Dovecot

Hi,

we are migrating from Cyrus 2.3.7 to Dovecot 2.2.13. We have ~7000 
maildirs with ~500GB. Our goal is to do the migration without users have 
notice and with the shortest service downtime. The users use IMAP (with 
shared folders and ACL), POP3 and sieve filters.

As a first choice, we tried the Dovecot's dsync tool. First tests were 
great, but we are not able to change the Cyrus auth backend for 
migration. Moreover, this migration seems too slow for us.

As a second try, we tried the cyrus2dovecot migrating Perl scripts (and 
their derivates) from Wiki2. More or less they works but we found we 
need more control during the migration.

So, as a third try, we wrote own migrating scripts. And thanks to the 
cyrus2dovecot it wasn't too much complicated. And there are my questions:

A) Files and dirs timestamps
The mtime of email file is important as an internal date as I found on 
Wiki2. But what about timestamps of cur/new/tmp directories or Dovecot's 
internal files line dovecot-uidlist? Do they play some role here?

B) The 128 bit mailbox UID
The Wiki2 speaks about 128 bit mailbox UID at first line of 
dovecot-uidlist. Cyrus preserves only 64 bit UID. Is this mailbox UID 
required by Dovecot? If so, can we use 0000000000000000501100008c4a11c1 
(Cyrus UID padded by zeros)?

C) Format of dovecot-uidlist records
Wiki2 shows two examples:
25006 :1276528487.M364837P9451.kurkku,S=1355,W=1394:2,
25017 W2481 :1276533073.M242911P3632.kurkku:2,F

Which format is preferred? Or what the benefits are?

D) Converting between CRLF and LF
If I understand correctly, Dovecot stores emails with LF only. We have 
all emails with CRLF now on Cyrus and converting them to LF only is a 
little more time consuming. Is there any benefit to do that? Or can we 
live with 'mail_save_crlf' without problems?

E) POP3 backend
I found many informations about IMAP internals but few on POP3 
internals. What do I need to do POP3 migration transparent for user?

Many thanks for any answers.

Regards, Milo

> On 27 Mar 2015, at 10:19, Miloslav H?la <miloslav.hula at gmail.com>
wrote:
> 
> Hi,
> 
> we are migrating from Cyrus 2.3.7 to Dovecot 2.2.13. We have ~7000 maildirs
with ~500GB. Our goal is to do the migration without users have notice and with
the shortest service downtime. The users use IMAP (with shared folders and ACL),
POP3 and sieve filters.
> 
> As a first choice, we tried the Dovecot's dsync tool. First tests were
great, but we are not able to change the Cyrus auth backend for migration.
Moreover, this migration seems too slow for us.
> 
> As a second try, we tried the cyrus2dovecot migrating Perl scripts (and
their derivates) from Wiki2. More or less they works but we found we need more
control during the migration.
> 
> So, as a third try, we wrote own migrating scripts. And thanks to the
cyrus2dovecot it wasn't too much complicated. And there are my questions:
> 
> A) Files and dirs timestamps
> The mtime of email file is important as an internal date as I found on
Wiki2. But what about timestamps of cur/new/tmp directories or Dovecot's
internal files line dovecot-uidlist? Do they play some role here?
No.
> B) The 128 bit mailbox UID
> The Wiki2 speaks about 128 bit mailbox UID at first line of
dovecot-uidlist. Cyrus preserves only 64 bit UID. Is this mailbox UID required
by Dovecot? If so, can we use 0000000000000000501100008c4a11c1 (Cyrus UID padded
by zeros)?
The mailbox GUID is internal to Dovecot. There's no standard IMAP way to see
it, so there's no need to migrate it. Better not to set it and let Dovecot
generate it automatically.
> C) Format of dovecot-uidlist records
> Wiki2 shows two examples:
> 25006 :1276528487.M364837P9451.kurkku,S=1355,W=1394:2,
> 25017 W2481 :1276533073.M242911P3632.kurkku:2,F
> 
> Which format is preferred? Or what the benefits are?
If W=size is in the filename, it never needs to be recalculated if
dovecot-uidlist is lost. Of course, dovecot-uidlist should never be lost. So I
don't think it makes a huge difference. If you care about performance,
sdbox/mdbox mailbox format would behave much better. sdbox is a close match to
Cyrus - so with Maildir you're actually likely making the disk I/O
performance somewhat slower in Dovecot than in Cyrus, although that also depends
on other things.
> D) Converting between CRLF and LF
> If I understand correctly, Dovecot stores emails with LF only. We have all
emails with CRLF now on Cyrus and converting them to LF only is a little more
time consuming. Is there any benefit to do that? Or can we live with
'mail_save_crlf' without problems?
Dovecot can automatically handle both mixed CRLF and LF mails, you can keep old
mails as CRLF and new mails as LF. mail_save_crlf setting only controls what is
used for new emails. If you want to save more disk space you can enable
compression.
> E) POP3 backend
> I found many informations about IMAP internals but few on POP3 internals.
What do I need to do POP3 migration transparent for user?
Just preserve the UIDL. See the pop3_uidl_format setting in
http://wiki2.dovecot.org/Migration/Cyrus

Hi Timo,

thank you for the valuable answers!

Milo


Dne 27.3.2015 v 22:12 Timo Sirainen napsal(a):
>> On 27 Mar 2015, at 10:19, Miloslav H?la <miloslav.hula at
gmail.com> wrote:
>>
>> Hi,
>>
>> we are migrating from Cyrus 2.3.7 to Dovecot 2.2.13. We have ~7000
maildirs with ~500GB. Our goal is to do the migration without users have notice
and with the shortest service downtime. The users use IMAP (with shared folders
and ACL), POP3 and sieve filters.
>>
>> As a first choice, we tried the Dovecot's dsync tool. First tests
were great, but we are not able to change the Cyrus auth backend for migration.
Moreover, this migration seems too slow for us.
>>
>> As a second try, we tried the cyrus2dovecot migrating Perl scripts (and
their derivates) from Wiki2. More or less they works but we found we need more
control during the migration.
>>
>> So, as a third try, we wrote own migrating scripts. And thanks to the
cyrus2dovecot it wasn't too much complicated. And there are my questions:
>>
>> A) Files and dirs timestamps
>> The mtime of email file is important as an internal date as I found on
Wiki2. But what about timestamps of cur/new/tmp directories or Dovecot's
internal files line dovecot-uidlist? Do they play some role here?
>
> No.
>
>> B) The 128 bit mailbox UID
>> The Wiki2 speaks about 128 bit mailbox UID at first line of
dovecot-uidlist. Cyrus preserves only 64 bit UID. Is this mailbox UID required
by Dovecot? If so, can we use 0000000000000000501100008c4a11c1 (Cyrus UID padded
by zeros)?
>
> The mailbox GUID is internal to Dovecot. There's no standard IMAP way
to see it, so there's no need to migrate it. Better not to set it and let
Dovecot generate it automatically.
>
>> C) Format of dovecot-uidlist records
>> Wiki2 shows two examples:
>> 25006 :1276528487.M364837P9451.kurkku,S=1355,W=1394:2,
>> 25017 W2481 :1276533073.M242911P3632.kurkku:2,F
>>
>> Which format is preferred? Or what the benefits are?
>
> If W=size is in the filename, it never needs to be recalculated if
dovecot-uidlist is lost. Of course, dovecot-uidlist should never be lost. So I
don't think it makes a huge difference. If you care about performance,
sdbox/mdbox mailbox format would behave much better. sdbox is a close match to
Cyrus - so with Maildir you're actually likely making the disk I/O
performance somewhat slower in Dovecot than in Cyrus, although that also depends
on other things.
>
>> D) Converting between CRLF and LF
>> If I understand correctly, Dovecot stores emails with LF only. We have
all emails with CRLF now on Cyrus and converting them to LF only is a little
more time consuming. Is there any benefit to do that? Or can we live with
'mail_save_crlf' without problems?
>
> Dovecot can automatically handle both mixed CRLF and LF mails, you can keep
old mails as CRLF and new mails as LF. mail_save_crlf setting only controls what
is used for new emails. If you want to save more disk space you can enable
compression.
>
>> E) POP3 backend
>> I found many informations about IMAP internals but few on POP3
internals. What do I need to do POP3 migration transparent for user?
>
> Just preserve the UIDL. See the pop3_uidl_format setting in
http://wiki2.dovecot.org/Migration/Cyrus