dovecot - May 2008 - MBOX to Maildir preserving the UIDL (outlook-tested, no redownloading)

TESTED AND RUNNING, MIGRATING from MBOX to MAILDIR, PRESERVE UID and OUTLOOK
sees the email as downloaded.
   I hope this helps a lot of people out there.
   Feel free to post this somewhere else too.
   By: Horn Wijaya (hwijaya at nodens.biz) 8th May 2008 

I have been searching for weeks for a foolproof method to migrate all my
users mbox to maildir/dovecot without having the pain of downloading all
emails again, especially for heavy users. It would cause frustration to
thousands of my users. 

So.... eventually I found out why Outlook will keep downloading the emails
despite all the configuration done in the dovecot.conf.
Apparently the UID is generated realtime. When it first generated, it took
the time the process runs.

Before I migrate, here's the email sample which I have downloaded to my
outlook (and keep email on server), and then I use telnet to pop to get the
uidl list. 
   +OK POP3 localhost.localdomain v2003.83rh server ready
   user *****
   +OK User name accepted, password please
   pass *****
   +OK Mailbox open, 444 messages
   uidl
   +OK Unique-ID listing follows
   1 47b1303d00004c2d
   2 47b1303d00004c2e
   3 47b1303d00004c2f
   ... 

47b1303d = 1202794557 = 12 Feb 2008 13:35:57 the date and time the mbox was
created 
00004c2d = 19501 the IMAP counter so far.... sequently added as you could
see with email #2 and email #3 

Then I switch my POP server to dovecot, and migrate my MBOX to Maildir, and
I do a telnet to get the UIDL 
   +OK Dovecot ready.
   user *****
   +OK
   pass *****
   +OK Logged in.
   uidl
   +OK
   1 4822815000000001
   2 4822815000000002
   3 4822815000000003
   ... 

48228150 = 1210220880 = NOW 
00000001 = counter starts at 1 No wonder outlook sees the emails are
NOT-DOWNLOADED-YET status. 

Then I ponder for another few days and the key is in the
dovecot-uidlist...... How to fill it with the correct info.... and there's a
way. Which I hope useful for converter developer, and could be implemented
in the next version of the converter.

Here's the step I do to reach my objective which is to get UID
47b1303d00004c2d in the first email: 

1. mb2md-3.20.pl -s /var/mail/hwijaya -d home/hwijaya/Maildir/
   that's converting 444 emails from mbox to 444 files in the Maildir/cur
folder
2. cd /home/hwijaya/Maildir/cur
3. grep IMAPbase *
   and I get
      1210219860.000000.mbox:2,S:X-IMAPbase: 1202794557 19944
   The last 2 strings are the KEY to the ORIGINAL UID 
4. So in order to prepare the environment to fool it, some calculations are
needed:
   How do we get 19501 as the first counter? as expected in the first POP
TEST above:
      19944 - 444 emails + 1 = 19501 
5. cd /home/hwijaya/Maildir
6. echo '3 V1202794557 N19501' > dovecot-uidlist 
7. DONE....... pop it via outlook and nothing will be downloaded, after pop
or imap is processed, the dovecot-uidlist is auto-filled, index auto
generated. 

Proof: 
   +OK Dovecot ready.
   user *****
   +OK
   pass *****
   +OK
   Logged in.
   uidl
   +OK
   1 47b1303d00004c2d
   2 47b1303d00004c2e
   3 47b1303d00004c2f
   ...

I assume the same method could be used for individual folders in the imap
(under .mailboxlist or .subscriptions) 

Please do let us know if there's any cases/environment that it won't
work.

Thanks very much, and a big migration job is in front of me now...  

------------------------------------------
Warmest Regards From:
Horn Wijaya
Consultant
hwijaya at nodens.biz
http://www.nodens.biz/

Nodens Solutions Pte Ltd
      "where there is freedom in your choice"

A small amendment regarding how to obtain the UID and COUNTER

I have tried on IMAP Folder. 
To get IMAP info like the IMAPbase, apparently it does not exist in the
files in cur folder of Drafts and Sent Items after I converted them to
Maildir

-----
mb2md-3.20.pl -s /var/mail/hwijaya/Drafts -d home/hwijaya/Maildir/.Drafts
mb2md-3.20.pl -s "/var/mail/hwijaya/Sent Items" -d
"/home/hwijaya/Maildir/.Sent Items"
/home/hwijaya/Maildir/subscriptions has to be created based on existing
folders, in this case:
Drafts
Sent Items
-----

cd /home/hwijaya/Maildir/.Drafts/cur
grep IMAP *
   and I get nothing
Apparently, I have to grep IMAP /home/hwijaya/Drafts (mbox format)
   that returns
      X-IMAP: 1210246344 0000000444
   Hence: echo '3 V1210246344 N1' >
/home/hwijaya/Maildir/.Drafts/dovecot-uidlist  

Same procedure with my Sent / Sent Items folder

so it's safe to just grep the IMAP from the mbox format (inbox, drafts,
sent, etc) itself before migration.
the rest of the previous procedure stay.

I have successfully migrated an mbx imap account into a Maildir imap account
(accessed by OUtlook and Squirrelmail). No outlook redownloading.

Cheers.

------------------------------------------
Warmest Regards From:
Horn Wijaya
Consultant
hwijaya at nodens.biz
http://www.nodens.biz/

Nodens Solutions Pte Ltd
      "where there is freedom in your choice"

Horn Wijaya wrote:
> TESTED AND RUNNING, MIGRATING from MBOX to MAILDIR, PRESERVE UID and
OUTLOOK
>
> 1. mb2md-3.20.pl -s /var/mail/hwijaya -d home/hwijaya/Maildir/
>    that's converting 444 emails from mbox to 444 files in the
Maildir/cur
> folder
> 2. cd /home/hwijaya/Maildir/cur
> 3. grep IMAPbase *
>    and I get
>       1210219860.000000.mbox:2,S:X-IMAPbase: 1202794557 19944
>    The last 2 strings are the KEY to the ORIGINAL UID 
> 4. So in order to prepare the environment to fool it, some calculations are
> needed:
>    How do we get 19501 as the first counter? as expected in the first POP
> TEST above:
>       19944 - 444 emails + 1 = 19501 
> 5. cd /home/hwijaya/Maildir
> 6. echo '3 V1202794557 N19501' > dovecot-uidlist 
>   
Hello,

 First of all, I was looking into this problem of getting the right 
converter for couple years. You can search list archives for that - 
there are more problems with perfect migration, than you know of.
 Second, your calculations assume two things, that are not always true -
1) original mailbox already has X-IMAPbase header (not every mail server 
puts these headers)
2) first message UID is last message UID-<number of messages>, which is 
not true, if you have deleted some messages in between.
 Third, usually servers use different UIDL format, not time+number.
 Last, if you are migrating from mbox to Maildir on IMAP server, then 
you have more than one folder. Probably you have "/" as folder
separator
on mbox-based system. And will have "." on Maildir. That means,
Outlook
will need to refresh it's folder list anyway. And that involves 
re-downloading all messages.

Yes, in your case you might be able to make a perfect migration tool, 
but this won't be ideal for everybody.


P.S. After couple years of thinking, planning and looking for a ideal 
solution I have modified mb2md-3.20.pl script a little and was able to 
migrate about 200Gb of mail with no problem. Yes, most clients had to 
re-download all their messages (I had to force POP3 clients to empty 
their server mailboxes), but it wasn't that big of a deal, as I was 
afraid of. And now, even with much slower hardware I see about 10 times 
lower LA than with mbox.

    FiL

http://dovecot.org/list/dovecot/2008-March/029736.html points to a  
script that should preserve the UIDs. It's also linked to in
http://wiki.dovecot.org/Migration/MailFormat

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 194 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20080508/59a68976/attachment-0002.bin>

Thanks for the input, will try that out see whether it will work on my
environment. 

------------------------------------------
Warmest Regards From:
Horn Wijaya
Consultant
hwijaya at nodens.biz
http://www.nodens.biz/

Nodens Solutions Pte Ltd
      "where there is freedom in your choice"
-----Original Message-----
From: Timo Sirainen [mailto:tss at iki.fi] 
Sent: Thursday, May 08, 2008 11:55 PM
To: Horn Wijaya
Cc: dovecot at dovecot.org
Subject: Re: [Dovecot] MBOX to Maildir preserving the UIDL (outlook-tested,
no redownloading)

http://dovecot.org/list/dovecot/2008-March/029736.html points to a  
script that should preserve the UIDs. It's also linked to in
http://wiki.dovecot.org/Migration/MailFormat

on second thought, after I search through my tests, That's the first script
I tested, and it failed. 
.py tested
.pl tested
they didn't support the preservation of UID.

The posting was meant for informative purposes, who knows someone who stuck
like me could find something out of it.
So yeah, I guess I will stick to my own method for my own environment, at
least I know it would work :)

------------------------------------------
Warmest Regards From:
Horn Wijaya
Consultant
hwijaya at nodens.biz
http://www.nodens.biz/

Nodens Solutions Pte Ltd
      "where there is freedom in your choice"
-----Original Message-----
From: dovecot-bounces+hwijaya=nodens.biz at dovecot.org
[mailto:dovecot-bounces+hwijaya=nodens.biz at dovecot.org] On Behalf Of Timo
Sirainen
Sent: Thursday, May 08, 2008 11:55 PM
To: Horn Wijaya
Cc: dovecot at dovecot.org
Subject: Re: [Dovecot] MBOX to Maildir preserving the UIDL
(outlook-tested,no redownloading)

http://dovecot.org/list/dovecot/2008-March/029736.html points to a  
script that should preserve the UIDs. It's also linked to in
http://wiki.dovecot.org/Migration/MailFormat