dovecot - Oct 2012 - how to best import Evolution/Thunderbird mail into dovecot?

Hi.

I'm migrating all my mail archive (some 60 GB) from Evolution (which is
really a broken piece of software) into dovecot.
Now I face the problem how to do this best...


Evolution (which is still a old 2.32.x version) itself uses mbox files,
in a special hierarchical structure to allow subfolders and that like.

It also stores it's own status info in X-Evolution and
X-Evolution-Source mail headers.

Unfortunately,... much of the mail was earlier from a Thunderbird
installation, which uses it's own status headers (X-Mozilla*) that were
not recognised by Evolution.


I have no idea which mbox subformat was always used throughout the
different programs and versions...
1) Any way to check for that?

To make things worse... Thunder(burden) seems to have used a modified
From_ line syntax... "^From -<address> <date>$"


Ideally I'd like migrate all mail into dovecot (for storage reasons
again, mbox) retaining the different status flags (read, forwarded,
etc.) and getting rid of the proprietary headers (of course only when
they were.



First thing I tried was to simply copy mail within Evolution (i.e.
dragging&dropping it from the local folders to the IMAP folders from
dovecot).
- that preserves the status from Evolution, but doesn't restore that
from Thunderbird
- it clutters up the information of all From_ lines... "<address>
<date>" becomes "<address of my default evolution account>
<now>"
- neither does it handle the special Thunderbird From_ lines
- neither does it remove the Thunderbird or the X-Evolution-Source
headers
- if Evolution has already had corrupted index files (and this is
extremely likely... as it happens even immediately during recreating all
of them)... so I may loose mail


So my idea was that I need a program that:
- can parse all the different mbox formats (those that use the quoted
">From" style and those that use Conent-Length)
- can differentiate message headers from body (so that I can drop the
proprietary headers and replace them by what dovecot uses as headers)
- must of course understand multiline message headers

2) Any idea for a tool like that?
The meaning of the X-Evolution and X-Mozilla* headers are easy to find
on the web.... so I can convert them.
So I basically "just" need a tool that parses all kinds of mbox
formats... allow me to drop/add headers... and spits out the rest
unmodified.


3) dovecot uses some special headers like X-UID and X-IMAPD... will it
create these on it's own, the first time it processes the new mbox file?
I mean these headers won't be there after creation.

4) Should I drop out (during) conversion... any other mail headers..
that dovecot uses as it's own?



Thanks,
Chris.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5450 bytes
Desc: not available
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20121017/e1769d45/attachment-0004.bin>

Hi again...

Things are even much much worse... (oh how I hate Evolution right now).


I found a bug in Evolution, where it apparently corrupts all mail by
incorrectly (not) quoting From_ lines in headers/bodies...
It quotes lines matching "^From (.*)$" as ">From \1" but
it does not
quote at all already quoted From_ lines, i.e. "^>+From .*$".

Now that means AFAICS, that it's not possible to repair that corruption
(you'll see my "happiness" about this, when reading the offensive
bug
report):
Details here: https://bugzilla.gnome.org/show_bug.cgi?id=686258


I'm not sure how this affects any of my migration/conversion plans...
any ideas?


Thanks,
a desperate Chris.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5450 bytes
Desc: not available
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20121017/a667f23a/attachment-0004.bin>

On Wed, Oct 17, 2012 at 01:21:14AM +0200, Christoph Anton Mitterer
wrote:
> Hi.
[..]
> First thing I tried was to simply copy mail within Evolution (i.e.
> dragging&dropping it from the local folders to the IMAP folders from
> dovecot).
This seems to be the smartest idea.
> - that preserves the status from Evolution, but doesn't restore that
> from Thunderbird
Why not use TB to copy the emails from your 'TB mboxes' to Dovecot?
This way I moved around 25 GiB of emails from >> 50 mbox files,
created with TB 3.6 way down to some 0.x beta, to Dovecot -- without
any issues I could recall.

[..]
> - neither does it remove the Thunderbird or the X-Evolution-Source
> headers
If they bug you remove them with sed or awk or perl or python or ...

Dennis

[..]