dovecot - May 2013 - should dovecot store maildir files with CRLF or LF?

Hi.

I've made a strange observation.
When having Dovecot (at least) with maildir and moving (via IMAP) mail
received by some client (Evolution 3.4) into it the following happens:

Regardless of whether the mail was originally(!) set with CRLF or LF
(i.e. when I use netcat to submit the plain SMTP to the relaying MTA).

When the client (Evolution) had received the mail via POP3 before moving
it via IMAP into Dovecot... then the maildir file within dovecot is all
LF.

When the client however received it via IMAP, before in turn moving it
on via IMAP into Dovecot, then the maildir file is mixed CRLF and LF,
i.e. the body is CRLF, the headers are terminated...


Well the actually bug here is probably in Evolution (as so many
others... o.O)... but I wondered... what is Dovecot expected to write
files? Platform end-of-line markers (i.e. LF in case of UNIX) or always
network end-of-line markers (CRLF)?


Cheers,
Chris.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5113 bytes
Desc: not available
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20130522/babaed69/attachment-0001.bin>

I'm not sure if this has any bearing on what you reported here, but I
do this:

fetchmail->procmail->dovecot->Claws-Mail

When I open certain emails, the empty lines between paragraphs are
missing. If this is part of what you're reporting, I can pay more
attention as to which emails display this anomaly.

Thanks,

SteveT

Steve Litt                *  http://www.troubleshooters.com/
Troubleshooting Training  *  Human Performance


On Wed, 22 May 2013 00:45:49 +0200
Christoph Anton Mitterer <calestyo at scientia.net> wrote:
> Hi.
> 
> I've made a strange observation.
> When having Dovecot (at least) with maildir and moving (via IMAP) mail
> received by some client (Evolution 3.4) into it the following happens:
> 
> Regardless of whether the mail was originally(!) set with CRLF or LF
> (i.e. when I use netcat to submit the plain SMTP to the relaying MTA).
> 
> When the client (Evolution) had received the mail via POP3 before
> moving it via IMAP into Dovecot... then the maildir file within
> dovecot is all LF.
> 
> When the client however received it via IMAP, before in turn moving it
> on via IMAP into Dovecot, then the maildir file is mixed CRLF and LF,
> i.e. the body is CRLF, the headers are terminated...
> 
> 
> Well the actually bug here is probably in Evolution (as so many
> others... o.O)... but I wondered... what is Dovecot expected to write
> files? Platform end-of-line markers (i.e. LF in case of UNIX) or
> always network end-of-line markers (CRLF)?
> 
> 
> Cheers,
> Chris.

At 12AM +0200 on 22/05/13 you (Christoph Anton Mitterer)
wrote:
> 
> I've made a strange observation.
> When having Dovecot (at least) with maildir and moving (via IMAP) mail
> received by some client (Evolution 3.4) into it the following happens:
> 
> Regardless of whether the mail was originally(!) set with CRLF or LF
> (i.e. when I use netcat to submit the plain SMTP to the relaying MTA).
Mail sent by SMTP has to have CRLF line endings. (Unless you're using
BINARYMIME, but I don't think that's at all widely implemented yet.) If
your mailserver accepts LF-only line endings it ought to translate them
into CRLF before sending the message on.
> When the client (Evolution) had received the mail via POP3 before moving
> it via IMAP into Dovecot... then the maildir file within dovecot is all
> LF.
What line endings is the POP server sending? Can you verify this without
involving Evolution? Is the POP server Dovecot or something else?

The original (djb) definition of Maildir assumed that messages would be
written to the maildir with LF line endings, and both MTA and POP server
would translate back to CRLF as needed. Dovecot (as a POP server) can
deal with messages in either format, and should always return them to
clients with CRLF.
> When the client however received it via IMAP, before in turn moving it
> on via IMAP into Dovecot, then the maildir file is mixed CRLF and LF,
> i.e. the body is CRLF, the headers are terminated...
IMAP is similar, in that line endings on the wire are always supposed to
be CRLF; it's a little more complicated in that clients can also upload
messages. If I APPEND a message with mixed line endings to a Maildir
Dovecot mailbox, the message is written to the Maildir with LF-only
endings and comes back over IMAP with CRLF-only.

Can you confirm what is actually being sent over the wire?

Ben

On 22.5.2013, at 1.45, Christoph Anton Mitterer <calestyo at scientia.net>
wrote:
> Hi.
> 
> I've made a strange observation.
> When having Dovecot (at least) with maildir and moving (via IMAP) mail
> received by some client (Evolution 3.4) into it the following happens:
> 
> Regardless of whether the mail was originally(!) set with CRLF or LF
> (i.e. when I use netcat to submit the plain SMTP to the relaying MTA).
> 
> When the client (Evolution) had received the mail via POP3 before moving
> it via IMAP into Dovecot... then the maildir file within dovecot is all
> LF.
mail_save_crlf setting controls if mails are saved as CRLF or LF to storage.
> When the client however received it via IMAP, before in turn moving it
> on via IMAP into Dovecot, then the maildir file is mixed CRLF and LF,
> i.e. the body is CRLF, the headers are terminated?
Sounds like the client was saving mails using CRCRLF, and Dovecot changed that
to CRLF. Perhaps Dovecot should have changed the other CR to some other
character.. You could verify by looking at the IMAP traffic, e.g.
http://wiki2.dovecot.org/Debugging/Rawlog