dovecot - Jan 2008 - Embedded From_ lines breaking Content-Length (and Dovecot)

I feel like I'm going totally crazy.  Is it just me, or have embedded  
From_ lines really been breaking mbox messages since (at least)  
dovecot 1.0?  I found a whole lot of broken messages in an old mailbox  
of mine and when I looked closer at it, it seemed like Dovecot was  
ignoring the Content-Length header and truncating the messages at the  
first embedded From_ line instead.  Figuring that this probably just  
was a fluke / old bug, I erased the index and upgraded Dovecot, first  
to 1.0.10, then to 1.1.beta14, but I'm still seeing the bug -- and  
it's even gotten worse!  Earlier versions would just quietly split the  
message at the From_ line, but 1.1.beta14 is more diligent and will  
complain with:
> dovecot[44795]: IMAP(lennart): FETCH for mailbox Drafts UID 196 got  
> too little data: 590 vs 719
before dropping the connection and exiting... which cause Thunderbird  
to reopen the connection and reissue the same request... which will  
cause Dovecot to complain and exit again... etc ad nauseam...

It's trivial to reproduce too -- just mail yourself a message with a  
valid From_ line in it (assuming that your delivery system isn't doing  
 >From-escaping), or put it in a draft plain text message and save  
it.  Then go and look in your Drafts folder...

(Like this one: "From someone Tue Jan 29 03:15:52 2008")

Just be sure that you can stop your spinning mail client and/or server  
if you're using 1.1.beta14...

---

OK, I am definitely not very familiar with the Dovecot source, but I  
spent some time trying to track this down today and it looks to me  
like i_stream_raw_mbox_read is being asked to find the end of the  
message (at the From_ line) before mbox_sync_parse_next_mail has had a  
chance to parse the Content-Length header and find out how big it is.   
Maybe someone who knows the code better can take a look at it and tell  
me if I'm barking up the totally wrong tree...

Thanks,
--Lennart

On Tue, Jan 29, 2008 at 04:09:26AM +0100, Lennart Lvstrand wrote:
> I feel like I'm going totally crazy.  Is it just me, or have embedded  
> From_ lines really been breaking mbox messages since (at least)  
> dovecot 1.0?
> 
> It's trivial to reproduce too -- just mail yourself a message with a  
> valid From_ line in it (assuming that your delivery system isn't doing
> >From-escaping), or put it in a draft plain text message and save  
> it.  Then go and look in your Drafts folder...
The "mbox" format, by definition, uses From_ lines as *the* separator.
If it uses anything else, it's not conventional mbox format.  There
are variants of mbox, sometimes described as mboxcl2 that use
Content-Length: as the defining separator, but that is *not*
conventional mbox format.

I'm sure others will clarify Dovecot's stance on this, but relying on
Content-Length: headers as the sole source of determining message
separation is VERY risky business unless you make absolutely sure that
the values given are 100% correct and that all software touching the
mailbox are also in agreement (i.e. POP daemons, UNIX readers, procmail,
other IMAP daemons, etc.)

Because Dovecot cannot control existing values of Content-Length
headers, that seems to be an extremely risky proposition.

--
Dean Brooks
dean at iglou.com

On Tue, 2008-01-29 at 04:09 +0100, Lennart L?vstrand
wrote:
> I feel like I'm going totally crazy.  Is it just me, or have embedded  
> From_ lines really been breaking mbox messages since (at least)  
> dovecot 1.0?
Finally fixed in v1.1:
http://hg.dovecot.org/dovecot-1.1/rev/7871b6219480

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20080504/fd2bccd1/attachment-0002.bin>