Jon Roma
2005-Aug-23 16:50 UTC
[Dovecot] Weird internaldate behavior with 1.0-test80 and mbox folder
Has anyone else encountered weird date behavior with mbox-style folders?
I am nearly ready to bite the bullet and convert my UW imap hierarchy
(which is in mbx format) to Dovecot. In so doing, I need to convert each
UW-specific mbx-format mailboxes to the more portable mbox format, which
Dovecot can read. (This is done with the 'mbxcvt' utility that's
part of
UW.)
I am then planning to copy those mbox-format folders into a staging area
in a directory I've defined to Dovecot as a mbox-format namespace. The
plan was then to use my normal GUI to reorganize my old data and copy it
into proper maildir folders into Dovecot's default namespace.
Since I'm naturally paranoid, I did a dry run of a couple folders first
and discovered a rather strange anomaly -- all the mail in the converted
folder is displayed by Dovecot with a timestamp exactly 200 minutes
(3 hours 20 minutes) into the future!
Investigating this via telneting to the respective IMAP ports shows that
UW imapd shows the correct timestamp both in the original mbx-format
folder and the converted mbox-format folder -- so it appears that
Dovecot is skewing the time.
This doesn't look like a timezone issue since 200 minutes ahead from
CST6CDT would be a pretty strange timezone -- not even Newfoundland
has this odd a TZ value! :)
(For the curious, Newfoundland is a Canadian province whose
time zone is TZ=NST3:30NDT, therefore being a half-hour "off"
from the rest of North America.)
I have distilled this into a degenerate case of a single folder with
a single message and can reproduce. As mentioned, I am running 1.0-test80.
My platform is AIX 5.1.
I created the test by sending a message to myself using
date | mail -s "Test at $(date)" roma at uiuc.edu
and using my MUA (Mulberry) to file it into the test folder, which
defaults to mbx format in UW's namespace. Here's what UW shows
when I select/fetch the "original" message:
* namespace
* NAMESPACE (("" "/")("#mhinbox"
NIL)("#mh/" "/")) (("~" "/"))
(("#shared/"
"/")("#ftp/" "/")("#news."
".")("#public/" "/"))
* OK NAMESPACE completed
* select "~/Test"
* 1 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1124811965] UID validity status
* OK [UIDNEXT 3] Predicted next UID
* FLAGS (\Answered \Flagged \Deleted \Draft \Seen)
* OK [PERMANENTFLAGS (\* \Answered \Flagged \Deleted \Draft \Seen)]
Permanent flags
* OK [READ-WRITE] SELECT completed
* fetch 1 internaldate
* 1 FETCH (INTERNALDATE "23-Aug-2005 11:16:37 -0500")
* OK FETCH completed
* fetch 1 envelope
* 1 FETCH (ENVELOPE ("Tue, 23 Aug 2005 11:16:30 -0500" "Test at
Tue Aug 23
11:16:29 CDT 2005" (("Jon Roma" NIL "roma"
"zippy.cso.uiuc.edu")) (("Jon
Roma" NIL "roma" "zippy.cso.uiuc.edu")) (("Jon
Roma" NIL "roma"
"zippy.cso.uiuc.edu")) ((NIL NIL "roma"
"uiuc.edu")) NIL NIL NIL
"<200508231616.j7NGGUrc022618 at zippy.cso.uiuc.edu>"))
* OK FETCH completed
Next I use the UW 'mbxcvt' to convert the folder, leaving it in mbox
format in a different folder (~/Test-mbox) -- still in UW's namespace.
Here's what the telnet connection shows:
* namespace
* NAMESPACE (("" "/")("#mhinbox"
NIL)("#mh/" "/")) (("~" "/"))
(("#shared/"
"/")("#ftp/" "/")("#news."
".")("#public/" "/"))
* OK NAMESPACE completed
* select "~/Test-mbox"
* 1 EXISTS
* 1 RECENT
* OK [UIDVALIDITY 1124813840] UID validity status
* OK [UIDNEXT 2] Predicted next UID
* FLAGS (\Answered \Flagged \Deleted \Draft \Seen)
* OK [PERMANENTFLAGS (\* \Answered \Flagged \Deleted \Draft \Seen)]
Permanent flags
* OK [READ-WRITE] SELECT completed
* fetch 1 internaldate
* 1 FETCH (INTERNALDATE "23-Aug-2005 11:16:37 -0500")
* OK FETCH completed
* fetch 1 envelope
* 1 FETCH (ENVELOPE ("Tue, 23 Aug 2005 11:16:30 -0500" "Test at
Tue Aug 23
11:16:29 CDT 2005" (("Jon Roma" NIL "roma"
"zippy.cso.uiuc.edu")) (("Jon
Roma" NIL "roma" "zippy.cso.uiuc.edu")) (("Jon
Roma" NIL "roma"
"zippy.cso.uiuc.edu")) ((NIL NIL "roma"
"uiuc.edu")) NIL NIL NIL
"<200508231616.j7NGGUrc022618 at zippy.cso.uiuc.edu>"))
* OK FETCH completed
Now, copying the same mbox-format folder into dovecot's namespace and
telneting to my Dovecot port gives the following:
* namespace
* NAMESPACE (("" ".")("Stage/" "/")) NIL
NIL
* OK Namespace completed.
* select "Stage/Test"
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags
permitted.
* 1 EXISTS
* 1 RECENT
* OK [UIDVALIDITY 1124815188] UIDs valid
* OK [UIDNEXT 2] Predicted next UID
* OK [READ-WRITE] Select completed.
* fetch 1 internaldate
* 1 FETCH (INTERNALDATE "23-Aug-2005 14:36:37 -0500")
* OK Fetch completed.
* fetch 1 envelope
* 1 FETCH (ENVELOPE ("Tue, 23 Aug 2005 11:16:30 -0500" "Test at
Tue Aug 23
11:16:29 CDT 2005" (("Jon Roma" NIL "roma"
"zippy.cso.uiuc.edu")) (("Jon
Roma" NIL "roma" "zippy.cso.uiuc.edu")) (("Jon
Roma" NIL "roma"
"zippy.cso.uiuc.edu")) ((NIL NIL "roma"
"uiuc.edu")) NIL NIL NIL
"<200508231616.j7NGGUrc022618 at zippy.cso.uiuc.edu>"))
* OK Fetch completed.
Notice the "internaldate" response. Dovecot reports the correct time
zone, but the internaldate value is 200 minutes ahead of what it should
be!!!
This is an interesting mystery; I seem to be living three hours and 20
minutes in the future.
I've attached the entire contents of the mbox-style mailbox that Dovecot
misinterprets.
Thanks in advance.
-------------- next part -------------->From MAILER-DAEMON Tue Aug 23 11:39:48 2005Date: 23 Aug 2005 11:39:48 -0500
From: Mail System Internal Data <MAILER-DAEMON at zippy.cso.uiuc.edu>
Subject: DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA
X-IMAP: 1124815188 0000000001
Status: RO
This text is part of the internal format of your mail folder, and is not
a real message. It is created automatically by the mail system software.
If deleted, important folder data will be lost, and it will be re-created
with the data reset to initial values.
>From roma at zippy.cso.uiuc.edu Tue Aug 23 11:16:37 2005 -0500
Status: RO
X-Status:
X-Keywords:
Received: via dmail-4.1(10) for +INBOX; Tue, 23 Aug 2005 11:16:37 -0500
(CDT)
Return-Path: <roma at zippy.cso.uiuc.edu>
Received: from zippy.cso.uiuc.edu (loopback [127.0.0.1])
by zippy.cso.uiuc.edu (8.12.10/8.12.10) with ESMTP id j7NGGUCQ027294
for <roma at uiuc.edu>; Tue, 23 Aug 2005 11:16:30 -0500
Received: (from roma at localhost)
by zippy.cso.uiuc.edu (8.12.10/8.12.10/Submit) id j7NGGUrc022618
for roma at uiuc.edu; Tue, 23 Aug 2005 11:16:30 -0500
Date: Tue, 23 Aug 2005 11:16:30 -0500
From: Jon Roma <roma at zippy.cso.uiuc.edu>
Message-Id: <200508231616.j7NGGUrc022618 at zippy.cso.uiuc.edu>
To: roma at uiuc.edu
Subject: Test at Tue Aug 23 11:16:29 CDT 2005
X-Spam-Checker-Version: SpamAssassin 3.0.1 (2004-10-22) on
zippy.cso.uiuc.edu
X-Spam-Level:
X-Spam-Status: No, score=-5.9 required=2.5 tests=ALL_TRUSTED,BAYES_00
autolearn=ham version=3.0.1
X-UID: 1
Tue Aug 23 11:16:29 CDT 2005
Timo Sirainen
2005-Aug-28 18:54 UTC
[Dovecot] Weird internaldate behavior with 1.0-test80 and mbox folder
On Tue, 2005-08-23 at 11:50 -0500, Jon Roma wrote:> Has anyone else encountered weird date behavior with mbox-style folders?Looks like timezone handling was completely broken in mboxes. I think most people haven't noticed it because most software nowadays don't seem to write timezones to the From-line. This patch should fix it? Index: src/lib-storage/index/mbox/mbox-from.c ==================================================================RCS file: /var/lib/cvs/dovecot/src/lib-storage/index/mbox/mbox-from.c,v retrieving revision 1.6 diff -u -r1.6 mbox-from.c --- src/lib-storage/index/mbox/mbox-from.c 19 Aug 2005 15:49:49 -0000 1.6 +++ src/lib-storage/index/mbox/mbox-from.c 28 Aug 2005 18:52:29 -0000 @@ -145,8 +145,8 @@ i_isdigit(msg[3]) && i_isdigit(msg[4]) && msg[5] == ' ') { /* numeric timezone, use it */ seen_timezone = TRUE; - timezone = (msg[1]-'0') * 1000 + (msg[2]-'0') * 100 + - (msg[3]-'0') * 10 +(msg[4]-'0'); + timezone = (msg[1]-'0') * 10*60*60 + (msg[2]-'0') * 60*60 + + (msg[3]-'0') * 10 + (msg[4]-'0'); if (msg[0] == '-') timezone = -timezone; msg += 6; } @@ -166,8 +166,8 @@ i_isdigit(msg[2]) && i_isdigit(msg[3]) && i_isdigit(msg[4]) && i_isdigit(msg[5])) { seen_timezone = TRUE; - timezone = (msg[2]-'0') * 1000 + (msg[3]-'0') * 100 + - (msg[4]-'0') * 10 +(msg[5]-'0'); + timezone = (msg[2]-'0') * 10*60*60 + (msg[3]-'0') * 60*60 + + (msg[4]-'0') * 10 + (msg[5]-'0'); if (msg[1] == '-') timezone = -timezone; } @@ -176,7 +176,7 @@ if (t == (time_t)-1) return -1; - t -= timezone * 60; + t -= timezone; *time_r = t; } else { /* assume local timezone */ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: <http://dovecot.org/pipermail/dovecot/attachments/20050828/e296876c/attachment.bin>
Jon Roma
2005-Sep-03 02:49 UTC
[Dovecot] Weird internaldate behavior with 1.0-test80 and mbox folder
Timo: Sorry for not writing faster. The patch you supplied (below) applied to 1.0-test80 and solves the weird date behavior. Thank you for your fast and efficient service! I'm already planning out my upgrade from UW to Dovecot. So far 1.0-test80 works fine for me, but once the patch gets applied, I'll probably grab one of the more recent versions. Thanks again. Timo Sirainen <tss at iki.fi> wrote:> On Tue, 2005-08-23 at 11:50 -0500, Jon Roma wrote: >> Has anyone else encountered weird date behavior with mbox-style folders? > > Looks like timezone handling was completely broken in mboxes. I think > most people haven't noticed it because most software nowadays don't seem > to write timezones to the From-line. > > This patch should fix it? > > Index: src/lib-storage/index/mbox/mbox-from.c > ==================================================================> RCS file: /var/lib/cvs/dovecot/src/lib-storage/index/mbox/mbox-from.c,v > retrieving revision 1.6 > diff -u -r1.6 mbox-from.c > --- src/lib-storage/index/mbox/mbox-from.c 19 Aug 2005 15:49:49 -0000 1.6 > +++ src/lib-storage/index/mbox/mbox-from.c 28 Aug 2005 18:52:29 -0000 > @@ -145,8 +145,8 @@ > i_isdigit(msg[3]) && i_isdigit(msg[4]) && msg[5] == ' ') { > /* numeric timezone, use it */ > seen_timezone = TRUE; > - timezone = (msg[1]-'0') * 1000 + (msg[2]-'0') * 100 + > - (msg[3]-'0') * 10 +(msg[4]-'0'); > + timezone = (msg[1]-'0') * 10*60*60 + (msg[2]-'0') * 60*60 + > + (msg[3]-'0') * 10 + (msg[4]-'0'); > if (msg[0] == '-') timezone = -timezone; > msg += 6; > } > @@ -166,8 +166,8 @@ > i_isdigit(msg[2]) && i_isdigit(msg[3]) && > i_isdigit(msg[4]) && i_isdigit(msg[5])) { > seen_timezone = TRUE; > - timezone = (msg[2]-'0') * 1000 + (msg[3]-'0') * 100 + > - (msg[4]-'0') * 10 +(msg[5]-'0'); > + timezone = (msg[2]-'0') * 10*60*60 + (msg[3]-'0') * 60*60 + > + (msg[4]-'0') * 10 + (msg[5]-'0'); > if (msg[1] == '-') timezone = -timezone; > } > > @@ -176,7 +176,7 @@ > if (t == (time_t)-1) > return -1; > > - t -= timezone * 60; > + t -= timezone; > *time_r = t; > } else { > /* assume local timezone */ >
Apparently Analagous Threads
- Misbehavior with Dovecot and Mulberry
- Test migration (IMAP copy) and INTERNALDATE?
- Problem with INTERNALDATE when the year in localtime and GMT different
- What INTERNALDATE does dovecot with mbox storage set on a COPY'd message?
- What INTERNALDATE does dovecot with mbox storage set on a COPY'd message?