dovecot - Jul 2008 - changing INTERNALDATE or similar

I have a question which others must have encountered. I switched from
uw-imap to dovecot-imap a few months ago and moved my fairly large mail
archive (~15 yrs) by simply mounting both accounts in a mail app and
simply copying my folders over. This worked well, and all my mail is in
order when I view it in evolution or thunderbird on any number of
machines. However, since our company uses MS Exchange Server, when I add
my dovecot imap account to outlook, everything is great except the
received dates - it looks like the date I made the transition, but in
all instances it's not simply the time stamp in the Maildir folder. I
googled around on this and have gathered that this date is coming
somewhere from within the dovecot index file. Is there a utility for
scanning that so I can ascertain where the date shown in Outlook is
coming from? Is there also a utility which would just reset the
INTERNALDATE (or whatever it is that Outlook is keying on) to the date
in the message on the "Date: ..." line?

This issue is not life threatening as I don't use Outlook much, but when
I do, it is confusing to not have the "Received" column be meaningful,
especially for sorting.

Thanks for any help - Karl

On Wed, 2008-07-16 at 16:56 -0700, Karl Rudnick wrote:
> I have a question which others must have encountered. I switched from
> uw-imap to dovecot-imap a few months ago and moved my fairly large mail
> archive (~15 yrs) by simply mounting both accounts in a mail app and
> simply copying my folders over. This worked well, and all my mail is in
> order when I view it in evolution or thunderbird on any number of
> machines. However, since our company uses MS Exchange Server, when I add
> my dovecot imap account to outlook, everything is great except the
> received dates - it looks like the date I made the transition, but in
> all instances it's not simply the time stamp in the Maildir folder. I
> googled around on this and have gathered that this date is coming
> somewhere from within the dovecot index file. 
INTERNALDATE comes from the maildir files' mtimes. So you can use
whatever utility/script you want to modify the mtime (I don't have any
suggestions). Then delete dovecot.index.cache file to make sure the old
value wasn't cached. You might also have to recreate the account in your
clients to make sure they haven't cached the old values.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20080717/b1326cd5/attachment-0002.bin>

Terrific - I'll just write a script to:

1. cd to all Maildir folders
2. remove dovecot.index* in each folder
3. For each mail file in each Mailder folder:
   a. grep/ask "Date:" 
   b. pipe the output of a into "touch --date" for that file

On my windows side, all the imap info is save in one pst file in
"C:\Documents and Settings\<user>\Local Settings\Application
Data\Microsoft\Outlook\Outlook*imap*.pst" and if I blow it away it gets
correctly rebuilt - I just tested it and the blown away index files get
rebuilt so that Outlook then has the correct Received date displayed.
Pretty easy to do, but surprised a script wasn't openly available - or
at least my googling skills weren't good enough. Since I have ~35,000
emails in about 450 folders, this needs to be scripted. I'll make it
available when I get it smoothed out.

Thanks again for the clarification.
- Karl

-----Original Message-----
From: Timo Sirainen [mailto:tss at iki.fi] 
Sent: Thursday, July 17, 2008 4:58 AM
To: Rudnick, Karl; Dovecot Mailing List
Subject: Re: [Dovecot] changing INTERNALDATE or similar

On Wed, 2008-07-16 at 16:56 -0700, Karl Rudnick wrote:
> I have a question which others must have encountered. I switched from
> uw-imap to dovecot-imap a few months ago and moved my fairly large
mail
> archive (~15 yrs) by simply mounting both accounts in a mail app and
> simply copying my folders over. This worked well, and all my mail is
in
> order when I view it in evolution or thunderbird on any number of
> machines. However, since our company uses MS Exchange Server, when I
add
> my dovecot imap account to outlook, everything is great except the
> received dates - it looks like the date I made the transition, but in
> all instances it's not simply the time stamp in the Maildir folder. I
> googled around on this and have gathered that this date is coming
> somewhere from within the dovecot index file. 
INTERNALDATE comes from the maildir files' mtimes. So you can use
whatever utility/script you want to modify the mtime (I don't have any
suggestions). Then delete dovecot.index.cache file to make sure the old
value wasn't cached. You might also have to recreate the account in your
clients to make sure they haven't cached the old values.

I successfully executed the script which "fixed" my mail store. I also
see now why there isn't a buttoned down solution out there. The basic
steps are simple as outlined below, but when dealing with ~35,000 emails
going back before 1993, there were just about a dozen that didn't adhere
to the "Date:" screening and subsequent capture for use by "touch
--date". Adding these exceptions to my script completely automated the
INTERNALDATE resetting, but I would not publish this publicly on the
internet. Nevertheless, I would share it with anyone who requests
through this mail group given the caveat that it is only guaranteed to
work for MY personal data store.
Thanks - Karl


-----Original Message-----
From: Rudnick, Karl 
Sent: Thursday, July 17, 2008 9:06 AM
To: dovecot at dovecot.org
Cc: Timo Sirainen
Subject: RE: [Dovecot] changing INTERNALDATE or similar

Terrific - I'll just write a script to:

1. create a list of all Maildir folders ending in "/cur"
2. remove dovecot.index* in each folder
3. For each mail file in each Mailder /cur folder:
   a. grep/sed "Date:" to grab first date/time stamp in email
   b. pipe the output of a. into "touch --date" for that file

On my windows side, all the imap info is saved in one pst file in
"C:\Documents and Settings\<user>\Local Settings\Application
Data\Microsoft\Outlook\Outlook*imap*.pst" and if I blow it away it gets
correctly rebuilt - I just tested it and the blown away index files get
rebuilt so that Outlook then has the correct Received date displayed.
Pretty easy to do, but surprised a script wasn't openly available - or
at least my googling skills weren't good enough. Since I have ~35,000
emails in about 450 folders, this needs to be scripted. I'll make it
available when I get it smoothed out.

Thanks again for the clarification.
- Karl

-----Original Message-----
From: Timo Sirainen [mailto:tss at iki.fi] 
Sent: Thursday, July 17, 2008 4:58 AM
To: Rudnick, Karl; Dovecot Mailing List
Subject: Re: [Dovecot] changing INTERNALDATE or similar

On Wed, 2008-07-16 at 16:56 -0700, Karl Rudnick wrote:
> I have a question which others must have encountered. I switched from
> uw-imap to dovecot-imap a few months ago and moved my fairly large
mail
> archive (~15 yrs) by simply mounting both accounts in a mail app and
> simply copying my folders over. This worked well, and all my mail is
in
> order when I view it in evolution or thunderbird on any number of
> machines. However, since our company uses MS Exchange Server, when I
add
> my dovecot imap account to outlook, everything is great except the
> received dates - it looks like the date I made the transition, but in
> all instances it's not simply the time stamp in the Maildir folder. I
> googled around on this and have gathered that this date is coming
> somewhere from within the dovecot index file. 
INTERNALDATE comes from the maildir files' mtimes. So you can use
whatever utility/script you want to modify the mtime (I don't have any
suggestions). Then delete dovecot.index.cache file to make sure the old
value wasn't cached. You might also have to recreate the account in your
clients to make sure they haven't cached the old values.

Looking at the IMAP4rev1 protocol (RFC 2060), we have the following
definition:

INTERNALDATE   The internal date of the message.
2.3.3.  Internal Date Message Attribute

   The internal date and time of the message on the server.  This is not
   the date and time in the [RFC-822] header, but rather a date and time
   which reflects when the message was received.  In the case of
   messages delivered via [SMTP], this SHOULD be the date and time of
   final delivery of the message as defined by [SMTP].  In the case of
   messages delivered by the IMAP4rev1 COPY command, this SHOULD be the
   internal date and time of the source message.  In the case of
   messages delivered by the IMAP4rev1 APPEND command, this SHOULD be
   the date and time as specified in the APPEND command description.
   All other cases are implementation defined.

How could any implementation of this protocol possibly use a file system 
time stamp to represent that important piece of meta-data,
no matter where the file lives? It seems totally reasonable that this is
what Outlook uses for the Received date (and I rarely defend Microsoft).

This seems like a real design flaw in the dovecot implementation. I am fairly
new to dovecot (and like many aspects of it over uw-imap), but having
to really be careful with my mail store's mtimes borders on the absurd.
I realize it is "implementation defined", but the intent of the
definition
surely does not refer to file system time stamps. Any chance this can
be reconsidered? Is this an actual dovecot issue or a more general Maildir
issue?

On Fri, 2008-07-18 at 02:00 -0700, dovecot-request at dovecot.org
wrote:
> ------------------------------
> 
> Message: 4
> Date: Thu, 17 Jul 2008 23:12:55 +0300
> From: Timo Sirainen <tss at iki.fi>
> Subject: Re: [Dovecot] changing INTERNALDATE or similar
> To: richs at whidbey.net
> Cc: Dovecot Mailing List <dovecot at dovecot.org>
> Message-ID: <1216325575.31765.349.camel at hurina>
> Content-Type: text/plain; charset="utf-8"
> 
> On Thu, 2008-07-17 at 13:04 -0700, richs at whidbey.net wrote:
> > I believe it's still true that any mail files that are IMAP
STORE'd
> > (uploaded)
> 
> You mean IMAP APPENDed.
> 
> > to Dovecot by a client will have a current mtime, rather than
> > the "Date" in the message.  That means might need to compare
mtime's
> to
> > "Date" headers for new files that appear too.
> 
> This is only because the client doesn't specify the date to APPEND
> command.
> 
> > Here we just patched Dovecot to treat "INTERNALDATE" the
same as the
> > "Date" header. Constantly verifying mtime's became too
much of a
> chore
> > for those clients that rely on it (although this technically is
> against
> > the RFC).
> 
> Would probably have been better to set the default INTERNALDATE based
> on
> the Date: header in APPEND command. Should be pretty easy to do with
> Maildir I think.