We are running Dovecot 1.0.0 using mbox format (currently in the midst
of conversion from UW IMAP). We discovered today that the Dovecot LDA
is accessing the user's INBOX at delivery time! Not only do our users
rely on this access time being when they last accessed their e-mail,
we also rely on it to migrate "inactive" users to cheaper disk.
I consider this a major bug for us and wondered if there is some reason
why the LDA is doing this. I can understand that in the absence of
index files it may choose to build them from scratch and therefore need
to access the INBOX, but in normal cases I don't see that being necessary.
It seems to me there should be a few options for how the LDA operates:
1) Deliver to INBOX, do not create or update the index.
This would avoid accessing the INBOX.
2) Deliver to INBOX, update the index only if it is already current.
This would avoid accessing the INBOX.
(not sure if this is possible)
3) Deliver to INBOX, update/create the index as necessary to make it current.
This obviously may involve accessing the INBOX.
We would rather not go with option #1 -- what's the point of using
Dovecot LDA in that scenario??
Comments appreciated.
--
Steven F. Siirila Office: Lind Hall, Room 130B
Internet Services E-mail: sfs at umn.edu
Office of Information Technology Voice: (612) 626-0244
University of Minnesota Fax: (612) 626-7593
Steven F Siirila wrote:> We are running Dovecot 1.0.0 using mbox format (currently in the midst > of conversion from UW IMAP). We discovered today that the Dovecot LDA > is accessing the user's INBOX at delivery time!Umm, yeah, that's kind of the point. mbox format is one big file with all of the messages concatenated together. So if a new message is delivered, then the INBOX file has to be opened to append the new message. This isn't a bug by any stretch, but rather a different operational method than what you were expecting. I'm guessing that UW IMAP doesn't retrieve new messages until the user asks for them (e.g. out of /var/mail/something). If you are using deliver, it actually, you know, _delivers_ the message. I think that there is a way to have the same behavior as UW IMAP, but I don't know off the top of my head how to do that (since we use maildir instead). You may want to check the list archives first, rather than claiming dovecot is doing something wrong... John -- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Blvd Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5747
On Wed, 2007-06-13 at 17:52 -0500, Steven F Siirila wrote:> We are running Dovecot 1.0.0 using mbox format (currently in the midst > of conversion from UW IMAP). We discovered today that the Dovecot LDA > is accessing the user's INBOX at delivery time!Well, there are two things: 1) It always makes sure that the mbox file ends with "\n". I guess this isn't all that important, but I'm not really interested in just removing the code. 2) If index files are fully synced, Dovecot writes X-UID: header. It also updates nextuid field in X-IMAP: / X-IMAPbase: header of the first message, which causes Dovecot to read() the file. The nextuid update isn't really required, but I think some other bug shows up if it isn't done. It would be possible to check the atime before any reading is done and then later after message is saved it could be updated back. Hmm. Well, attached a patch that seems to work. I'm not sure if I want to add it to v1.0.1. -------------- next part -------------- A non-text attachment was scrubbed... Name: mbox-atime-preserve.diff Type: text/x-patch Size: 3636 bytes Desc: not available URL: <http://dovecot.org/pipermail/dovecot/attachments/20070614/9495cd01/attachment-0004.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: <http://dovecot.org/pipermail/dovecot/attachments/20070614/9495cd01/attachment-0005.bin>
> We are running Dovecot 1.0.0 using mbox format (currently in the midst > of conversion from UW IMAP). We discovered today that the Dovecot LDA > is accessing the user's INBOX at delivery time! Not only do our usersIf you're willing to pass on delivery-time indexing, and you assign one unix uid per account, then you can instead consider using other delivery agents - and just use dovecot to serve. I use procmail on my system; and on some mailboxes, I use deliver; some I deliver direct via procmail to reduce deliver times to accomodate the flow rates that hit those specific mailboxes (alerts in a very large production environment that are still smtp based).
On Thu, Jun 14, 2007 at 06:22:45AM +0300, Timo Sirainen wrote:> On Wed, 2007-06-13 at 17:52 -0500, Steven F Siirila wrote: > > We are running Dovecot 1.0.0 using mbox format (currently in the midst > > of conversion from UW IMAP). We discovered today that the Dovecot LDA > > is accessing the user's INBOX at delivery time! > > Well, there are two things: > > 1) It always makes sure that the mbox file ends with "\n". I guess this > isn't all that important, but I'm not really interested in just removing > the code.If that were the only reason for opening in read mode I'd lobby for this to be changed. :)> 2) If index files are fully synced, Dovecot writes X-UID: header. It > also updates nextuid field in X-IMAP: / X-IMAPbase: header of the first > message, which causes Dovecot to read() the file. The nextuid update > isn't really required, but I think some other bug shows up if it isn't > done.Do these operations apply to Deliver or just IMAP/POP? Not knowing enough about how the indexes work.... Question: Is it possible for Deliver to append a message w/o writing an X-UID header, leaving that operation to the IMAP/POP client code, and still maintain an updated index? If not, are there any options I can provide to Dovecot LDA to make it function without index file updating?> It would be possible to check the atime before any reading is done and > then later after message is saved it could be updated back. Hmm. Well, > attached a patch that seems to work. I'm not sure if I want to add it to > v1.0.1.Hmm. I wonder how much of a timing window is left; this could be a viable option for us. However, I would like to consider our future options for mailbox formats other than mbox (we will eventually migrate I'm sure). What we have is a daemon called "mailattrd" on our mail servers which runs on a TCP port and simply does a stat() on a user's mailbox and returns the atime, mtime, ctime values of said mailbox. That "service" is used by a variety of our applications. We were thinking of another possibility: changing Dovecot IMAP/POP to update the last access time (open in READ) of a specific file other than the user's INBOX. However, we'd rather not maintain local modifications unless absolutely necessary. Any thoughts on alternatives which might be mailbox-format-independent? Thanks Timo for your super and timely support -- it is greatly appreciated!! -- Steven F. Siirila Office: Lind Hall, Room 130B Internet Services E-mail: sfs at umn.edu Office of Information Technology Voice: (612) 626-0244 University of Minnesota Fax: (612) 626-7593