dovecot - Feb 2007 - Planning a dovecot deployment: issues and questions

Hi folks,

I am currently evaluating Dovecot for deployment to ~200 software
engineers (so, heavy email users); I'm hoping this list could provide
wisdom about how best to do the configuration, evaluation and deployment,
and what testing folks have done to determine whether dovecot is right for
their site.  Below I've listed the diffs between the dovecot-example.conf
and our config.

I've been playing with 1.0rc19 running on Solaris Nevada.  We have set it
up on our server on an alternate port from the existing IMAPd, which is
something ancient, horrible and problematic.  We're using sendmail and
mail.local as our MTA and MDA, respectively, as these ship with the OS,
and we need to test them.

I have some users who have volunteered to experiment with the server, and
they are starting to do so.  I've also seen the "flags" problems
others
have mentioned on the list, and I do not think I could deploy to
production with that bug affecting the users.

I also have some issues which I'm not sure have been previously reported:
One user (of my 4 test users) is occasionally seeing a bug in which mutt
1.4.2.1 claims to be "[downloading xxx/30724]" headers.  Upon getting
very
near to the 30724 total, the total rises to something like: "[downloading
xxx/50132]".  And on and on.  The actual number of messages in the box is
30724.  This is not very reproducible, and we do not yet know if it's a
problem with mutt or dovecot.  Another, using Apple's Mail.App reported:

  "Everything was working fine until i quit out of my mail app (Mac
  mail) and restarted.  Then i saw weird things:

    - messages wouldn't get marked unread
    - messages from eons ago started showing up in my Drafts folder
    - Inbox would keep spinning/loading
    - then eventually Inbox just hung showing me no messages"

Any thoughts about how to approach these issues?  I don't really have a
feel for how to go from user-observed-app-doing-X to imapd-is-doing-Y.

The big challenge for our site may be mailbox size: the median
mailbox size is 62M, the average is 176M and the largest is 1.1GB.
I'd be curious to know what sort of mailbox sizes other sites have
experienced, and how widespread the use of the Dovecot MDA is-- is this
likely to make a big difference for us, and will we have to migrate our
(approximately 25) procmail users to sieve?

Having read the recent discussions about "1.0", I think my team would
really appreciate knowing how to obtain something that was thoroughly
vetted and worked highly reliably.  If major changes are coming, I think
it'd be great to have a 1.0.x stability focused tree, and a 1.1.x (or
2.x.x) feature tree.

Thanks in advance for your comments,

        -dp

------- ------- ------- ------- ------- ------- ------- ------- -------
[note that IMAPS is off for now, but that's just temporary]

$ diff dovecot.conf dovecot-example.conf 
21d20
< protocols = imap
42,45d40
< protocol imap {
<       listen = *:150
< }
< 
50c45
< disable_plaintext_auth = no
---
> #disable_plaintext_auth = yes
89c84
< ssl_disable = yes
---
> #ssl_disable = no
214,215d208
< mail_location = mbox:~:INBOX=/var/mail/%u:INDEX=%h/Mail/imapd-indices
< 
258d250
< mail_extra_groups = mail
290d281
< dotlock_use_excl = yes
296d286
< lock_method = fcntl
441,444d430
< # dp: we'll see.
< mbox_read_locks = dotlock fcntl
< mbox_write_locks = dotlock fcntl
< 
------- ------- ------- ------- ------- ------- ------- ------- -------

--
Daniel Price - Solaris Kernel Engineering - dp at eng.sun.com - blogs.sun.com/dp

On 02/01/2007 04:44 PM, Dan Price wrote:
> Hi folks,
>
> $ diff dovecot.conf dovecot-example.conf 
>   
> < mail_location = mbox:~:INBOX=/var/mail/%u:INDEX=%h/Mail/imapd-indices
>   
That one does sound a bit fishy. Are the index files even compatible to 
each other?

Other than that, a little bit of info:
- Dovecot running on two different servers, providing IMAP and POP3 
(both with and without SSL)
- Storage format is maildir
- MTA is sendmail, MDA is a custom bit of software doing funky stuff 
like MySQL DB integration, autoresponders, spam/virusscan and so on and 
so forth
- LDA is dovecots delivery agent, plopping mails pre-sorted into various 
folders
- One server has about 50 users out of which 15 use IMAP, the rest uses 
POP3, the other server has about 10 IMAP-only users.
- Mail quota is set to 5GByte for both servers
- Clients use Outlook/Outlook Express/Thunderbird - the latter with 
check_quota plugin

Biggest single folder size is around 1.5GByte, no idea how many mails. 
Average access time to that folder from Thunderbird is about 15 seconds 
or so for initial loading, pretty much instant when opening any mail in 
there. Indexing it from scratch takes quite a bit longer though, about 7 
minutes.

Hope that helps
Jen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3237 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20070201/7abe599a/attachment-0002.bin>

On Thu, 2007-02-01 at 15:44 -0800, Dan Price wrote:
> I have some users who have volunteered to experiment with the server, and
> they are starting to do so.  I've also seen the "flags"
problems others
> have mentioned on the list, and I do not think I could deploy to
> production with that bug affecting the users.
I'll release rc20 in a few hours, and it's fixed there.
> I also have some issues which I'm not sure have been previously
reported:
> One user (of my 4 test users) is occasionally seeing a bug in which mutt
> 1.4.2.1 claims to be "[downloading xxx/30724]" headers.  Upon
getting very
> near to the 30724 total, the total rises to something like:
"[downloading
> xxx/50132]".
Never heard of this before..
>   "Everything was working fine until i quit out of my mail app (Mac
>   mail) and restarted.  Then i saw weird things:
> 
>     - messages wouldn't get marked unread
>     - messages from eons ago started showing up in my Drafts folder
I think I once saw something similar to this when the mailbox's
UIDVALIDITY changed. Mail.app didn't understand that change, and instead
of fetching the new message with UID 1234, it looked up the old UID 1234
from its cache and showed it to me instead.

The reason for the UIDVALIDITY change then has probably something to do
with you still running UW-IMAP(?) also. Or was the user only using
Dovecot?
>     - Inbox would keep spinning/loading
>     - then eventually Inbox just hung showing me no messages"
Don't know about these. I've been using Mail.app fine..
> Any thoughts about how to approach these issues?  I don't really have a
> feel for how to go from user-observed-app-doing-X to imapd-is-doing-Y.
rawlog could be helpful in knowing what exactly is happening.
http://dovecot.org/bugreport.html
> The big challenge for our site may be mailbox size: the median
> mailbox size is 62M, the average is 176M and the largest is 1.1GB.
> I'd be curious to know what sort of mailbox sizes other sites have
> experienced, and how widespread the use of the Dovecot MDA is-- is this
> likely to make a big difference for us, and will we have to migrate our
> (approximately 25) procmail users to sieve?
mbox_very_dirty_syncs=yes would at least help if you're not going to use
Dovecot MDA.
> Having read the recent discussions about "1.0", I think my team
would
> really appreciate knowing how to obtain something that was thoroughly
> vetted and worked highly reliably.  If major changes are coming, I think
> it'd be great to have a 1.0.x stability focused tree, and a 1.1.x (or
> 2.x.x) feature tree.
There are major changes in CVS HEAD already. I've mostly been trying to
fix the last few bugs, but looks like I managed to break some things
with those fixes in rc19..
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20070202/2e1996b5/attachment.bin>

Quoting Dan Price <dp at eng.sun.com>:
> I am currently evaluating Dovecot for deployment to ~200 software
> engineers (so, heavy email users); I'm hoping this list could provide
> wisdom about how best to do the configuration, evaluation and deployment,
> and what testing folks have done to determine whether dovecot is right for
> their site.  Below I've listed the diffs between the
dovecot-example.conf
> and our config.
I would say it is a great IMAP/POP server for almost any site.
Best would be to wait for the release version of course, but most
(not all) of the RC releases are stable.
> Any thoughts about how to approach these issues?  I don't really have a
> feel for how to go from user-observed-app-doing-X to imapd-is-doing-Y.
Report them individually to the mailing list, and you will get instructions
on how to debug/repor them...
> The big challenge for our site may be mailbox size: the median
> mailbox size is 62M, the average is 176M and the largest is 1.1GB.
> I'd be curious to know what sort of mailbox sizes other sites have
> experienced, and how widespread the use of the Dovecot MDA is-- is this
> likely to make a big difference for us, and will we have to migrate our
> (approximately 25) procmail users to sieve?
We use mbox format and have no problems, though our largest mailboxe
is 600 MB, so I have no idea how this would work with a 1.1 GB mailbox.
I'd be sure you have lots of memory in the machine if you want to be
handling mailboxes that size.

We use sendmail for MTA, procmail for filters, dovecot for pop3/imap,
Horde/IMP for webmail, and have no problems.  You don't have to move
to maildir, or sieve, or anything else.  dovecot plays nice with almost
any software...

-- 
Eric Rostetter
The Department of Physics
The University of Texas at Austin

Go Longhorns!