dovecot - Jul 2008 - Need suggestion regarding index file and Quota settings.

Dear All,

My system has about 30,000 accounts and these accounts are
created as unix system users. Each account has its own entry
in /etc/passwd with different UID and home directory.

We plan to deploy an Layer 4 switch as a load balancer. Proposed
architecure is as the following:

---- internet ---- Layer 4 switch ---- mail server1 ---- NFS storage
                                           |                           |
                                           |--- mail server2 -----|

Both mail servers will share the same storage which contains users'
home with Maildir inside. Dovecot will be installed on the mail servers
serving pop3 and imap4 protocols.

Since the layer 4 switch will direct connection to backend mail server
in random(or some of its algorithm), users can not guarantee which mail
server to use. My first question is which kind of INDEX settings should
I apply?
1) INDEX=MEMORY
2) INDEX=/nfs/%u
3) INDEX=/var/spool/%u

Will INDEX stored on disk outperform INDEX stored in memory?
If INDEX stored on NFS storage, we can be sure the consistency of INDEX
file,
but what about the performance and locking problem?
If INDEX stored on local disk, since users may connect to different mail
servers
every time. Will that impose the problem of inconsistency? What kind of
syndrome
will happen if the INDEX became inconsistent to real maildir content?

My second question is about quota. We use maildir as mail storage format.
Our users
were divided into 3-4 groups, each group of users with different quota size
ranging
from 100MB - 1G. To my best knowledge, dovecot's quota setting in config
file
supports only "one global configuration". (We are not using db, we use
only
/etc/passwd).
dovecot.conf
   quota_rule = *:storage=10MB
Since dovecot supports both fs and maildir++ quota, I would like to ask for
your suggestion:

1. Should I use system quota instead? How about the comparision of
performance?
   Is there any drawback/limitation using system quota?
2. Is there any settings of dovecot that could fullfil my situation?
3. My thought is to modify dovecot.conf, comment out quota_rule *:storage=10MB
and leave
   only the quota = maildir in the plugin { block. According to my
experiment, dovecot
   will read the maildirsize file under user's home. Then I have to set the
maildirsize
   file for every user individually! It seems to be a little crappy but
works.

Looking for your valuable suggestions!
Thanks very much.

Tim Chen

Off the wall, but have you also considered using Dovecot (or other 
software, eg nginx) proxy options?

I only have a small setup, but I have seen others use the idea that all 
backend servers are also frontend servers, but the proxy option then 
forwards the connection to the correct other machine (with only a few 
servers the probability they hit the correct server first time is also 
respectable).

You can also stick a single proxy only machine in front of 
everythingthing else to point connections to the correct backend 
straight off.  Dovecot is apparently respectable as a high performance 
frontend proxy - some other people like nginx (no personal experience)

I believe you will find instructions on using indexes on NFS on either 
the wiki or a couple of previous answers from Timo here.  I think 
basically that v1.1 is best for indexes on NFS?

Good luck

Ed W

On Fri, 2008-07-18 at 15:11 +0800, Tim Chen wrote:
> ---- internet ---- Layer 4 switch ---- mail server1 ---- NFS storage
>                                            |                           |
>                                            |--- mail server2 -----|
> 
> Both mail servers will share the same storage which contains users'
> home with Maildir inside. Dovecot will be installed on the mail servers
> serving pop3 and imap4 protocols.
> 
> Since the layer 4 switch will direct connection to backend mail server
> in random(or some of its algorithm), users can not guarantee which mail
> server to use. 
Would be best if you at least could redirect users from the same IP to
the same server.
> My first question is which kind of INDEX settings should
> I apply?
> 1) INDEX=MEMORY
> 2) INDEX=/nfs/%u
> 3) INDEX=/var/spool/%u
What OS do you use? What kind of IMAP clients are there going to be?
With webmail clients you'll definitely want to use index files. With
Outlook/TB type of clients it might not matter that much.

In any case be sure to use v1.1 and read http://wiki.dovecot.org/NFS
> Will INDEX stored on disk outperform INDEX stored in memory?
Yes, definitely, because it's only temporarily stored in memory.
> If INDEX stored on NFS storage, we can be sure the consistency of INDEX
> file,
> but what about the performance and locking problem?
If fcntl locking is buggy, you can fallback to dotlock files, although
their performance is worse.
> If INDEX stored on local disk, since users may connect to different mail
> servers
> every time. Will that impose the problem of inconsistency? What kind of
> syndrome
> will happen if the INDEX became inconsistent to real maildir content?
Dovecot just updates the index if it's out-of-date. Although I've heard
there have been some issues here with v1.0, but hopefully they're fixed
in v1.1.
> My second question is about quota. We use maildir as mail storage format.
> Our users
> were divided into 3-4 groups, each group of users with different quota size
> ranging
> from 100MB - 1G. To my best knowledge, dovecot's quota setting in
config
> file
> supports only "one global configuration". (We are not using db,
we use only
> /etc/passwd).
> dovecot.conf
>    quota_rule = *:storage=10MB
> Since dovecot supports both fs and maildir++ quota, I would like to ask for
> your suggestion:
> 
> 1. Should I use system quota instead? How about the comparision of
> performance?
>    Is there any drawback/limitation using system quota?
See http://wiki.dovecot.org/Quota/FS
> 2. Is there any settings of dovecot that could fullfil my situation?
How about generating a passwd-file from /etc/passwd and your own quota
configuration file, and then using:

userdb passwd-file {
  args = /etc/dovecot.passwd
}
userdb passwd {
}

The dovecot.passwd could contain only those users who have non-default
quota value. If user isn't found from there, it fallbacks to passwd.
> 3. My thought is to modify dovecot.conf, comment out quota_rule >
*:storage=10MB and leave
>    only the quota = maildir in the plugin { block. According to my
> experiment, dovecot
>    will read the maildirsize file under user's home. Then I have to set
the
> maildirsize
>    file for every user individually! It seems to be a little crappy but
> works.
That'd work too.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20080720/68d84248/attachment-0002.bin>