I'm working on bring up a new mail server to replace our current one. Our current mail server is running dovecot 1.1.16, with postfix using mbox format. User inboxes are stored locally on the mail server and all other mail folders in users home directory under mail which is NFS mounted on the mail server. For our new mail server I'm looking to switch to the Maildir format. Some years ago I remember reading that it was not recommended to run Maildir format over NFS. Now I'm looking at several posts that seem to indicate that Maildir should run fine over NFS. I'm a little concerned about running Maildir over NFS, especially from the howto conversion pages I read would move all messages over to the User ~/Maildir folder including the inbox. So having every single mail transaction going over NFS doesn't seem the smart thing to do. So question I have for the dovecot team, does running Maildir over NFS work well? Or would you recommend that all user mail folders be stored locally on the mail server when using Maildir? We have about 3400 users, doing about 30k mail deliveries daily. Some users have 10's of thousands of mail messages in hundreds of mail folders. Thanks... -- C. J. Keist Email: cj.keist at colostate.edu UNIX/Network Manager Phone: 970-491-0630 Engineering Network Services Fax: 970-491-5569 College of Engineering, CSU Ft. Collins, CO 80523-1301 All I want is a chance to prove 'Money can't buy happiness'
On 8/6/2010 12:31, CJ Keist wrote:> > So question I have for the dovecot team, does running Maildir over NFS > work well? Or would you recommend that all user mail folders be stored > locally on the mail server when using Maildir? > > We have about 3400 users, doing about 30k mail deliveries daily. Some > users have 10's of thousands of mail messages in hundreds of mail folders. >Maildir (and Dovecot in general) will work fine with NFS. http://wiki.dovecot.org/NFS ~Seth
Quoting CJ Keist <cj.keist at colostate.edu>:> I'm working on bring up a new mail server to replace our current > one. Our current mail server is running dovecot 1.1.16, with > postfix using mbox format. User inboxes are stored locally on the > mail server and all other mail folders in users home directory under > mail which is NFS mounted on the mail server. > > For our new mail server I'm looking to switch to the Maildir format. > Some years ago I remember reading that it was not recommended to > run Maildir format over NFS. Now I'm looking at several posts that > seem to indicate that Maildir should run fine over NFS. I'm a > little concerned about running Maildir over NFS, especially from the > howto conversion pages I read would move all messages over to the > User ~/Maildir folder including the inbox. So having every single > mail transaction going over NFS doesn't seem the smart thing to do. > > So question I have for the dovecot team, does running Maildir over > NFS work well? Or would you recommend that all user mail folders be > stored locally on the mail server when using Maildir? > > We have about 3400 users, doing about 30k mail deliveries daily. > Some users have 10's of thousands of mail messages in hundreds of > mail folders. >I'm happy with maildir on nfs (netapp), doing 3million receiving per day. Looking at the new methods for v2.0, still on the fence though with them.
On Fri, 2010-08-06 at 13:31 -0600, CJ Keist wrote:> > For our new mail server I'm looking to switch to the Maildir format. > Some years ago I remember reading that it was not recommended to run > Maildir format over NFS. Now I'm looking at several posts that seem to > indicate that Maildir should run fine over NFS. I'm a little concernedMaildir was designed with NFS in mind, it is, always has been, and always will be, perfectly suited to any mail based system. It was, is, and likely always will be, mbox that is dangerous to use over NFS.> about running Maildir over NFS, especially from the howto conversion > pages I read would move all messages over to the User ~/Maildir folder > including the inbox. So having every single mail transaction going over > NFS doesn't seem the smart thing to do. >Actually you will not notice any difference. How do you think all the big boys do it now :) Granted some opted for the SAN approach over NAS, but for mail, NAS is better way to go IMHO and plenty of large services, ISP, corporations, and universities etc, all use NAS. One thing I strongly suggest however, for added security, is to use an internal private RFC1918 based LAN for NFS using your second ethernet port.> So question I have for the dovecot team, does running Maildir over NFSFor years, safely and happily.> We have about 3400 users, doing about 30k mail deliveries daily. Some > users have 10's of thousands of mail messages in hundreds of mail folders. >Run systems where multiple front end MTA's each processing 1.1million messages a day, and that's accepted messages each, not counting rejected connections, along with the several pop3 and webmail servers, all talking to this same mail system, best to use a dedicated device like a NetApp filer as the NAS though, not a garden variety server, but, even so, so long as the hardware is good, I can't see how you could go wrong with even that. Oh, and I trust you are using a virtual users and not system users, might be a good time to at least "think" about that if your still using system users, since you can expand to multiple servers easily, in fact, with only 2 minutes effort. Cheers -------------- next part -------------- A non-text attachment was scrubbed... Name: stock_smiley-1.png Type: image/png Size: 873 bytes Desc: not available URL: <http://dovecot.org/pipermail/dovecot/attachments/20100807/ed72642a/attachment-0002.png>
On 6.8.2010, at 20.31, CJ Keist wrote:> So question I have for the dovecot team, does running Maildir over NFS work well? Or would you recommend that all user mail folders be stored locally on the mail server when using Maildir?As long as you have only a single Dovecot server accessing mails at the same time (that includes Dovecot LDA, but not non-Dovecot LDA), there won't be any reliability problems. For better performance you could then also put index files on local disk. If you have multiple servers accessing mails at the same time, I recommend v2.0 with director enabled.
Noel Butler put forth on 8/6/2010 4:29 PM:> Actually you will not notice any difference. How do you think all the > big boys do it now :) Granted some opted for the SAN approach over NAS, > but for mail, NAS is better way to go IMHO and plenty of large services, > ISP, corporations, and universities etc, all use NAS.The protocol overhead of the NFS stack is such that one way latency is in the 1-50 millisecond range, depending on specific implementations and server load. The one way latency of a fibre channel packet is in the sub 100 microsecond range and is fairly immune to system load. The performance of fibre channel is equal to local disk plus approximately one millisecond of additional effective head seek time due to switch latency, SAN array controller latency, and latency due to cable length. A filesystem block served out of SAN array controller cache returns to the kernel quicker than a block read from local disk that is not in cache because the former suffers no mechanical latency. Due to the complexity of the stack, NFS is far slower than either. Those who would recommend NFS/NAS over fibre channel SAN have no experience with fibre channel SANs. I'm no fan of iSCSI SANs due to the reliance on TCP/IP for transport, and the low performance due to stck processing. However, using the same ethernet switches for both, iSCSI SAN arrays will also outperform NFS/NAS boxen by a decent margin. Regarding the OP's case, given the low cost of new hardware, specifically locally attached RAID and the massive size and low cost of modern disks, I'd recommend storing user mail on the new mail host. It's faster and more cost effective than both NFS/SAN. Unless his current backup solution "requires" user mail dirs to be on that NFS server for nightly backup, local disk is definitely the way to go. Four 300GB 15k SAS drives on a good PCIe RAID card w/256-512MB cache in a RAID 10 configuration would yield ~350-400MB/s of real filesystem bandwidth, seek throughput equivalent to a 2 disk stripe--about 600 random seeks/s, 600GB of usable space, ability to sustain two simultaneous disk failures (assuming 1 failure per mirror pair), and cost effectiveness. -- Stan