Hi all, I plan to run a dovecot IMAPS and POPS service on our network. We handle about 3 000 mailboxes. I thought first buying a topnotch server (8 cores and 16 Go RAM) with equalogic iSCSI SAN SAS 15K for storage backend. On second though (and after a comprhensive read of dovecot features), I saw in http://wiki.dovecot.org/MailLocation that index files can be created on a separete local filesystem. My question is : for 3000 users, is it possible to have only a SATA backend attached to my topnotch server (to handle bigger mail quotas) by storing index on local hard drives (SAS drives) ? Extra question, what is the better : iSCSI SATA backend or NFS share ? NFS share is more convenient to have a failover server. Thanks for your help. Regards, -- Nicolas Gren?che - Orl?ans University - France http://blog.garnett.fr (in french)
Quoting Nicolas GRENECHE <nicolas.greneche at gmail.com>:> I plan to run a dovecot IMAPS and POPS service on our network. We > handle about 3 000 mailboxes. I thought first buying a topnotch server > (8 cores and 16 Go RAM) with equalogic iSCSI SAN SAS 15K for storage > backend.Sounds like overkill to me, but if you have the money go for it. :) I run mine on an 8 core (dual quad core) system with 4G RAM, using SATA (would have preferred SAS, but cost was an issue for us).> created on a separete local filesystem. My question is : for 3000 > users, is it possible to have only a SATA backend attached to my > topnotch server (to handle bigger mail quotas) by storing index on > local hard drives (SAS drives) ?Sure. If you only have one dovecot server without any failover this is fine. If you have multiple (active or passive) servers then more care is required, and you need to decide on the level of risk you want to take.> Extra question, what is the better : iSCSI SATA backend or NFS share ?iSCSI would be better than NFS IMHO.> NFS share is more convenient to have a failover server.If you introduce NFS and/or a failover server, your local index question gets much more complex... Is that a design requirement, desire, or future option?> Thanks for your help. > > Regards, > > -- > Nicolas Gren?che - Orl?ans University - France > http://blog.garnett.fr (in french)-- Eric Rostetter The Department of Physics The University of Texas at Austin This message is provided "AS IS" without warranty of any kind, either expressed or implied. Use this message at your own risk.
2009/11/16 Eric Jon Rostetter <eric.rostetter at physics.utexas.edu>:> Quoting Nicolas GRENECHE <nicolas.greneche at gmail.com>: > >> I plan to run a dovecot IMAPS and POPS service on our network. We >> handle about 3 000 mailboxes. I thought first buying a topnotch server >> (8 cores and 16 Go RAM) with equalogic iSCSI SAN SAS 15K for storage >> backend. > > Sounds like overkill to me, but if you have the money go for it. :) >I prefer disk space to useless storage velocity ;-)> I run mine on an 8 core (dual quad core) system with 4G RAM, using > SATA (would have preferred SAS, but cost was an issue for us). > >> created on a separete local filesystem. My question is : for 3000 >> users, is it possible to have only a SATA backend attached to my >> topnotch server (to handle bigger mail quotas) by storing index on >> local hard drives (SAS drives) ? > > Sure. ?If you only have one dovecot server without any failover this > is fine. ?If you have multiple (active or passive) servers then more care > is required, and you need to decide on the level of risk you want to take. > >> Extra question, what is the better : iSCSI SATA backend or NFS share ? > > iSCSI would be better than NFS IMHO. > >> NFS share is more convenient to have a failover server. > > If you introduce NFS and/or a failover server, your local index question > gets much more complex... >I know it, index may be handled on NFS server on fast local hardrives. The whole stuff should be exported to dovecot server.> Is that a design requirement, desire, or future option? >It should be a future option, but index management will be more tricky as you stated.>> Thanks for your help. >> >> Regards, >> >> -- >> Nicolas Gren?che - Orl?ans University - France >> http://blog.garnett.fr (in french) > > > -- > Eric Rostetter > The Department of Physics > The University of Texas at Austin > > This message is provided "AS IS" without warranty of any kind, > either expressed or implied. ?Use this message at your own risk. >
El Lunes 16 Noviembre 2009 a las 14:00, Nicolas GRENECHE escribi?:> Extra question, what is the better : iSCSI SATA backend or NFS > share ? NFS share is more convenient to have a failover server. >In my experience, NFS for maildirs is fast enough, and well known and supported. For indeces, in my benchmarks iSCSI is 3 times faster than nfs, and almost as fast as local disk. But I guess this depends a lot on your exact setup. -- Joseba Torre. Vicegerencia de TICs, ?rea de Explotaci?n
> The "generally don't see any bugs/issues" is the part I'm worried about > (generally isn't comforting).I was covering my **** on that one. Sods law dictates as soon as I say 'never' we'll discover an issue.> If you use a more traditional filesystem like ext2/ext3/ufs/etc then yes. > But you can use a cluster filesystem to get around this, and run > active-active.I've spent a week looking at the likes of PVFS, GFS, Lustre and a whole host of different systems, including pNFS (NFS 4.1) At the risk of diverting the thread away from the SATA backend, is there any recommendation for a fault tolerant file service. I'm really looking for 3 or 4 boxes to store data/metadata to support 10 Apache and Dovecot servers. The things I don't like are having a single metadata server be a single point of failure. Regards John
> > Extra question, what is the better : iSCSI SATA backend or NFS share ? > NFS share is more convenient to have a failover server.Everyone wants to use nfs for mailstorage, because it is convenient, but nfs is the wrong storage model for mail. NFS shines on big files but metadata performance is low, too much overhead! We currently have a mailcluster with 10000 mailboxes and the data ist stored on a netapp storage. If you have a Mailbox with more then 1000 mails, it gets slow over nfs. Our new architecuture will look like this: No NFS, no iSCSI we use DAS. There will be multiple dedicated mailbox server in pairs with a small postfix instance and dovecot. These Servers will only be used for delivering/IMAP/POP3. To get HA we use DRBD und Heartbeat in active/passive setup. In the front there will be a IMAP/POP3 proxy to direct the user to his mailbox server. This setup scales and is cheap, you only have to find a way to distribute the mailboxes around all mailbox servers. I think rackspace email uses a similar setup with drbd and dovecot.
alex handle schrieb:>> Extra question, what is the better : iSCSI SATA backend or NFS share ? >> NFS share is more convenient to have a failover server. > > Everyone wants to use nfs for mailstorage, because it is convenient, > but nfs is the wrong storage model for mail. > NFS shines on big files but metadata performance is low, too much overhead! > > We currently have a mailcluster with 10000 mailboxes and the data ist > stored on a netapp storage. > If you have a Mailbox with more then 1000 mails, it gets slow over nfs. > > Our new architecuture will look like this: > > No NFS, no iSCSI we use DAS. > There will be multiple dedicated mailbox server in pairs with a small > postfix instance and dovecot. > These Servers will only be used for delivering/IMAP/POP3. > To get HA we use DRBD und Heartbeat in active/passive setup. > In the front there will be a IMAP/POP3 proxy to direct the user to his > mailbox server. > > This setup scales and is cheap, you only have to find a way to > distribute the mailboxes around all mailbox servers. > > I think rackspace email uses a similar setup with drbd and dovecot.sorry for the stupid question what is "DAS" do you have a link etc for it, to get more info -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Robert Schetterer <robert at schetterer.org> writes:> sorry for the stupid question what is "DAS" > do you have a link etc for it, to get more infoSee http://en.wikipedia.org/wiki/Direct-attached_storage -- Nicolas
On Fri, Nov 20, 2009 at 12:28 PM, Nicolas KOWALSKI <niko at petole.demisel.net> wrote:> Robert Schetterer <robert at schetterer.org> writes: > >> sorry for the stupid question what is "DAS" >> do you have a link etc for it, to get more info > > See http://en.wikipedia.org/wiki/Direct-attached_storage >to put it simply: local storage We use Dell R710 in pair, each with 6 15K SAS Disks and RAID 10 - iSCSI or NFS can hardly be faster :)
On 11/16/2009 8:00 AM, Nicolas GRENECHE wrote:> Hi all, > > I plan to run a dovecot IMAPS and POPS service on our network. We > handle about 3 000 mailboxes. I thought first buying a topnotch server > (8 cores and 16 Go RAM) with equalogic iSCSI SAN SAS 15K for storage > backend.We run about 300 mailboxes, ~1 to 1.5 million inbound connections per month and about 1-1.5 million messages delivered to Dovecot per month on a low-end server. The O/S drive is 10k RPM SATA and the MailDir folders are stored on a 4-disk RAID-10 7200rpm SATA. It's a dual-core, ~2GHz, 64bit CentOS 5 server with only 4GB RAM. We also do anti-virus and spam filtering (SpamAssassin) scoring on that system along with using it for a few other tasks. The system chugs at times (during the daily backup window) but otherwise I'd say we're at about 25-30% load currently. Fortunately, providing basic mail service isn't that system intensive. I think your sizing issue is going to be more about how many messages per month you're pushing through the system then sheer number of mailboxes. And maybe the overall size of the mail store. (Heck, our old mail server was a Solaris 200MHz x86 box with 256MB RAM and a pair of 80GB IDE drives setup before I took over administration of the mail system.) Our current server was a test case that we put in about 18 months ago. Hopefully next year we can upgrade to better equipment (more cores, more memory, and more and faster spindles). And maybe some HA stuff like DRBD and Heartbeat.