Hello list, I built an email system using a proxy / director pair (IMAP, POP3, LMTP) and a backend pair. To have an HA system, I would like to understand if it is better to use an NFS export or replication to save emails and index files NFS is provided by a NAS (in HA), while for replication I would use the local backend disks Which of the two systems is more reliable? Are there any drawbacks for one or the other? Thanks, Andrea -- __________________________ Buy a Pentium 586/200 so you can reboot faster. __________________________ TIM San Marino S.p.A. Andrea Gabellini Engineering R&D TIM San Marino S.p.A. - https://www.telecomitalia.sm Via Ventotto Luglio, 212 - Piano -2 47893 - Borgo Maggiore - Republic of San Marino Tel: (+378) 0549 886237 Fax: (+378) 0549 886188 -- Informativa Privacy Questa email ha per destinatari dei contatti presenti negli archivi di TIM San Marino S.p.A.. Tutte le informazioni vengono trattate e tutelate nel rispetto della normativa vigente sulla protezione dei dati personali (Reg. EU 2016/679). Per richiedere informazioni e/o variazioni e/o la cancellazione dei vostri dati presenti nei nostri archivi potete inviare una email a privacy at telecomitalia.sm. Avviso di Riservatezza Il contenuto di questa e-mail e degli eventuali allegati e' strettamente confidenziale e destinato alla/e persona/e a cui e' indirizzato. Se avete ricevuto per errore questa e-mail, vi preghiamo di segnalarcelo immediatamente e di cancellarla dal vostro computer. E' fatto divieto di copiare e divulgare il contenuto di questa e-mail. Ogni utilizzo abusivo delle informazioni qui contenute da parte di persone terze o comunque non indicate nella presente e-mail potra' essere perseguito ai sensi di legge.
> On Jul 15, 2020, at 12:33 PM, Andrea Gabellini <andrea.gabellini at telecomitalia.sm> wrote: > > Hello list, > > I built an email system using a proxy / director pair (IMAP, POP3, LMTP) > and a backend pair. > > To have an HA system, I would like to understand if it is better to use > an NFS export or replication to save emails and index files > > NFS is provided by a NAS (in HA), while for replication I would use the > local backend disks > > Which of the two systems is more reliable? Are there any drawbacks for > one or the other? >The biggest problem with using NFS is that you?re using NFS and bringing along all the baggage that comes with it. Writes over the network are inherently slower than writes to local storage, plus locking gets interesting, to say the least. I posted a while back about using something similar to Joyent's manatee to bootstrap replication. (If IMAP replication works anything like databases, a system could join the cluster, get a base state by streaming a ZFS snapshot of an existing peer to the new peer, and from there it catches up via the normal replication mechanisms.) I don?t know if that would be feasible, but it?s certainly something I might try to make work. I also don?t know whether it gets more dicey in a multiple-primary situation.) But long and short of it. Avoid NFS if you can. The last time I used NFS for mail was last century, and even with everybody using native *nix MUAs like pine and elm, we could run into fun locking issues. -- Coy Hile coy.hile at coyhile.com
On 2020-07-15 17:33, Andrea Gabellini wrote:> Hello list, > > I built an email system using a proxy / director pair (IMAP, POP3, > LMTP) > and a backend pair. > > To have an HA system, I would like to understand if it is better to use > an NFS export or replication to save emails and index files > > NFS is provided by a NAS (in HA), while for replication I would use the > local backend disks > > Which of the two systems is more reliable? Are there any drawbacks for > one or the other?Another option to consider is DRBD replication of the disks at the block level. Despite what you might expect, performance and latency is quite good. A number of years ago I ran such a setup hosting a high traffic MySQL database, and it worked well. The disks where the limiting factor, not the network. In my case the two servers where directly connected by one cable without using a switch or suchlike. One thing to be aware of with DRBD is that the slave disk is not accessible at all until you trigger a fail-over, so you can't use it for read traffic. -- David Pottage
> I built an email system using a proxy / director pair (IMAP, POP3, LMTP) > and a backend pair. > > To have an HA system, I would like to understand if it is better to use > an NFS export or replication to save emails and index files > > NFS is provided by a NAS (in HA), while for replication I would use the > local backend disks > > Which of the two systems is more reliable? Are there any drawbacks for > one or the other?This decision is more about how many users you have in total and how you can partition them. A) 200 domains with 10 IMAP accounts each For high availability two dovecot servers with replication are sufficient, no director/nfs needed. Return both server ips via dns for imap.domain.com and you get active/active load balancing for free. There is no shared storage which means no locking problems. Dovecot can use optimizations like mmap which is not possible with nfs. B) 200000 IMAP accounts, all within the same domain You cannot partition by domain and a single server cannot handle the load. Here imap.domain.com could return e.g. 5 ips via DNS that point to your directors. The director's job is to send all connections of one particular user to the same backend, i.e. Outlook at work, Thunderbird at home and K9 Mail on a mobile phone could be active at the same time, but all are directed to the same backend server. This way locking issues with nfs are avoided because only one server is accessing the mailbox at a time. IIRC you need to monitor your backend servers and add/remove them on failure. If the nfs mount is not available on the backend, dovecot may create a new (empty) mailbox, which could break things. You need to set permissions in a way that cannot happen. C) like B) but with a static proxy mapping where users are assigned to a certain backend server by configuration, that could be replicated like A) without nfs. While A) in principle has a higher performance due to local disks and optimizations B) can have a higher overall performance as dedicated storage appliances usually have a lot more disks (ssd caching, ...) and 10G+ networking. C) avoids nfs but may introduce more complexity when software like pacemaker is used to provide failover. See https://wiki2.dovecot.org/Director and https://wiki2.dovecot.org/NFS Best regards Gerald
Thank you all for replies!!! Some missing infos... - As load balancer I'm using a pair of keepalived with simple setup and not the DNS - Load balancer algorithm is "Weighted Least-Connection" - About 20 domains and 3000 email - I'm monitoring my backend servers with poolmon - The backend servers are virtual machine (vmware) with datastore on "all flash" storage based on yours notes, I think the better choice is Replication. Correct? Thanks, Andrea Il 16/07/20 01:43, Gerald Galster ha scritto:>> I built an email system using a proxy / director pair (IMAP, POP3, LMTP) >> and a backend pair. >> >> To have an HA system, I would like to understand if it is better to use >> an NFS export or replication to save emails and index files >> >> NFS is provided by a NAS (in HA), while for replication I would use the >> local backend disks >> >> Which of the two systems is more reliable? Are there any drawbacks for >> one or the other? > This decision is more about how many users you have in total and how you > can partition them. > > A) 200 domains with 10 IMAP accounts each > > For high availability two dovecot servers with replication are sufficient, > no director/nfs needed. Return both server ips via dns for imap.domain.com > and you get active/active load balancing for free. > > There is no shared storage which means no locking problems. > Dovecot can use optimizations like mmap which is not possible with nfs. > > > B) 200000 IMAP accounts, all within the same domain > > You cannot partition by domain and a single server cannot handle the load. > > Here imap.domain.com could return e.g. 5 ips via DNS that point to your directors. > The director's job is to send all connections of one particular user to the > same backend, i.e. Outlook at work, Thunderbird at home and K9 Mail on a > mobile phone could be active at the same time, but all are directed to the > same backend server. This way locking issues with nfs are avoided because > only one server is accessing the mailbox at a time. > > IIRC you need to monitor your backend servers and add/remove them on failure. > > If the nfs mount is not available on the backend, dovecot may create > a new (empty) mailbox, which could break things. You need to set permissions > in a way that cannot happen. > > > C) like B) but with a static proxy mapping where users are assigned to a > certain backend server by configuration, that could be replicated like A) > without nfs. > > > While A) in principle has a higher performance due to local disks and > optimizations B) can have a higher overall performance as dedicated > storage appliances usually have a lot more disks (ssd caching, ...) > and 10G+ networking. > > C) avoids nfs but may introduce more complexity when software like pacemaker > is used to provide failover. > > See https://wiki2.dovecot.org/Director and https://wiki2.dovecot.org/NFS > > > Best regards > Gerald > > > >-- __________________________ One person's error is another person's data. __________________________ TIM San Marino S.p.A. Andrea Gabellini Engineering R&D TIM San Marino S.p.A. - https://www.telecomitalia.sm Via Ventotto Luglio, 212 - Piano -2 47893 - Borgo Maggiore - Republic of San Marino Tel: (+378) 0549 886237 Fax: (+378) 0549 886188 -- Informativa Privacy Questa email ha per destinatari dei contatti presenti negli archivi di TIM San Marino S.p.A.. Tutte le informazioni vengono trattate e tutelate nel rispetto della normativa vigente sulla protezione dei dati personali (Reg. EU 2016/679). Per richiedere informazioni e/o variazioni e/o la cancellazione dei vostri dati presenti nei nostri archivi potete inviare una email a privacy at telecomitalia.sm. Avviso di Riservatezza Il contenuto di questa e-mail e degli eventuali allegati e' strettamente confidenziale e destinato alla/e persona/e a cui e' indirizzato. Se avete ricevuto per errore questa e-mail, vi preghiamo di segnalarcelo immediatamente e di cancellarla dal vostro computer. E' fatto divieto di copiare e divulgare il contenuto di questa e-mail. Ogni utilizzo abusivo delle informazioni qui contenute da parte di persone terze o comunque non indicate nella presente e-mail potra' essere perseguito ai sensi di legge.