Hello,
I have recently setup mailserver solution using 2-node master-master
setup (mainly based on MySQL M-M replication and GlusterFS with 2
replica volume) on Ubuntu 14.04 (Dovecot 2.2.9).
Unfortunately even with shared-storage-aware setting:
mail_nfs_index = yes
mail_nfs_storage = yes
mail_fsync = always
mmap_disable = yes
..I have hit strange issues pretty soon especially when user was
manipulating same mailbox from multiple devices at the same time.
Most issues was about corrupted indexes which was solved easily by just
putting them on local storage of each node:
mail_location = maildir:/srv/mail/%d/%u:INDEX=/var/lib/dovecot/index/%d/%u
But I still hit issues like this one:
dovecot: lmtp(6276, user at example.com): Error: Broken file
/srv/mail/example.com/user at example.com/dovecot-uidlist line 8529: UIDs not
ordered (8527 >= 8527)
Which I am not sure how serious it is or if it's possible to solve or
workaround?
Anyway because of the above and high possibility of GlusterFS
split-brains, I have decided to setup Dovecot Director according to the
docs [1] but I have a couple of questions:
- is custom monitoring still required? Poolmon [2] is 4 year old so I
would suppose there's some progress since that?
- it's not possible to have same backends and directors in Dovecot
<2.2.17. I can backport newer Dovecot for Ubuntu Trusty, so this is
not an issue, but..
- documentation states that it still doesn't work for LMTP [3]?
Which is probably important for my setup, because both Postfix servers
are using dovecot-lmtp for mail delivery so there can be still some
issues (but probably less frequent?) when both servers will deliver
new mails for one user at once.
So do I really have to split directors from backends?
Anyone has experience with clustered Dovecot setup?
Why is Dovecot behaving so bad when it pretends to be shared storage
friendly? Are these issues only specific for older Dovecot?
Or is there something wrong in my architecture design?
Thanks for any help,
Filip
---
[1] http://wiki2.dovecot.org/Director
[2] https://github.com/brandond/poolmon/
[3] "LMTP however doesn't currently support mixing recipients to both
being proxied and store locally."
---
BTW if someone is interested in SaltStack, here are Salt formulas for
Dovecot + Postfix + GlusterFS + Roundcube + Mailman setup which we are
using:
https://github.com/tcpcloud/salt-formula-dovecot
https://github.com/tcpcloud/salt-formula-postfix
https://github.com/tcpcloud/salt-formula-roundcube
https://github.com/tcpcloud/salt-formula-glusterfs
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20151205/9c72aa88/attachment-0001.sig>
Hi Filip, On 12/05/2015 10:42 AM, Filip Pytloun wrote:> I have recently setup mailserver solution using 2-node master-master > setup (mainly based on MySQL M-M replication and GlusterFS with 2 > replica volume) on Ubuntu 14.04 (Dovecot 2.2.9).that's no good idea due to different reasons - see below.> Anyway because of the above and high possibility of GlusterFS > split-brains, I have decided to setup Dovecot Director according to the > docs [1] but I have a couple of questions: > > - is custom monitoring still required? Poolmon [2] is 4 year old so I > would suppose there's some progress since that?Using the dovecot director, poolmon is strongly recommended - see the official dovecot wiki. The tool seems to be old, but the task is the same since a couple of years ;-)> - it's not possible to have same backends and directors in Dovecot > <2.2.17. I can backport newer Dovecot for Ubuntu Trusty, so this is > not an issue, but..You have to use 2 instances of dovecot when running the director and the backend on the same server. Otherwise, you have to use 2 systems.> - documentation states that it still doesn't work for LMTP [3]? > Which is probably important for my setup, because both Postfix servers > are using dovecot-lmtp for mail delivery so there can be still some > issues (but probably less frequent?) when both servers will deliver > new mails for one user at once. > So do I really have to split directors from backends?At the moment, I cannot recognize the requirement for using lmtp over the directors. When using postfix for delivering e-mails to the backend, do this directly with an corresponding MX record.> Why is Dovecot behaving so bad when it pretends to be shared storage > friendly? Are these issues only specific for older Dovecot? > Or is there something wrong in my architecture design?IMHO, this is not a problem of dovecot. In general, using cluster filesystems in such a setup is no good idea. You've already mentioned all the problems of such a setup. In particular, the performance of GlusterFS is absolutely not suitable for (bigger) mailserver cluster. Asking any search engine for "dovecot clustering" shows a lot of results for good cluster designs. In particular, a shared mailbox storage should be avoided. Here, the dovecot internal replication mechanism on the backends seems to be the best solution. The rest of the setup (e.g., clustered directors or smtp servers) is trivial. Due to the fact, that MUAs are able to use A records only and the behavior using round robin A records is catastrophically, a service IP address is recommended. Here, keepalived is your friend. HTH. Best regards, Gordon -- Technischer Leiter & stellv. Direktor Universit?tsrechenzentrum (URZ) E.-M.-Arndt-Universit?t Greifswald Felix-Hausdorff-Str. 12 17489 Greifswald Germany Tel. +49 3834 86 1456 Fax. +49 3834 86 1401
Il 05.12.2015 10:42 Filip Pytloun ha scritto:> Hello, > > I have recently setup mailserver solution using 2-node master-master > setup (mainly based on MySQL M-M replication and GlusterFS with 2 > replica volume) on Ubuntu 14.04 (Dovecot 2.2.9). > > Unfortunately even with shared-storage-aware setting: > > mail_nfs_index = yes > mail_nfs_storage = yes > mail_fsync = always > mmap_disable = yesWith only these setting you don't solve the problem of shared storage.> ..I have hit strange issues pretty soon especially when user was > manipulating same mailbox from multiple devices at the same time. > > Most issues was about corrupted indexes which was solved easily by just > putting them on local storage of each node: > > mail_location = > maildir:/srv/mail/%d/%u:INDEX=/var/lib/dovecot/index/%d/%u > > But I still hit issues like this one: > > dovecot: lmtp(6276, user at example.com): Error: Broken file > /srv/mail/example.com/user at example.com/dovecot-uidlist line 8529: UIDs > not ordered (8527 >= 8527) > > Which I am not sure how serious it is or if it's possible to solve or > workaround?You need Director for POP/IMAP and also LMTP so you can solve all "Broken file" and "corrupted indexes" problems.> > Anyway because of the above and high possibility of GlusterFS > split-brains, I have decided to setup Dovecot Director according to the > docs [1] but I have a couple of questions: > > - is custom monitoring still required? Poolmon [2] is 4 year old so I > would suppose there's some progress since that?For me poolmon works fine.> - it's not possible to have same backends and directors in Dovecot > <2.2.17. I can backport newer Dovecot for Ubuntu Trusty, so this is > not an issue, but..Yes is possibile (also with < 2.2.17), create two instances, like dovecot and director, two config directory /etc/dovecot/ and /etc/director/ and bind on differents IPs.> - documentation states that it still doesn't work for LMTP [3]? > Which is probably important for my setup, because both Postfix > servers > are using dovecot-lmtp for mail delivery so there can be still some > issues (but probably less frequent?) when both servers will deliver > new mails for one user at once. > So do I really have to split directors from backends?I'm running Director and backend on the same server for POP/IMAP, and in another configuration and Director for LMTP is on the same server (but with 2.2.19).> Anyone has experience with clustered Dovecot setup? > Why is Dovecot behaving so bad when it pretends to be shared storage > friendly? Are these issues only specific for older Dovecot? > Or is there something wrong in my architecture design?You need Director, Dovecot has not problems with shared storage, big installation are always using shared storage (like NFS). -- Alessio Cecchi Postmaster AT http://www.qboxmail.it http://www.linkedin.com/in/alessice
Hello Alessio and Gordon, thank you for answers. Dsync-based architecture looks promising, but I would preffer to stay with GlusterFS for now as I also use it as a storage for other components. So director is the way to go, I don't want to setup more than two nodes to keep this setup as simple as possible - so I will probably update to 2.2.19 and have director and backend on the same servers (and Dovecot instance). I asked about poolmon, because I think that Dovecot should have some internal mechanism on how to recognize broken backend by default. But if it works nicely, I am going to use it as well :-)> At the moment, I cannot recognize the requirement for using lmtp over > the directors. When using postfix for delivering e-mails to the > backend, do this directly with an corresponding MX record.I have two MX records of the same weight with postfix using dovecot-lmtp for delivery. So that's why I wanted to use LMTP over directors. Using lower weight for second MX is an option, but not truly master-master setup :-) Filip On 2015/12/06 02:31, Alessio Cecchi wrote:> Il 05.12.2015 10:42 Filip Pytloun ha scritto: > >Hello, > > > >I have recently setup mailserver solution using 2-node master-master > >setup (mainly based on MySQL M-M replication and GlusterFS with 2 > >replica volume) on Ubuntu 14.04 (Dovecot 2.2.9). > > > >Unfortunately even with shared-storage-aware setting: > > > > mail_nfs_index = yes > > mail_nfs_storage = yes > > mail_fsync = always > > mmap_disable = yes > > With only these setting you don't solve the problem of shared storage. > > >..I have hit strange issues pretty soon especially when user was > >manipulating same mailbox from multiple devices at the same time. > > > >Most issues was about corrupted indexes which was solved easily by just > >putting them on local storage of each node: > > > > mail_location > >maildir:/srv/mail/%d/%u:INDEX=/var/lib/dovecot/index/%d/%u > > > >But I still hit issues like this one: > > > > dovecot: lmtp(6276, user at example.com): Error: Broken file > >/srv/mail/example.com/user at example.com/dovecot-uidlist line 8529: UIDs > >not ordered (8527 >= 8527) > > > >Which I am not sure how serious it is or if it's possible to solve or > >workaround? > > You need Director for POP/IMAP and also LMTP so you can solve all "Broken > file" and "corrupted indexes" problems. > > > > >Anyway because of the above and high possibility of GlusterFS > >split-brains, I have decided to setup Dovecot Director according to the > >docs [1] but I have a couple of questions: > > > >- is custom monitoring still required? Poolmon [2] is 4 year old so I > > would suppose there's some progress since that? > > For me poolmon works fine. > > >- it's not possible to have same backends and directors in Dovecot > > <2.2.17. I can backport newer Dovecot for Ubuntu Trusty, so this is > > not an issue, but.. > > Yes is possibile (also with < 2.2.17), create two instances, like dovecot > and director, two config directory /etc/dovecot/ and /etc/director/ and bind > on differents IPs. > > >- documentation states that it still doesn't work for LMTP [3]? > > Which is probably important for my setup, because both Postfix servers > > are using dovecot-lmtp for mail delivery so there can be still some > > issues (but probably less frequent?) when both servers will deliver > > new mails for one user at once. > > So do I really have to split directors from backends? > > I'm running Director and backend on the same server for POP/IMAP, and in > another configuration and Director for LMTP is on the same server (but with > 2.2.19). > > >Anyone has experience with clustered Dovecot setup? > >Why is Dovecot behaving so bad when it pretends to be shared storage > >friendly? Are these issues only specific for older Dovecot? > >Or is there something wrong in my architecture design? > > You need Director, Dovecot has not problems with shared storage, big > installation are always using shared storage (like NFS). > -- > Alessio Cecchi > Postmaster AT http://www.qboxmail.it > http://www.linkedin.com/in/alessice-------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: Digital signature URL: <http://dovecot.org/pipermail/dovecot/attachments/20151206/f476910d/attachment-0001.sig>
On 05 Dec 2015, at 11:42, Filip Pytloun <filip at pytloun.cz> wrote:> > Anyway because of the above and high possibility of GlusterFS > split-brains, I have decided to setup Dovecot Director according to the > docs [1] but I have a couple of questions: > > - is custom monitoring still required? Poolmon [2] is 4 year old so I > would suppose there's some progress since that?I think it's always going to be a separate script. In different environments people may want to do it slightly differently.> - documentation states that it still doesn't work for LMTP [3]? > Which is probably important for my setup, because both Postfix servers > are using dovecot-lmtp for mail delivery so there can be still some > issues (but probably less frequent?) when both servers will deliver > new mails for one user at once. > So do I really have to split directors from backends?You can run director and backend in the same servers, but they'd have to be in different config files (so two dovecot instances). Or you could also do it with a single instance:> [3] "LMTP however doesn't currently support mixing recipients to both > being proxied and store locally."This is a problem only if there are multiple recipients in the same LMTP session. So if you configure your MTA to restrict the recipient limit to 1, this mixing can't happen and a single Dovecot instance can work. v2.3 will solve this problem completely.
We ran a load test using glusterfs and were able to deliver mail (I can't remember specifically how much per second, maybe 100 messages per second?) without any issues. We did use the glusterfs fuse client and not nfs, and used regular maildir. We developed a mail bot cluster that would deliver mail, and simultaneously receive and delete it with pop and IMAP and we ran into zero issues. We even had the replicas stretched between two datacenters. Not sure what the difference here is but it can be done.> On Dec 5, 2015, at 3:42 AM, Filip Pytloun <filip at pytloun.cz> wrote: > > Hello, > > I have recently setup mailserver solution using 2-node master-master > setup (mainly based on MySQL M-M replication and GlusterFS with 2 > replica volume) on Ubuntu 14.04 (Dovecot 2.2.9). > > Unfortunately even with shared-storage-aware setting: > > mail_nfs_index = yes > mail_nfs_storage = yes > mail_fsync = always > mmap_disable = yes > > ..I have hit strange issues pretty soon especially when user was > manipulating same mailbox from multiple devices at the same time. > > Most issues was about corrupted indexes which was solved easily by just > putting them on local storage of each node: > > mail_location = maildir:/srv/mail/%d/%u:INDEX=/var/lib/dovecot/index/%d/%u > > But I still hit issues like this one: > > dovecot: lmtp(6276, user at example.com): Error: Broken file /srv/mail/example.com/user at example.com/dovecot-uidlist line 8529: UIDs not ordered (8527 >= 8527) > > Which I am not sure how serious it is or if it's possible to solve or > workaround? > > Anyway because of the above and high possibility of GlusterFS > split-brains, I have decided to setup Dovecot Director according to the > docs [1] but I have a couple of questions: > > - is custom monitoring still required? Poolmon [2] is 4 year old so I > would suppose there's some progress since that? > > - it's not possible to have same backends and directors in Dovecot > <2.2.17. I can backport newer Dovecot for Ubuntu Trusty, so this is > not an issue, but.. > > - documentation states that it still doesn't work for LMTP [3]? > Which is probably important for my setup, because both Postfix servers > are using dovecot-lmtp for mail delivery so there can be still some > issues (but probably less frequent?) when both servers will deliver > new mails for one user at once. > So do I really have to split directors from backends? > > > Anyone has experience with clustered Dovecot setup? > Why is Dovecot behaving so bad when it pretends to be shared storage > friendly? Are these issues only specific for older Dovecot? > Or is there something wrong in my architecture design? > > Thanks for any help, > Filip > > > --- > [1] http://wiki2.dovecot.org/Director > [2] https://github.com/brandond/poolmon/ > [3] "LMTP however doesn't currently support mixing recipients to both > being proxied and store locally." > > --- > BTW if someone is interested in SaltStack, here are Salt formulas for > Dovecot + Postfix + GlusterFS + Roundcube + Mailman setup which we are > using: > > https://github.com/tcpcloud/salt-formula-dovecot > https://github.com/tcpcloud/salt-formula-postfix > https://github.com/tcpcloud/salt-formula-roundcube > https://github.com/tcpcloud/salt-formula-glusterfs
Am Sonntag, 6. Dezember 2015, 20:33:32 schrieb list at airstreamcomm.net:> We ran a load test using glusterfs and were able to deliver mail (I can't > remember specifically how much per second, maybe 100 messages per second?) > without any issues. We did use the glusterfs fuse client and not nfs, and > used regular maildir. We developed a mail bot cluster that would deliver > mail, and simultaneously receive and delete it with pop and IMAP and we ran > into zero issues. We even had the replicas stretched between two > datacenters. Not sure what the difference here is but it can be done. > > On Dec 5, 2015, at 3:42 AM, Filip Pytloun <filip at pytloun.cz> wrote: > > > > Hello, > > > > I have recently setup mailserver solution using 2-node master-master > > setup (mainly based on MySQL M-M replication and GlusterFS with 2 > > replica volume) on Ubuntu 14.04 (Dovecot 2.2.9). > > > > Unfortunately even with shared-storage-aware setting: > > > > mail_nfs_index = yes > > mail_nfs_storage = yes > > mail_fsync = always > > mmap_disable = yes > > > > ..I have hit strange issues pretty soon especially when user was > > manipulating same mailbox from multiple devices at the same time. > > > > Most issues was about corrupted indexes which was solved easily by just > > putting them on local storage of each node: > > > > mail_location = maildir:/srv/mail/%d/%u:INDEX=/var/lib/dovecot/index/%d/%u > > > > But I still hit issues like this one: > > > > dovecot: lmtp(6276, user at example.com): Error: Broken file > > /srv/mail/example.com/user at example.com/dovecot-uidlist line 8529: UIDs > > not ordered (8527 >= 8527) > > > > Which I am not sure how serious it is or if it's possible to solve or > > workaround?hi, I did experiments with glusterfs and dovecot about one year ago and ran into the same trouble. It was quite easy to corrupt users mailboxes delivering mail somultaniously on different nodes. I tried a lot of different configurations (see NFS recomendations, ...) but nothig really solved the problem. Also from the list I did not get any useful comments. So I decided that plain dovecot / glusterfs is not usable. I would be glad to hear of any other experiance and configuration tweaks. Mit freundlichen Gr??en, Michael Schwartzkopff -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64, +49 (162) 165 0044 Franziskanerstra?e 15, 81669 M?nchen Sitz der Gesellschaft: M?nchen, Amtsgericht M?nchen: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 230 bytes Desc: This is a digitally signed message part. URL: <http://dovecot.org/pipermail/dovecot/attachments/20151207/38c02702/attachment.sig>