Hello list, we are experimenting with using Dovecot as part of a simplified mail stack with SQL for a internal messaging / notification system for our customers. Sometimes we need to send a message to thousands of customers at once and we do it by having aliases resolve to 10k and 20k+ mail accounts. When we send such a message the server load (obviously) is very high for a few minutes (in case of an alias containing 10k addresses its around 6-7 minutes for a 2 core machine with load around 20-30). Are there any optimizations or ways how to design the system around this problem ? Eg. is there a way how to deliver messages more slowly, and spread the load over longer time ? The stack is very simple with Postfix receiving the messages and Dovecot acting as a LMTP and IMAP server. Mailboxes are in the Maildir format on disk. Customers get the messages on login, a PHP script connects to the mail account associated with their ID via IMAP and sends unseen messages in the form of a JSON. Best regards, gnd/
<!doctype html>
<html>
<head>
<meta charset="UTF-8">
</head>
<body>
<div>
<br>
</div>
<blockquote type="cite">
<div>
On 14/07/2020 10:52 gnd <<a
href="mailto:gnd@itchybit.org">gnd@itchybit.org</a>>
wrote:
</div>
<div>
<br>
</div>
<div>
<br>
</div>
<div>
Hello list,
</div>
<div>
<br>
</div>
<div>
we are experimenting with using Dovecot as part of a simplified mail
</div>
<div>
stack with SQL for a internal messaging / notification system for our
</div>
<div>
customers. Sometimes we need to send a message to thousands of customers
</div>
<div>
at once and we do it by having aliases resolve to 10k and 20k+ mail
</div>
<div>
accounts.
</div>
<div>
<br>
</div>
<div>
When we send such a message the server load (obviously) is very high for
</div>
<div>
a few minutes (in case of an alias containing 10k addresses its around
</div>
<div>
6-7 minutes for a 2 core machine with load around 20-30).
</div>
<div>
<br>
</div>
<div>
Are there any optimizations or ways how to design the system around this
</div>
<div>
problem ? Eg. is there a way how to deliver messages more slowly, and
</div>
<div>
spread the load over longer time ?
</div>
<div>
<br>
</div>
<div>
The stack is very simple with Postfix receiving the messages and Dovecot
</div>
<div>
acting as a LMTP and IMAP server. Mailboxes are in the Maildir format on
</div>
<div>
disk. Customers get the messages on login, a PHP script connects to the
</div>
<div>
mail account associated with their ID via IMAP and sends unseen messages
</div>
<div>
in the form of a JSON.
</div>
<div>
<br>
</div>
<div>
Best regards,
</div>
<div>
<br>
</div>
<div>
gnd/
</div>
</blockquote>
<div class="default-style">
<br>
</div>
<div class="default-style">
Can you provide output of doveconf -n?
</div>
<div class="default-style">
<br>
</div>
<div class="default-style">
Also, verbose_proctitle=yes gives more insight what the processes are doing.
</div>
<div class="default-style">
<br>
</div>
<div class="default-style">
Aki
</div>
</body>
</html>
gnd> we are experimenting with using Dovecot as part of a simplified gnd> mail stack with SQL for a internal messaging / notification gnd> system for our customers. Sometimes we need to send a message to gnd> thousands of customers at once and we do it by having aliases gnd> resolve to 10k and 20k+ mail accounts. gnd> When we send such a message the server load (obviously) is very gnd> high for a few minutes (in case of an alias containing 10k gnd> addresses its around 6-7 minutes for a 2 core machine with load gnd> around 20-30). gnd> Are there any optimizations or ways how to design the system gnd> around this problem ? Eg. is there a way how to deliver messages gnd> more slowly, and spread the load over longer time ? This is not a dovecot issue, but a postfix issue. You need to setup some queues in postfix to handle mail floods like this. You should take this to the postfix mailing list on postfix.org, and first look in the archives, there have been recent discussions about this type of issue. gnd> The stack is very simple with Postfix receiving the messages and gnd> Dovecot acting as a LMTP and IMAP server. Mailboxes are in the gnd> Maildir format on disk. Customers get the messages on login, a gnd> PHP script connects to the mail account associated with their ID gnd> via IMAP and sends unseen messages in the form of a JSON. Postfix is also sending the emails too. :-) John
Yes you are right, i realized that just after i sent the email, that it was a bit in haste. anyway, if there is anyway how to optimize dovecot for effective delivery of 20k+ emails within a few minutes, id be glad to know. .. and i will have to ask the same q at postfix mailinglists gnd/ On 7/14/20 8:12 PM, John Stoffel wrote:> > gnd> we are experimenting with using Dovecot as part of a simplified > gnd> mail stack with SQL for a internal messaging / notification > gnd> system for our customers. Sometimes we need to send a message to > gnd> thousands of customers at once and we do it by having aliases > gnd> resolve to 10k and 20k+ mail accounts. > > gnd> When we send such a message the server load (obviously) is very > gnd> high for a few minutes (in case of an alias containing 10k > gnd> addresses its around 6-7 minutes for a 2 core machine with load > gnd> around 20-30). > > gnd> Are there any optimizations or ways how to design the system > gnd> around this problem ? Eg. is there a way how to deliver messages > gnd> more slowly, and spread the load over longer time ? > > This is not a dovecot issue, but a postfix issue. You need to setup > some queues in postfix to handle mail floods like this. You should > take this to the postfix mailing list on postfix.org, and first look > in the archives, there have been recent discussions about this type of > issue. > > gnd> The stack is very simple with Postfix receiving the messages and > gnd> Dovecot acting as a LMTP and IMAP server. Mailboxes are in the > gnd> Maildir format on disk. Customers get the messages on login, a > gnd> PHP script connects to the mail account associated with their ID > gnd> via IMAP and sends unseen messages in the form of a JSON. > > Postfix is also sending the emails too. :-) > > John >