thr3ads.net - dovecot - Quota status to postfix in distributed environment [Feb 2018]

If this information is useful, please help other people find it:
Share via:

SAAHIL IFTEKHAR

2018-Feb-26 16:28 UTC

Quota status to postfix in distributed environment

Hi

I have implemented Quota status to postfix in our setup. I have an imap
server (dovecot) and mail server (postfix) in every node. I am able to send
quota status to postfix and mails are rejected after 100% mail quota is
crossed. This rejection is happening both in across the nodes and within
the nodes.

The problem is if I am sending mails to any node and if any other node's
dovecot is down, mails are not going. For example, I am sending an email
within the system but if some other node's dovecot is down then email
within the system also will not go.


My dovecot version is 2.2.10.
My postfix version is 2.1.10.


*doveconf -n output is below:-*

# 2.2.10: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.0-514.el7.x86_64 x86_64 Red Hat Enterprise Linux Server
release 7.3 (Maipo) xfs
auth_debug = yes
base_dir = /var/run/dovecot/
first_valid_gid = 5000
first_valid_uid = 5000
hostname = CmdHQ
login_greeting = ^^^^^^^^^^Dovecot ready^^^^^^^^^^
mail_debug = yes
mail_gid = 6000
mail_location = Maildir:/var/mail/vmail/tcs.mil.in/%n
mail_plugins = " quota"
mail_uid = 6000
mbox_write_locks = fcntl
passdb {
  args = /etc/dovecot/dovecot-ldap.conf
  driver = ldap
}
plugin {
  quota = maildir:User quota
  quota_rule = *:storage=8KB
  quota_rule2 = *:messages=12B
  quota_status_nouser = DUNNO
  quota_status_overquota = 552 5.2.2 Mailbox is over quota / mailbox is full
  quota_status_success = DUNNO
  quota_warning = storage=80%% quota-warning 80 %u
}
postmaster_address = postmaster at tcs.mil.in
service auth {
  unix_listener auth-userdb {
    mode = 0600
    user = postfix
  }
}
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    group = postfix
    mode = 0600
    user = postfix
  }
}
service quota-status {
  client_limit = 1
  executable = quota-status -p postfix
  inet_listener {
    port = 54317
  }
}
service quota-warning {
  executable = script /usr/local/bin/quota-warning.sh
  unix_listener quota-warning {
    group = postfix
    mode = 0666
    user = postfix
  }
  user = postfix
}
ssl = required
ssl_ca = </etc/dovecot/certs/cacert.pem
ssl_cert = </etc/dovecot/certs/1CorpHQ_IMAP_Admin at tcs.mil.in.pem
ssl_key = </etc/dovecot/certs/1CorpHQ_IMAP_Admin at tcs.mil.in.key
userdb {
  args = /etc/dovecot/dovecot-ldap.conf
  driver = ldap
}
verbose_ssl = yes
protocol lmtp {
  info_log_path = /var/log/dovecot-lmtp.log
  mail_plugins = " quota"
}
protocol lda {
  info_log_path = /var/log/dovecot-lda.log
  log_path = /var/log/dovecot-lda-errors.log
  mail_plugins = " quota"
}
protocol imap {
  mail_plugins = " quota"
}

Here "service quota status" is the concerned section in conf file.
________________________________________________________________________________________________________

*Postfix configuration is below:- *

smtpd_relay_restrictions           check_policy_service inet:201.123.80.9:54317
          check_policy_service inet:201.123.80.23:54317


virtual_transport=lmtp:unix:private/dovecot-lmtp


Here, I am querying both two nodes. 201.123.80.9 is the other node.
201.123.80.23 is the node within which, email is sent.
___________________________________________________________________________________________________________



*logs while sending mail is below:-F*eb 22 12:43:24 1CorpHQ
postfix/proxymap[7327]: In dict_changed_name
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: initializing the server-side
TLS engine
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: In dict_changed_name
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: match_list_match: 1CorpHQ: no
match
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: match_list_match: 201.123.80.23:
no match
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: match_list_match: 1CorpHQ: no
match
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: match_list_match: 201.123.80.23:
no match
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 220
1CorpHQserver.tcs.mil.in ESMTP Postfix
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text EHLO 1CorpHQ
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: match_list_match: 1CorpHQ: no
match
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: match_list_match: 201.123.80.23:
no match
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text
250-1CorpHQserver.tcs.mil.in
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-PIPELINING
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-SIZE 10240000
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-VRFY
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-ETRN
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-STARTTLS
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-ENHANCEDSTATUSCODES
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-8BITMIME
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250 DSN
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text STARTTLS
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 220 2.0.0 Ready to start
TLS
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: setting up TLS connection from
1CorpHQ[201.123.80.23]
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: 1CorpHQ[201.123.80.23]: TLS
cipher list "aNULL:-aNULL:ALL:!EXPORT:!LOW:+RC4:@STRENGTH"
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:before/accept
initialization
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 read client
hello A
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 write server
hello A
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 write
certificate A
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 write key
exchange A
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 write server
done A
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 flush data
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 read client
key exchange A
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 read finished
A
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 write change
cipher spec A
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 write
finished A
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 flush data
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: Anonymous TLS connection
established from 1corphq[201.123.80.23]: TLSv1 with cipher
ECDHE-RSA-AES256-SHA (256/256 bits)
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text EHLO 1CorpHQ
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: match_list_match: 1CorpHQ: no
match
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: match_list_match: 201.123.80.23:
no match
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text
250-1CorpHQserver.tcs.mil.in
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-PIPELINING
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-SIZE 10240000
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-VRFY
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-ETRN
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-ENHANCEDSTATUSCODES
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-8BITMIME
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250 DSN
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text MAIL FROM:<
Cdr.1CorpHQ at tcs.mil.in>
Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match:
transport_maps: no match
Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match:
transport_maps: no match
Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: In dict_changed_name
Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match:
tcs.mil.in: no match
Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match:
tcs.mil.in: no match
Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match:
tcs.mil.in: no match
Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match:
tcs.mil.in: no match
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: In valid verify sender addr
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250 2.1.0 Ok
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text RCPT TO:<
CO.1CorpHQ at tcs.mil.in>
Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match:
tcs.mil.in: no match
Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match:
tcs.mil.in: no match
Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match:
tcs.mil.in: no match
Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match:
tcs.mil.in: no match
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: In valid verify sender addr
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: match_list_match:
permit_mynetworks: no match
Feb 22 12:43:24 1CorpHQ dovecot: quota-status: Debug: Loading modules from
directory: /usr/lib64/dovecot
Feb 22 12:43:24 1CorpHQ dovecot: quota-status: Debug: Module loaded:
/usr/lib64/dovecot/lib10_quota_plugin.so
Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: Loading modules from
directory: /usr/lib64/dovecot/auth
Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: Module loaded:
/usr/lib64/dovecot/auth/libdriver_sqlite.so
Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: Loading modules from
directory: /usr/lib64/dovecot/auth
Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: Module loaded:
/usr/lib64/dovecot/auth/libauthdb_ldap.so
Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: Read auth token secret from
/var/run/dovecot//auth-token-secret.dat
Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: master in: USER#0111#
011CO.1CorpHQ at tcs.mil.in#011service=quota-status
Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: ldap(co.1corphq at tcs.mil.in):
user search: base=dc=tcs,dc=mil,dc=in scope=subtree
filter=(&(objectClass=person)(uid=co.1corphq))
fields=homeDirectory,uidNumber,gidNumber
Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: ldap(co.1corphq at tcs.mil.in):
no fields returned by the server
Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: ldap(co.1corphq at tcs.mil.in):
result:  homeDirectory missing; uidNumber missing; gidNumber missing
Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: userdb out: USER#0111#
011co.1corphq at tcs.mil.in
Feb 22 12:43:24 1CorpHQ dovecot: quota-status: Debug: auth input:
co.1corphq at tcs.mil.in
Feb 22 12:43:24 1CorpHQ dovecot: quota-status: Debug: changed username to
co.1corphq at tcs.mil.in
Feb 22 12:43:24 1CorpHQ dovecot: quota-status: Debug: Added userdb setting:
plugin/=yes
Feb 22 12:43:24 1CorpHQ dovecot: quota-status(co.1corphq at tcs.mil.in):
Debug: Effective uid=6000, gid=6000, homeFeb 22 12:43:24 1CorpHQ dovecot:
quota-status(co.1corphq at tcs.mil.in):
Debug: Quota root: name=User quota backend=maildir argsFeb 22 12:43:24 1CorpHQ
dovecot: quota-status(co.1corphq at tcs.mil.in):
Debug: Quota rule: root=User quota mailbox=* bytes=8192 messages=0
Feb 22 12:43:24 1CorpHQ dovecot: quota-status(co.1corphq at tcs.mil.in):
Debug: Quota rule: root=User quota mailbox=* bytes=8192 messages=12
Feb 22 12:43:24 1CorpHQ dovecot: quota-status(co.1corphq at tcs.mil.in):
Debug: Quota warning: bytes=6553 (80%) messages=0 reverse=no
command=quota-warning 80 co.1corphq at tcs.mil.in
Feb 22 12:43:24 1CorpHQ dovecot: quota-status(co.1corphq at tcs.mil.in):
Debug: Quota grace: root=User quota bytes=819 (10%)
Feb 22 12:43:24 1CorpHQ dovecot: quota-status(co.1corphq at tcs.mil.in):
Debug: maildir++: root=/var/mail/vmail/tcs.mil.in/co.1corphq, index=,
indexpvt=, control=, inbox=/var/mail/vmail/tcs.mil.in/co.1corphq, altFeb 22
12:43:24 1CorpHQ postfix/smtpd[7326]: warning: connect to
201.123.80.9:54317: Connection refused
Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: warning: problem talking to
server 201.123.80.9:54317: Connection refused
Feb 22 12:43:25 1CorpHQ postfix/smtpd[7326]: warning: connect to
201.123.80.9:54317: Connection refused
Feb 22 12:43:25 1CorpHQ postfix/smtpd[7326]: warning: problem talking to
server 201.123.80.9:54317: Connection refused
Feb 22 12:43:25 1CorpHQ postfix/smtpd[7326]: NOQUEUE: reject: RCPT from
1CorpHQ[201.123.80.23]: 451 4.3.5 Server configuration problem; from=<
Cdr.1CorpHQ at tcs.mil.in> to=<CO.1CorpHQ at tcs.mil.in> proto=ESMTP
helo=<1CorpHQ>
Feb 22 12:43:25 1CorpHQ postfix/smtpd[7326]: text 451 4.3.5 Server
configuration problem
Feb 22 12:43:25 1CorpHQ postfix/smtpd[7326]: text RSET
Feb 22 12:43:25 1CorpHQ postfix/smtpd[7326]: text 250 2.0.0 Ok
Feb 22 12:43:25 1CorpHQ postfix/smtpd[7326]: lost connection after RSET
from 1CorpHQ[201.123.80.23]

I am understanding what the logs are trying to say. But I am not able to
resolve the issue even after searching solution on internet and trying
different hit and trials by myself. I want that if i am sending email to
any node or within node, the configuration relating to "check
_policy_service" for other node does not interfere and mail goes properly.
At the same time I can also fetch quota status from other nodes.

If I can get any help regarding this it will be really appreciable as I
have tried a lot of options already.


Regards
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20180226/5d218daf/attachment-0001.html>

@lbutlr

2018-Feb-26 16:32 UTC

head link

Quota status to postfix in distributed environment

On Feb 26, 2018, at 08:28, SAAHIL IFTEKHAR <response2saahil at gmail.com>
wrote:> My dovecot version is 2.2.10.
> My postfix version is 2.1.10.
What happens if you use current-ish versions. 

-- 
This is my signature. There are many like it, but this one is mine.

Karol Augustin

2018-Feb-27 14:18 UTC

head link

Quota status to postfix in distributed environment

On 2018-02-26 16:28, SAAHIL IFTEKHAR wrote:
> Hi
> 
> I have implemented Quota status to postfix in our setup. I have an imap
server (dovecot) and mail server (postfix) in every node. I am able to send
quota status to postfix and mails are rejected after 100% mail quota is crossed.
This rejection is happening both in across the nodes and within the nodes.
> 
> The problem is if I am sending mails to any node and if any other
node's dovecot is down, mails are not going. For example, I am sending an
email within the system but if some other node's dovecot is down then email
within the system also will not go.
> 
> My dovecot version is 2.2.10. My postfix version is 2.1.10.
> 
> doveconf -n output is below:-
> 
[snap]> 
> Here "service quota status" is the concerned section in conf
file.
>
________________________________________________________________________________________________________
> 
> Postfix configuration is below:- 
> smtpd_relay_restrictions = 
> check_policy_service inet:201.123.80.9:54317
> check_policy_service inet:201.123.80.23:54317
> 
> virtual_transport=lmtp:unix:private/dovecot-lmtp
> 
> Here, I am querying both two nodes. 201.123.80.9 is the other node.
201.123.80.23 is the node within which, email is sent.
>
___________________________________________________________________________________________________________
> 
> logs while sending mail is below:-
> [snap]
> 
> I am understanding what the logs are trying to say. But I am not able to
resolve the issue even after searching solution on internet and trying different
hit and trials by myself. I want that if i am sending email to any node or
within node, the configuration relating to "check _policy_service" for
other node does not interfere and mail goes properly. At the same time I can
also fetch quota status from other nodes.
> 
> If I can get any help regarding this it will be really appreciable as I
have tried a lot of options already.
> Your problem is that if the dovecot is unreachable Postfix can't check
the quota for the user.

http://www.postfix.org/SMTPD_POLICY_README.html
explains how to resolve this problem:
smtpd_policy_service_default_action (default: 451 4.3.5 Server
configuration problem): The default action when an SMTPD policy service
request fails. Available with Postfix 3.0 and later.

The default action when an SMTPD policy service request fails. Specify
"DUNNO" to behave as if the failed SMTPD policy service request was
not
sent, and to continue processing other access restrictions, if any. You
can configure it per policy also.

The issue you might encounter with this setup is that if both of your
dovecot nodes are unreachable Postfix will accept the e-mail and try to
deliver it. When the nodes come online it will fail if the user is over
quota and generate bounce to the envelope sender of the message, which
might produce backscatter.

I resolved this problem by keeping quota information in Mysql table and
using custom policy server to check if user is allowed to receive more
e-mail.

Also this approach might not work with aliases and other redirections,
the e-mail address checked by smtpd policy is the one before alias
expansion. You should check if it suits your environment.

Best,
Karol

-- 
Karol Augustin
karol at augustin.pl
http://karolaugustin.pl/
+353 85 775 5312

LuKreme

2018-Feb-27 15:26 UTC

head link

Quota status to postfix in distributed environment

Out of curiosity would it be possible to configure dovecot in such a way that if
a message had been accepted by postfix because it couldn't check quota the
message was delivered anyway, despite being over quota?

I'd much rather that then risk backscatter.

(I used to have quotas, but with the size of disks I've found it easier to
simply have a "clue bat?approach and a reasonable max-message size. When a
user gets "too much?mail ask them to purge their older mails with
attachments.

Granted, that's probably less workable than quotas for a large user base,
but for a period of time I did automate these notices by listing users total
storage and sending a heads-up message when they got over a specific number.
Granted, that was in the days where a 100MB mail store was a reasonable limit.

-- 
My main job is trying to come up with new and innovative and effective ways to
reject even more mail. I'm up to about 97% now.
> On Feb 27, 2018, at 06:18, Karol Augustin <karol at augustin.pl>
wrote:
> 
> The issue you might encounter with this setup is that if both of your
> dovecot nodes are unreachable Postfix will accept the e-mail and try to
> deliver it. When the nodes come online it will fail if the user is over
> quota and generate bounce to the envelope sender of the message, which
> might produce backscatter.
> 
> I resolved this problem by keeping quota information in Mysql table and
> using custom policy server to check if user is allowed to receive more
> e-mail.

Possibly Parallel Threads

Search for more maybe matching threads

dovecot - Feb 2018 - Quota status to postfix in distributed environment

Quota status to postfix in distributed environment

Quota status to postfix in distributed environment

Quota status to postfix in distributed environment

Quota status to postfix in distributed environment

Possibly Parallel Threads