after replication with compression quotas are wrong

Aki Tuomi
Mon, 08 Nov 2021 10:30:55 -0000

Recalculation won't fix replication mistakes.

Did you ensure on the *target* server that it has only dummy-c-1 in replication
and that the file count on the target server matches source server?

You might also get something useful out of

`doveadm -D quota recalc|get -u user`

Aki

> On 08/11/2021 12:22 Arnaud Ab?lard <arnaud.abelard at univ-nantes.fr>
wrote:
>
>
> On 08/11/2021 10:39, Aki Tuomi wrote:
> > Try doveadm replicator remove dummy-c-1 at univ-nantes.fr
>
> I did so on the source server, checked indeed that
> dummy-c-1 at univ-nantes.fr wasn't showing up in the doveadm replicator
> status list and it was indeed the case, so far so good. On the
> destination server, dummy-c-1's mailbox was still showing a 115% quota
> so I tried to make dovecot recalculate the quota but it did not change
> anything. I ended up deleting the user's mailbox and forced a
> replication and, to my surprise, the new mailbox still has a 115% quota.
>
> Arnaud
>
> >
> > Aki
> >
> >> On 08/11/2021 11:32 Arnaud Ab?lard <arnaud.abelard at
univ-nantes.fr> wrote:
> >>
> >>
> >> Hello Aki,
> >>
> >> Thanks for the tip. I changed postfix configuration so it will not
use
> >> the domain part anymore and added "auth_username_format =
%Ln" just in
> >> case (and reloaded dovecot). So I shouldn't have any more new
> >> user at domain users but for all the existing users, doveadm
replicator
> >> status still show user at domain clone and on the new server mails
are
> >> still counted twice.
> >>
> >> Our userdb and passdb don't have the @domain defined anywhere
so I don't
> >> think it's coming from there.
> >>
> >> Is there a local cache that still has a list of those user at
domain
> >> usernames that I could reset or something?
> >>
> >> Thanks,
> >>
> >> Arnaud
> >>
> >>
> >>
> >> On 08/11/2021 09:48, Aki Tuomi wrote:
> >>> This sounds like you are not normalizing usernames properly.
> >>>
> >>> Either use
> >>>
> >>> auth_username_format = %Ln
> >>>
> >>> as global setting, or return `user` attribute in both userdb
and passdb lookups.
> >>>
> >>> Aki
> >>>
> >>>> On 07/11/2021 20:31 Arnaud Ab?lard <arnaud.abelard at
univ-nantes.fr> wrote:
> >>>>
> >>>>
> >>>> Hello again,
> >>>>
> >>>> I've found out that some mailboxes  are actually
duplicated. Doveadm
> >>>> replicator status on the production server gives this:
> >>>>
> >>>> ~# doveadm replicator status 'dummy-c-1*'
> >>>> username                              priority fast sync
full sync
> >>>> success sync failed
> >>>> dummy-c-1                             none     01:13:19
01:13:19
> >>>> 01:13:19     -
> >>>> dummy-c-1 at univ-nantes.fr              none     00:15:28
00:15:28
> >>>> 00:15:28     -
> >>>>
> >>>> That'd explain why mails are counted twice when
replicated on the new
> >>>> server but where does this come from since I don't
have this quota
> >>>> problem on the production server?
> >>>>
> >>>>    From the logs, it seems that postfix uses username at
univ-nantes.fr when
> >>>> calling dovecot's LDA so I suppose that the
non-'@univ-nantes.fr'
> >>>> mailboxes are the erroneous ones.
> >>>>
> >>>> The users aren't duplicated in our ldap user backend
and aren't using
> >>>> the @univ-nantes.fr part.
> >>>>
> >>>> ~# doveadm user 'dummy-c-1*'
> >>>> dummy-c-1
> >>>>
> >>>> Any ideas?
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Arnaud
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On 05/11/2021 16:21, Arnaud Ab?lard wrote:
> >>>>> Hello,
> >>>>>
> >>>>> We are very long time happy dovecot users (since 2008
at least). We have
> >>>>> around 90k mailboxes and since we had to move away
from our NAS storage
> >>>>> to a ceph storage I jumped on the opportunity to
enable compression with
> >>>>> the zlib plugin and dovecot's replication
mecanism. We are using
> >>>>> debian's dovecot 2.2.27 packages on production and
our new server is
> >>>>> running dovecot's own ce-2.3.17 packages.
> >>>>>
> >>>>> On the production server everything works fine but on
the new server,
> >>>>> replicated mailboxes' quota is all wrong:
> >>>>>
> >>>>> on production:
> >>>>> # doveadm quota get -u dummy-c-1
> >>>>> Quota name??????? Type???? Value?
Limit???????????????????????????? %
> >>>>> Quota Utilisateur STORAGE 660026
976563??????????????????????????? 67
> >>>>> Quota Utilisateur MESSAGE?? 8651?????
-???????????????????????????? 0
> >>>>>
> >>>>> on new server:
> >>>>> doveadm quota get -u dummy-c-1
> >>>>> Quota name??????? Type????? Value?
Limit?????????????????????????? %
> >>>>> Quota Utilisateur STORAGE 1125251
976563???????????????????????? 115
> >>>>> Quota Utilisateur MESSAGE?? 16646?????
-?????????????????????????? 0
> >>>>>
> >>>>> If I add all the S flag from the filenames n both
servers I get exactly
> >>>>> the same usage, which is coherent with the quota on
the production server:
> >>>>>
> >>>>> # find . -type f | grep 'S=' | awk
-F'S=' '{print $2}' | awk -F','
> >>>>> '{print $1}' | awk -F':' '{print
$1}' | paste -sd+ | bc -l
> >>>>> 675865938
> >>>>>
> >>>>> And I have exactly the same amountof mails on the two
server, the
> >>>>> replication works as expected, no unwanted duplication
of mails occurs.
> >>>>>
> >>>>> Of course, I've tried to ask dovecot to
recalculate quotas with doveadm
> >>>>> quota recalc -u <username>, but it doesn't
fix the problem.
> >>>>>
> >>>>> What am I missing?
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Arnaud
> >>>>>
> >>>>> PS: Here is my doveconf -n output:
> >>>>>
> >>>>> # 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf
> >>>>> # Pigeonhole version 0.5.17 (054dddfa)
> >>>>> # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11
> >>>>> # Hostname: gromel-test
> >>>>> auth_cache_size = 10 k
> >>>>> auth_verbose = yes
> >>>>> disable_plaintext_auth = no
> >>>>> doveadm_password = # hidden, use -P to show it
> >>>>> hostname = gromel1.univ-nantes.prive
> >>>>> lda_mailbox_autosubscribe = yes
> >>>>> listen = *
> >>>>> log_timestamp = "%Y-%m-%d %H:%M:%S "
> >>>>> login_trusted_networks = (...)
> >>>>> mail_gid = 5000
> >>>>> mail_location = maildir:%h
> >>>>> mail_plugins = quota zlib notify replication
> >>>>> mail_privileged_group = vmail
> >>>>> mail_uid = 5000
> >>>>> maildir_stat_dirs = yes
> >>>>> maildir_very_dirty_syncs = yes
> >>>>> managesieve_notify_capability = mailto
> >>>>> managesieve_sieve_capability = fileinto reject
envelope
> >>>>> encoded-character subaddress
comparator-i;ascii-numeric relational regex
> >>>>> imap4flags copy include variables body enotify
environment mailbox date
> >>>>> index ihave duplicate mime foreverypart extracttext
> >>>>> namespace {
> >>>>>    ? inbox = yes
> >>>>>    ? location > >>>>>    ? prefix =
INBOX.
> >>>>>    ? separator = .
> >>>>>    ? type = private
> >>>>> }
> >>>>> passdb {
> >>>>>    ? args = /etc/dovecot/dovecot-ldap.conf
> >>>>>    ? driver = ldap
> >>>>> }
> >>>>> plugin {
> >>>>>    ? quota = maildir:Quota Utilisateur
> >>>>>    ? quota_exceeded_message = Cet utilisateur a
d?pass? son quota, votre
> >>>>> message n'a pu lui ?tre livr?.
> >>>>>    ? quota_full_tempfail = yes
> >>>>>    ? quota_rule = *:storage=1000M
> >>>>>    ? quota_rule2 = INBOX.Trash:storage=+100M
> >>>>>    ? sieve = ~/dovecot.sieve
> >>>>>    ? sieve_dir = ~/sieve
> >>>>>    ? sieve_extensions = -vacation
> >>>>>    ? sieve_global_dir = /var/lib/dovecot/sieve/global/
> >>>>>    ? sieve_max_redirects = 1
> >>>>>    ? zlib_save = gz
> >>>>>    ? zlib_save_level = 6
> >>>>> }
> >>>>> postmaster_address = postmaster@<snip>
> >>>>> protocols = imap pop3 sieve
> >>>>> replication_max_conns = 50
> >>>>> service auth {
> >>>>>    ? client_limit = 49452
> >>>>>    ? unix_listener auth-userdb {
> >>>>>    ??? group = vmail
> >>>>>    ??? mode = 0600
> >>>>>    ??? user = vmail
> >>>>>    ? }
> >>>>>    ? user = root
> >>>>> }
> >>>>> service dict {
> >>>>>    ? unix_listener dict {
> >>>>>    ??? mode = 0600
> >>>>>    ??? user = vmail
> >>>>>    ? }
> >>>>> }
> >>>>> service doveadm {
> >>>>>    ? inet_listener {
> >>>>>    ??? port = 12345
> >>>>>    ? }
> >>>>> }
> >>>>> service imap-login {
> >>>>>    ? process_min_avail = 8
> >>>>>    ? service_count = 0
> >>>>>    ? user = mail
> >>>>> }
> >>>>> service imap {
> >>>>>    ? executable = imap
> >>>>>    ? process_limit = 16384
> >>>>>    ? vsz_limit = 2 G
> >>>>> }
> >>>>> service managesieve-login {
> >>>>>    ? inet_listener sieve {
> >>>>>    ??? port = 4190
> >>>>>    ? }
> >>>>>    ? process_min_avail = 8
> >>>>>    ? service_count = 0
> >>>>>    ? user = mail
> >>>>>    ? vsz_limit = 2 G
> >>>>> }
> >>>>> service managesieve {
> >>>>>    ? drop_priv_before_exec = yes
> >>>>>    ? process_limit = 16384
> >>>>> }
> >>>>> service pop3-login {
> >>>>>    ? process_min_avail = 8
> >>>>>    ? service_count = 0
> >>>>>    ? user = mail
> >>>>> }
> >>>>> service pop3 {
> >>>>>    ? drop_priv_before_exec = yes
> >>>>>    ? process_limit = 16384
> >>>>> }
> >>>>> ssl = no
> >>>>> userdb {
> >>>>>    ? args = /etc/dovecot/dovecot-ldap.conf
> >>>>>    ? driver = ldap
> >>>>> }
> >>>>> verbose_proctitle = yes
> >>>>> protocol imap {
> >>>>>    ? mail_max_userip_connections = 50
> >>>>>    ? mail_plugins = quota zlib notify replication
imap_quota zlib
> >>>>> }
> >>>>> protocol pop3 {
> >>>>>    ? mail_plugins = quota zlib notify replication
> >>>>> }
> >>>>> protocol sieve {
> >>>>>    ? mail_max_userip_connections = 10
> >>>>> }
> >>>>> protocol lda {
> >>>>>    ? mail_plugins = quota zlib notify replication
sieve zlib
> >>>>> }
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> Arnaud Ab?lard
> >>>> Responsable p?le Syst?me et Stockage
> >>>> Service Infrastructures
> >>>> DSIN Universit? de Nantes
> >>>> -
> >>
> >> --
> >> Arnaud Ab?lard
> >> Responsable p?le Syst?me et Stockage
> >> Service Infrastructures
> >> DSIN Universit? de Nantes
> >> -
>
> --
> Arnaud Ab?lard
> Responsable p?le Syst?me et Stockage
> Service Infrastructures
> DSIN Universit? de Nantes
> -