I've just removed dummy-c-1's mailbox completely on the target server,
removed the quota plugin, forced a sync for dummy-c-1 only and once it
was done reactivated the quota plugins: double usage again.
I also downgraded from dovecot 2.3.17 to 2.2.27 on the target server in
order to have both server running the same version, without improvement.
I'm puzzled. Is dovecot storing anything outside of the user's mailbox?
Like a cache, a sqlite database of some kind somewhere?
Arnaud
On 08/11/2021 11:48, Arnaud Ab?lard wrote:> On the target, I enabled the replication service without mail_replica and:
>
> doveadm replicator status 'dummy-c-1*'
> username ??? priority fast sync full sync success sync failed
> dummy-c-1 ??? none???? -???????? -???????? -??????????? -
>
> It only knows of dummy-c-1, no trace of his evil twin
> dummy-c-1 at univ-nantes.fr.
>
> On the target, I do have the same number of files
>
> find . -type f |wc -l
> 8705
>
> which is around half of what the quota is reporting (plus de index
> files, etc):
>
> ~# doveadm -D quota get -u dummy-c-1
> Debug: Loading modules from directory: /usr/lib/dovecot/modules
> Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so
> Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so
> Debug: Module loaded: /usr/lib/dovecot/modules/lib20_replication_plugin.so
> Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so
> Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm
> Debug: Skipping module doveadm_acl_plugin, because dlopen() failed:
> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined
> symbol: acl_user_module (this is usually intentional, so just ignore
> this message)
> Debug: Skipping module doveadm_expire_plugin, because dlopen() failed:
> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so:
> undefined symbol: expire_set_deinit (this is usually intentional, so
> just ignore this message)
> Debug: Module loaded:
> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so
> Debug: Module loaded:
> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so
> Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen()
> failed:
> /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so:
> undefined symbol: lucene_index_iter_deinit (this is usually intentional,
> so just ignore this message)
> Debug: Skipping module doveadm_fts_plugin, because dlopen() failed:
> /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined
> symbol: fts_user_get_language_list (this is usually intentional, so just
> ignore this message)
> Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen()
> failed:
> /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so:
> undefined symbol: mail_crypt_box_get_pvt_digests (this is usually
> intentional, so just ignore this message)
> doveadm(dummy-c-1)<24051><>: Debug: auth USER input: dummy-c-1
> home=/vmail/d/u/dummy-c-1/ quota_rule=*:backend=1000000000S
> doveadm(dummy-c-1)<24051><>: Debug: Added userdb setting:
> plugin/quota_rule=*:backend=1000000000S
> doveadm(dummy-c-1): Debug: Effective uid=5000, gid=5000,
> home=/vmail/d/u/dummy-c-1/
> doveadm(dummy-c-1): Debug: Quota root: name=Quota Utilisateur
> backend=maildir args> doveadm(dummy-c-1): Debug: Quota rule: root=Quota
Utilisateur mailbox=*
> bytes=1000000000 messages=0
> doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur
> mailbox=INBOX.Trash bytes=+104857600 messages=0
> doveadm(dummy-c-1): Debug: Quota grace: root=Quota Utilisateur
> bytes=100000000 (10%)
> doveadm(dummy-c-1): Debug: replication: No mail_replica setting -
> replication disabled
> doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=INBOX.,
> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
> location=maildir:/vmail/d/u/dummy-c-1/
> doveadm(dummy-c-1): Debug: maildir++: root=/vmail/d/u/dummy-c-1, index=,
> indexpvt=, control=, inbox=/vmail/d/u/dummy-c-1, alt>
doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=, sep=,
> inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none
> doveadm(dummy-c-1): Debug: none: root=, index=, indexpvt=, control=,
> inbox=, alt> doveadm(dummy-c-1): Debug: quota: quota_over_flag check:
> quota_over_script unset - skipping
> Quota name??????? Type????? Value? Limit
> ??????????????????????????????????????????? %
> Quota Utilisateur STORAGE 1126751 976563
> ????????????????????????????????????????? 115
> Quota Utilisateur MESSAGE?? 16686????? -
> ??????????????????????????????????????????? 0
>
> The debug mode isn't much of any help here.
>
> Arnaud
>
> On 08/11/2021 11:30, Aki Tuomi wrote:
>> Recalculation won't fix replication mistakes.
>>
>> Did you ensure on the *target* server that it has only dummy-c-1 in
>> replication and that the file count on the target server matches
>> source server?
>>
>> You might also get something useful out of
>>
>> `doveadm -D quota recalc|get -u user`
>>
>> Aki
>>
>>> On 08/11/2021 12:22 Arnaud Ab?lard <arnaud.abelard at
univ-nantes.fr>
>>> wrote:
>>>
>>> On 08/11/2021 10:39, Aki Tuomi wrote:
>>>> Try doveadm replicator remove dummy-c-1 at univ-nantes.fr
>>>
>>> I did so on the source server, checked indeed that
>>> dummy-c-1 at univ-nantes.fr wasn't showing up in the doveadm
replicator
>>> status list and it was indeed the case, so far so good. On the
>>> destination server, dummy-c-1's mailbox was still showing a
115% quota
>>> so I tried to make dovecot recalculate the quota but it did not
change
>>> anything. I ended up deleting the user's mailbox and forced a
>>> replication and, to my surprise, the new mailbox still has a 115%
quota.
>>>
>>> Arnaud
>>>
>>>>
>>>> Aki
>>>>
>>>>> On 08/11/2021 11:32 Arnaud Ab?lard <arnaud.abelard at
univ-nantes.fr>
>>>>> wrote:
>>>>>
>>>>> Hello Aki,
>>>>>
>>>>> Thanks for the tip. I changed postfix configuration so it
will not use
>>>>> the domain part anymore and added
"auth_username_format = %Ln" just in
>>>>> case (and reloaded dovecot). So I shouldn't have any
more new
>>>>> user at domain users but for all the existing users,
doveadm replicator
>>>>> status still show user at domain clone and on the new
server mails are
>>>>> still counted twice.
>>>>>
>>>>> Our userdb and passdb don't have the @domain defined
anywhere so I
>>>>> don't
>>>>> think it's coming from there.
>>>>>
>>>>> Is there a local cache that still has a list of those user
at domain
>>>>> usernames that I could reset or something?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Arnaud
>>>>>
>>>>>
>>>>>
>>>>> On 08/11/2021 09:48, Aki Tuomi wrote:
>>>>>> This sounds like you are not normalizing usernames
properly.
>>>>>>
>>>>>> Either use
>>>>>>
>>>>>> auth_username_format = %Ln
>>>>>>
>>>>>> as global setting, or return `user` attribute in both
userdb and
>>>>>> passdb lookups.
>>>>>>
>>>>>> Aki
>>>>>>
>>>>>>> On 07/11/2021 20:31 Arnaud Ab?lard
>>>>>>> <arnaud.abelard at univ-nantes.fr> wrote:
>>>>>>>
>>>>>>> Hello again,
>>>>>>>
>>>>>>> I've found out that some mailboxes? are
actually duplicated. Doveadm
>>>>>>> replicator status on the production server gives
this:
>>>>>>>
>>>>>>> ~# doveadm replicator status 'dummy-c-1*'
>>>>>>> username????????????????????????????? priority fast
sync full sync
>>>>>>> success sync failed
>>>>>>> dummy-c-1???????????????????????????? none????
01:13:19? 01:13:19
>>>>>>> 01:13:19???? -
>>>>>>> dummy-c-1 at univ-nantes.fr????????????? none????
00:15:28? 00:15:28
>>>>>>> 00:15:28???? -
>>>>>>>
>>>>>>> That'd explain why mails are counted twice when
replicated on the
>>>>>>> new
>>>>>>> server but where does this come from since I
don't have this quota
>>>>>>> problem on the production server?
>>>>>>>
>>>>>>> ??? From the logs, it seems that postfix uses
>>>>>>> username at univ-nantes.fr when
>>>>>>> calling dovecot's LDA so I suppose that the
non-'@univ-nantes.fr'
>>>>>>> mailboxes are the erroneous ones.
>>>>>>>
>>>>>>> The users aren't duplicated in our ldap user
backend and aren't
>>>>>>> using
>>>>>>> the @univ-nantes.fr part.
>>>>>>>
>>>>>>> ~# doveadm user 'dummy-c-1*'
>>>>>>> dummy-c-1
>>>>>>>
>>>>>>> Any ideas?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Arnaud
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 05/11/2021 16:21, Arnaud Ab?lard wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> We are very long time happy dovecot users
(since 2008 at least).
>>>>>>>> We have
>>>>>>>> around 90k mailboxes and since we had to move
away from our NAS
>>>>>>>> storage
>>>>>>>> to a ceph storage I jumped on the opportunity
to enable
>>>>>>>> compression with
>>>>>>>> the zlib plugin and dovecot's replication
mecanism. We are using
>>>>>>>> debian's dovecot 2.2.27 packages on
production and our new
>>>>>>>> server is
>>>>>>>> running dovecot's own ce-2.3.17 packages.
>>>>>>>>
>>>>>>>> On the production server everything works fine
but on the new
>>>>>>>> server,
>>>>>>>> replicated mailboxes' quota is all wrong:
>>>>>>>>
>>>>>>>> on production:
>>>>>>>> # doveadm quota get -u dummy-c-1
>>>>>>>> Quota name??????? Type???? Value
>>>>>>>> Limit???????????????????????????? %
>>>>>>>> Quota Utilisateur STORAGE 660026
>>>>>>>> 976563??????????????????????????? 67
>>>>>>>> Quota Utilisateur MESSAGE?? 8651
>>>>>>>> -???????????????????????????? 0
>>>>>>>>
>>>>>>>> on new server:
>>>>>>>> doveadm quota get -u dummy-c-1
>>>>>>>> Quota name??????? Type????? Value
>>>>>>>> Limit?????????????????????????? %
>>>>>>>> Quota Utilisateur STORAGE 1125251 976563
>>>>>>>> 115
>>>>>>>> Quota Utilisateur MESSAGE?? 16646
>>>>>>>> -?????????????????????????? 0
>>>>>>>>
>>>>>>>> If I add all the S flag from the filenames n
both servers I get
>>>>>>>> exactly
>>>>>>>> the same usage, which is coherent with the
quota on the
>>>>>>>> production server:
>>>>>>>>
>>>>>>>> # find . -type f | grep 'S=' | awk
-F'S=' '{print $2}' | awk -F','
>>>>>>>> '{print $1}' | awk -F':'
'{print $1}' | paste -sd+ | bc -l
>>>>>>>> 675865938
>>>>>>>>
>>>>>>>> And I have exactly the same amountof mails on
the two server, the
>>>>>>>> replication works as expected, no unwanted
duplication of mails
>>>>>>>> occurs.
>>>>>>>>
>>>>>>>> Of course, I've tried to ask dovecot to
recalculate quotas with
>>>>>>>> doveadm
>>>>>>>> quota recalc -u <username>, but it
doesn't fix the problem.
>>>>>>>>
>>>>>>>> What am I missing?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Arnaud
>>>>>>>>
>>>>>>>> PS: Here is my doveconf -n output:
>>>>>>>>
>>>>>>>> # 2.3.17 (e2aa53df5b):
/etc/dovecot/dovecot.conf
>>>>>>>> # Pigeonhole version 0.5.17 (054dddfa)
>>>>>>>> # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11
>>>>>>>> # Hostname: gromel-test
>>>>>>>> auth_cache_size = 10 k
>>>>>>>> auth_verbose = yes
>>>>>>>> disable_plaintext_auth = no
>>>>>>>> doveadm_password = # hidden, use -P to show it
>>>>>>>> hostname = gromel1.univ-nantes.prive
>>>>>>>> lda_mailbox_autosubscribe = yes
>>>>>>>> listen = *
>>>>>>>> log_timestamp = "%Y-%m-%d %H:%M:%S "
>>>>>>>> login_trusted_networks = (...)
>>>>>>>> mail_gid = 5000
>>>>>>>> mail_location = maildir:%h
>>>>>>>> mail_plugins = quota zlib notify replication
>>>>>>>> mail_privileged_group = vmail
>>>>>>>> mail_uid = 5000
>>>>>>>> maildir_stat_dirs = yes
>>>>>>>> maildir_very_dirty_syncs = yes
>>>>>>>> managesieve_notify_capability = mailto
>>>>>>>> managesieve_sieve_capability = fileinto reject
envelope
>>>>>>>> encoded-character subaddress
comparator-i;ascii-numeric
>>>>>>>> relational regex
>>>>>>>> imap4flags copy include variables body enotify
environment
>>>>>>>> mailbox date
>>>>>>>> index ihave duplicate mime foreverypart
extracttext
>>>>>>>> namespace {
>>>>>>>> ??? ? inbox = yes
>>>>>>>> ??? ? location >>>>>>>>
??? ? prefix = INBOX.
>>>>>>>> ??? ? separator = .
>>>>>>>> ??? ? type = private
>>>>>>>> }
>>>>>>>> passdb {
>>>>>>>> ??? ? args = /etc/dovecot/dovecot-ldap.conf
>>>>>>>> ??? ? driver = ldap
>>>>>>>> }
>>>>>>>> plugin {
>>>>>>>> ??? ? quota = maildir:Quota Utilisateur
>>>>>>>> ??? ? quota_exceeded_message = Cet utilisateur
a d?pass? son
>>>>>>>> quota, votre
>>>>>>>> message n'a pu lui ?tre livr?.
>>>>>>>> ??? ? quota_full_tempfail = yes
>>>>>>>> ??? ? quota_rule = *:storage=1000M
>>>>>>>> ??? ? quota_rule2 = INBOX.Trash:storage=+100M
>>>>>>>> ??? ? sieve = ~/dovecot.sieve
>>>>>>>> ??? ? sieve_dir = ~/sieve
>>>>>>>> ??? ? sieve_extensions = -vacation
>>>>>>>> ??? ? sieve_global_dir =
/var/lib/dovecot/sieve/global/
>>>>>>>> ??? ? sieve_max_redirects = 1
>>>>>>>> ??? ? zlib_save = gz
>>>>>>>> ??? ? zlib_save_level = 6
>>>>>>>> }
>>>>>>>> postmaster_address = postmaster@<snip>
>>>>>>>> protocols = imap pop3 sieve
>>>>>>>> replication_max_conns = 50
>>>>>>>> service auth {
>>>>>>>> ??? ? client_limit = 49452
>>>>>>>> ??? ? unix_listener auth-userdb {
>>>>>>>> ??? ??? group = vmail
>>>>>>>> ??? ??? mode = 0600
>>>>>>>> ??? ??? user = vmail
>>>>>>>> ??? ? }
>>>>>>>> ??? ? user = root
>>>>>>>> }
>>>>>>>> service dict {
>>>>>>>> ??? ? unix_listener dict {
>>>>>>>> ??? ??? mode = 0600
>>>>>>>> ??? ??? user = vmail
>>>>>>>> ??? ? }
>>>>>>>> }
>>>>>>>> service doveadm {
>>>>>>>> ??? ? inet_listener {
>>>>>>>> ??? ??? port = 12345
>>>>>>>> ??? ? }
>>>>>>>> }
>>>>>>>> service imap-login {
>>>>>>>> ??? ? process_min_avail = 8
>>>>>>>> ??? ? service_count = 0
>>>>>>>> ??? ? user = mail
>>>>>>>> }
>>>>>>>> service imap {
>>>>>>>> ??? ? executable = imap
>>>>>>>> ??? ? process_limit = 16384
>>>>>>>> ??? ? vsz_limit = 2 G
>>>>>>>> }
>>>>>>>> service managesieve-login {
>>>>>>>> ??? ? inet_listener sieve {
>>>>>>>> ??? ??? port = 4190
>>>>>>>> ??? ? }
>>>>>>>> ??? ? process_min_avail = 8
>>>>>>>> ??? ? service_count = 0
>>>>>>>> ??? ? user = mail
>>>>>>>> ??? ? vsz_limit = 2 G
>>>>>>>> }
>>>>>>>> service managesieve {
>>>>>>>> ??? ? drop_priv_before_exec = yes
>>>>>>>> ??? ? process_limit = 16384
>>>>>>>> }
>>>>>>>> service pop3-login {
>>>>>>>> ??? ? process_min_avail = 8
>>>>>>>> ??? ? service_count = 0
>>>>>>>> ??? ? user = mail
>>>>>>>> }
>>>>>>>> service pop3 {
>>>>>>>> ??? ? drop_priv_before_exec = yes
>>>>>>>> ??? ? process_limit = 16384
>>>>>>>> }
>>>>>>>> ssl = no
>>>>>>>> userdb {
>>>>>>>> ??? ? args = /etc/dovecot/dovecot-ldap.conf
>>>>>>>> ??? ? driver = ldap
>>>>>>>> }
>>>>>>>> verbose_proctitle = yes
>>>>>>>> protocol imap {
>>>>>>>> ??? ? mail_max_userip_connections = 50
>>>>>>>> ??? ? mail_plugins = quota zlib notify
replication imap_quota zlib
>>>>>>>> }
>>>>>>>> protocol pop3 {
>>>>>>>> ??? ? mail_plugins = quota zlib notify
replication
>>>>>>>> }
>>>>>>>> protocol sieve {
>>>>>>>> ??? ? mail_max_userip_connections = 10
>>>>>>>> }
>>>>>>>> protocol lda {
>>>>>>>> ??? ? mail_plugins = quota zlib notify
replication sieve zlib
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Arnaud Ab?lard
>>>>>>> Responsable p?le Syst?me et Stockage
>>>>>>> Service Infrastructures
>>>>>>> DSIN Universit? de Nantes
>>>>>>> -
>>>>>
>>>>> --
>>>>> Arnaud Ab?lard
>>>>> Responsable p?le Syst?me et Stockage
>>>>> Service Infrastructures
>>>>> DSIN Universit? de Nantes
>>>>> -
>>>
>>> --
>>> Arnaud Ab?lard
>>> Responsable p?le Syst?me et Stockage
>>> Service Infrastructures
>>> DSIN Universit? de Nantes
>>> -
>
--
Arnaud Ab?lard
Responsable p?le Syst?me et Stockage
Service Infrastructures
DSIN Universit? de Nantes
-