On 10/11/2021 15:52, Arnaud Ab?lard wrote:> I've just removed dummy-c-1's mailbox completely on the target
server,
> removed the quota plugin, forced a sync for dummy-c-1 only and once it
> was done reactivated the quota plugins: double usage again.
>
> I also downgraded from dovecot 2.3.17 to 2.2.27 on the target server in
> order to have both server running the same version, without improvement.
Actually, I got that wrong, test server is running dovecot 2.3.4 and
production server dovecot 2.2.7. I copied the dovecot.conf file from the
production server to the test one, then rsync'ed the dummy-c-1 mailbox
from the production server to the test server, restarted dovecot and the
quota is still 115% after recalc. The only difference now is dovecot's
version. Were there any changes related to quota on dovecot 2.3?
Thanks,
Arnaud
>
> I'm puzzled. Is dovecot storing anything outside of the user's
mailbox?
> Like a cache, a sqlite database of some kind somewhere?
>
> Arnaud
>
> On 08/11/2021 11:48, Arnaud Ab?lard wrote:
>> On the target, I enabled the replication service without mail_replica
>> and:
>>
>> doveadm replicator status 'dummy-c-1*'
>> username ??? priority fast sync full sync success sync failed
>> dummy-c-1 ??? none???? -???????? -???????? -??????????? -
>>
>> It only knows of dummy-c-1, no trace of his evil twin
>> dummy-c-1 at univ-nantes.fr.
>>
>> On the target, I do have the same number of files
>>
>> find . -type f |wc -l
>> 8705
>>
>> which is around half of what the quota is reporting (plus de index
>> files, etc):
>>
>> ~# doveadm -D quota get -u dummy-c-1
>> Debug: Loading modules from directory: /usr/lib/dovecot/modules
>> Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so
>> Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so
>> Debug: Module loaded:
>> /usr/lib/dovecot/modules/lib20_replication_plugin.so
>> Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so
>> Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm
>> Debug: Skipping module doveadm_acl_plugin, because dlopen() failed:
>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so:
>> undefined symbol: acl_user_module (this is usually intentional, so
>> just ignore this message)
>> Debug: Skipping module doveadm_expire_plugin, because dlopen() failed:
>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so:
>> undefined symbol: expire_set_deinit (this is usually intentional, so
>> just ignore this message)
>> Debug: Module loaded:
>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so
>> Debug: Module loaded:
>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so
>> Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen()
>> failed:
>> /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so:
>> undefined symbol: lucene_index_iter_deinit (this is usually
>> intentional, so just ignore this message)
>> Debug: Skipping module doveadm_fts_plugin, because dlopen() failed:
>> /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so:
>> undefined symbol: fts_user_get_language_list (this is usually
>> intentional, so just ignore this message)
>> Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen()
>> failed:
>> /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so:
>> undefined symbol: mail_crypt_box_get_pvt_digests (this is usually
>> intentional, so just ignore this message)
>> doveadm(dummy-c-1)<24051><>: Debug: auth USER input:
dummy-c-1
>> home=/vmail/d/u/dummy-c-1/ quota_rule=*:backend=1000000000S
>> doveadm(dummy-c-1)<24051><>: Debug: Added userdb setting:
>> plugin/quota_rule=*:backend=1000000000S
>> doveadm(dummy-c-1): Debug: Effective uid=5000, gid=5000,
>> home=/vmail/d/u/dummy-c-1/
>> doveadm(dummy-c-1): Debug: Quota root: name=Quota Utilisateur
>> backend=maildir args>> doveadm(dummy-c-1): Debug: Quota rule:
root=Quota Utilisateur
>> mailbox=* bytes=1000000000 messages=0
>> doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur
>> mailbox=INBOX.Trash bytes=+104857600 messages=0
>> doveadm(dummy-c-1): Debug: Quota grace: root=Quota Utilisateur
>> bytes=100000000 (10%)
>> doveadm(dummy-c-1): Debug: replication: No mail_replica setting -
>> replication disabled
>> doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=INBOX.,
>> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
>> location=maildir:/vmail/d/u/dummy-c-1/
>> doveadm(dummy-c-1): Debug: maildir++: root=/vmail/d/u/dummy-c-1,
>> index=, indexpvt=, control=, inbox=/vmail/d/u/dummy-c-1, alt>>
doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=, sep=,
>> inbox=no, hidden=yes, list=no, subscriptions=no
>> location=fail::LAYOUT=none
>> doveadm(dummy-c-1): Debug: none: root=, index=, indexpvt=, control=,
>> inbox=, alt>> doveadm(dummy-c-1): Debug: quota: quota_over_flag
check:
>> quota_over_script unset - skipping
>> Quota name??????? Type????? Value? Limit
>> ???????????????????????????????????????????? %
>> Quota Utilisateur STORAGE 1126751 976563
>> ?????????????????????????????????????????? 115
>> Quota Utilisateur MESSAGE?? 16686????? -
>> ???????????????????????????????????????????? 0
>>
>> The debug mode isn't much of any help here.
>>
>> Arnaud
>>
>> On 08/11/2021 11:30, Aki Tuomi wrote:
>>> Recalculation won't fix replication mistakes.
>>>
>>> Did you ensure on the *target* server that it has only dummy-c-1 in
>>> replication and that the file count on the target server matches
>>> source server?
>>>
>>> You might also get something useful out of
>>>
>>> `doveadm -D quota recalc|get -u user`
>>>
>>> Aki
>>>
>>>> On 08/11/2021 12:22 Arnaud Ab?lard <arnaud.abelard at
univ-nantes.fr>
>>>> wrote:
>>>>
>>>> On 08/11/2021 10:39, Aki Tuomi wrote:
>>>>> Try doveadm replicator remove dummy-c-1 at univ-nantes.fr
>>>>
>>>> I did so on the source server, checked indeed that
>>>> dummy-c-1 at univ-nantes.fr wasn't showing up in the
doveadm replicator
>>>> status list and it was indeed the case, so far so good. On the
>>>> destination server, dummy-c-1's mailbox was still showing a
115% quota
>>>> so I tried to make dovecot recalculate the quota but it did not
change
>>>> anything. I ended up deleting the user's mailbox and forced
a
>>>> replication and, to my surprise, the new mailbox still has a
115%
>>>> quota.
>>>>
>>>> Arnaud
>>>>
>>>>>
>>>>> Aki
>>>>>
>>>>>> On 08/11/2021 11:32 Arnaud Ab?lard <arnaud.abelard
at univ-nantes.fr>
>>>>>> wrote:
>>>>>>
>>>>>> Hello Aki,
>>>>>>
>>>>>> Thanks for the tip. I changed postfix configuration so
it will not
>>>>>> use
>>>>>> the domain part anymore and added
"auth_username_format = %Ln"
>>>>>> just in
>>>>>> case (and reloaded dovecot). So I shouldn't have
any more new
>>>>>> user at domain users but for all the existing users,
doveadm replicator
>>>>>> status still show user at domain clone and on the new
server mails are
>>>>>> still counted twice.
>>>>>>
>>>>>> Our userdb and passdb don't have the @domain
defined anywhere so I
>>>>>> don't
>>>>>> think it's coming from there.
>>>>>>
>>>>>> Is there a local cache that still has a list of those
user at domain
>>>>>> usernames that I could reset or something?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Arnaud
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 08/11/2021 09:48, Aki Tuomi wrote:
>>>>>>> This sounds like you are not normalizing usernames
properly.
>>>>>>>
>>>>>>> Either use
>>>>>>>
>>>>>>> auth_username_format = %Ln
>>>>>>>
>>>>>>> as global setting, or return `user` attribute in
both userdb and
>>>>>>> passdb lookups.
>>>>>>>
>>>>>>> Aki
>>>>>>>
>>>>>>>> On 07/11/2021 20:31 Arnaud Ab?lard
>>>>>>>> <arnaud.abelard at univ-nantes.fr> wrote:
>>>>>>>>
>>>>>>>> Hello again,
>>>>>>>>
>>>>>>>> I've found out that some mailboxes? are
actually duplicated.
>>>>>>>> Doveadm
>>>>>>>> replicator status on the production server
gives this:
>>>>>>>>
>>>>>>>> ~# doveadm replicator status
'dummy-c-1*'
>>>>>>>> username????????????????????????????? priority
fast sync full sync
>>>>>>>> success sync failed
>>>>>>>> dummy-c-1???????????????????????????? none????
01:13:19? 01:13:19
>>>>>>>> 01:13:19???? -
>>>>>>>> dummy-c-1 at univ-nantes.fr?????????????
none???? 00:15:28? 00:15:28
>>>>>>>> 00:15:28???? -
>>>>>>>>
>>>>>>>> That'd explain why mails are counted twice
when replicated on
>>>>>>>> the new
>>>>>>>> server but where does this come from since I
don't have this quota
>>>>>>>> problem on the production server?
>>>>>>>>
>>>>>>>> ??? From the logs, it seems that postfix uses
>>>>>>>> username at univ-nantes.fr when
>>>>>>>> calling dovecot's LDA so I suppose that the
non-'@univ-nantes.fr'
>>>>>>>> mailboxes are the erroneous ones.
>>>>>>>>
>>>>>>>> The users aren't duplicated in our ldap
user backend and aren't
>>>>>>>> using
>>>>>>>> the @univ-nantes.fr part.
>>>>>>>>
>>>>>>>> ~# doveadm user 'dummy-c-1*'
>>>>>>>> dummy-c-1
>>>>>>>>
>>>>>>>> Any ideas?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Arnaud
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 05/11/2021 16:21, Arnaud Ab?lard wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> We are very long time happy dovecot users
(since 2008 at
>>>>>>>>> least). We have
>>>>>>>>> around 90k mailboxes and since we had to
move away from our NAS
>>>>>>>>> storage
>>>>>>>>> to a ceph storage I jumped on the
opportunity to enable
>>>>>>>>> compression with
>>>>>>>>> the zlib plugin and dovecot's
replication mecanism. We are using
>>>>>>>>> debian's dovecot 2.2.27 packages on
production and our new
>>>>>>>>> server is
>>>>>>>>> running dovecot's own ce-2.3.17
packages.
>>>>>>>>>
>>>>>>>>> On the production server everything works
fine but on the new
>>>>>>>>> server,
>>>>>>>>> replicated mailboxes' quota is all
wrong:
>>>>>>>>>
>>>>>>>>> on production:
>>>>>>>>> # doveadm quota get -u dummy-c-1
>>>>>>>>> Quota name??????? Type???? Value
>>>>>>>>> Limit???????????????????????????? %
>>>>>>>>> Quota Utilisateur STORAGE 660026
>>>>>>>>> 976563??????????????????????????? 67
>>>>>>>>> Quota Utilisateur MESSAGE?? 8651
-???????????????????????????? 0
>>>>>>>>>
>>>>>>>>> on new server:
>>>>>>>>> doveadm quota get -u dummy-c-1
>>>>>>>>> Quota name??????? Type????? Value
>>>>>>>>> Limit?????????????????????????? %
>>>>>>>>> Quota Utilisateur STORAGE 1125251 976563
115
>>>>>>>>> Quota Utilisateur MESSAGE?? 16646
-?????????????????????????? 0
>>>>>>>>>
>>>>>>>>> If I add all the S flag from the filenames
n both servers I get
>>>>>>>>> exactly
>>>>>>>>> the same usage, which is coherent with the
quota on the
>>>>>>>>> production server:
>>>>>>>>>
>>>>>>>>> # find . -type f | grep 'S=' | awk
-F'S=' '{print $2}' | awk -F','
>>>>>>>>> '{print $1}' | awk -F':'
'{print $1}' | paste -sd+ | bc -l
>>>>>>>>> 675865938
>>>>>>>>>
>>>>>>>>> And I have exactly the same amountof mails
on the two server, the
>>>>>>>>> replication works as expected, no unwanted
duplication of mails
>>>>>>>>> occurs.
>>>>>>>>>
>>>>>>>>> Of course, I've tried to ask dovecot to
recalculate quotas with
>>>>>>>>> doveadm
>>>>>>>>> quota recalc -u <username>, but it
doesn't fix the problem.
>>>>>>>>>
>>>>>>>>> What am I missing?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Arnaud
>>>>>>>>>
>>>>>>>>> PS: Here is my doveconf -n output:
>>>>>>>>>
>>>>>>>>> # 2.3.17 (e2aa53df5b):
/etc/dovecot/dovecot.conf
>>>>>>>>> # Pigeonhole version 0.5.17 (054dddfa)
>>>>>>>>> # OS: Linux 4.19.0-18-amd64 x86_64 Debian
10.11
>>>>>>>>> # Hostname: gromel-test
>>>>>>>>> auth_cache_size = 10 k
>>>>>>>>> auth_verbose = yes
>>>>>>>>> disable_plaintext_auth = no
>>>>>>>>> doveadm_password = # hidden, use -P to show
it
>>>>>>>>> hostname = gromel1.univ-nantes.prive
>>>>>>>>> lda_mailbox_autosubscribe = yes
>>>>>>>>> listen = *
>>>>>>>>> log_timestamp = "%Y-%m-%d %H:%M:%S
"
>>>>>>>>> login_trusted_networks = (...)
>>>>>>>>> mail_gid = 5000
>>>>>>>>> mail_location = maildir:%h
>>>>>>>>> mail_plugins = quota zlib notify
replication
>>>>>>>>> mail_privileged_group = vmail
>>>>>>>>> mail_uid = 5000
>>>>>>>>> maildir_stat_dirs = yes
>>>>>>>>> maildir_very_dirty_syncs = yes
>>>>>>>>> managesieve_notify_capability = mailto
>>>>>>>>> managesieve_sieve_capability = fileinto
reject envelope
>>>>>>>>> encoded-character subaddress
comparator-i;ascii-numeric
>>>>>>>>> relational regex
>>>>>>>>> imap4flags copy include variables body
enotify environment
>>>>>>>>> mailbox date
>>>>>>>>> index ihave duplicate mime foreverypart
extracttext
>>>>>>>>> namespace {
>>>>>>>>> ??? ? inbox = yes
>>>>>>>>> ??? ? location
>>>>>>>>> ??? ? prefix = INBOX.
>>>>>>>>> ??? ? separator = .
>>>>>>>>> ??? ? type = private
>>>>>>>>> }
>>>>>>>>> passdb {
>>>>>>>>> ??? ? args = /etc/dovecot/dovecot-ldap.conf
>>>>>>>>> ??? ? driver = ldap
>>>>>>>>> }
>>>>>>>>> plugin {
>>>>>>>>> ??? ? quota = maildir:Quota Utilisateur
>>>>>>>>> ??? ? quota_exceeded_message = Cet
utilisateur a d?pass? son
>>>>>>>>> quota, votre
>>>>>>>>> message n'a pu lui ?tre livr?.
>>>>>>>>> ??? ? quota_full_tempfail = yes
>>>>>>>>> ??? ? quota_rule = *:storage=1000M
>>>>>>>>> ??? ? quota_rule2 =
INBOX.Trash:storage=+100M
>>>>>>>>> ??? ? sieve = ~/dovecot.sieve
>>>>>>>>> ??? ? sieve_dir = ~/sieve
>>>>>>>>> ??? ? sieve_extensions = -vacation
>>>>>>>>> ??? ? sieve_global_dir =
/var/lib/dovecot/sieve/global/
>>>>>>>>> ??? ? sieve_max_redirects = 1
>>>>>>>>> ??? ? zlib_save = gz
>>>>>>>>> ??? ? zlib_save_level = 6
>>>>>>>>> }
>>>>>>>>> postmaster_address =
postmaster@<snip>
>>>>>>>>> protocols = imap pop3 sieve
>>>>>>>>> replication_max_conns = 50
>>>>>>>>> service auth {
>>>>>>>>> ??? ? client_limit = 49452
>>>>>>>>> ??? ? unix_listener auth-userdb {
>>>>>>>>> ??? ??? group = vmail
>>>>>>>>> ??? ??? mode = 0600
>>>>>>>>> ??? ??? user = vmail
>>>>>>>>> ??? ? }
>>>>>>>>> ??? ? user = root
>>>>>>>>> }
>>>>>>>>> service dict {
>>>>>>>>> ??? ? unix_listener dict {
>>>>>>>>> ??? ??? mode = 0600
>>>>>>>>> ??? ??? user = vmail
>>>>>>>>> ??? ? }
>>>>>>>>> }
>>>>>>>>> service doveadm {
>>>>>>>>> ??? ? inet_listener {
>>>>>>>>> ??? ??? port = 12345
>>>>>>>>> ??? ? }
>>>>>>>>> }
>>>>>>>>> service imap-login {
>>>>>>>>> ??? ? process_min_avail = 8
>>>>>>>>> ??? ? service_count = 0
>>>>>>>>> ??? ? user = mail
>>>>>>>>> }
>>>>>>>>> service imap {
>>>>>>>>> ??? ? executable = imap
>>>>>>>>> ??? ? process_limit = 16384
>>>>>>>>> ??? ? vsz_limit = 2 G
>>>>>>>>> }
>>>>>>>>> service managesieve-login {
>>>>>>>>> ??? ? inet_listener sieve {
>>>>>>>>> ??? ??? port = 4190
>>>>>>>>> ??? ? }
>>>>>>>>> ??? ? process_min_avail = 8
>>>>>>>>> ??? ? service_count = 0
>>>>>>>>> ??? ? user = mail
>>>>>>>>> ??? ? vsz_limit = 2 G
>>>>>>>>> }
>>>>>>>>> service managesieve {
>>>>>>>>> ??? ? drop_priv_before_exec = yes
>>>>>>>>> ??? ? process_limit = 16384
>>>>>>>>> }
>>>>>>>>> service pop3-login {
>>>>>>>>> ??? ? process_min_avail = 8
>>>>>>>>> ??? ? service_count = 0
>>>>>>>>> ??? ? user = mail
>>>>>>>>> }
>>>>>>>>> service pop3 {
>>>>>>>>> ??? ? drop_priv_before_exec = yes
>>>>>>>>> ??? ? process_limit = 16384
>>>>>>>>> }
>>>>>>>>> ssl = no
>>>>>>>>> userdb {
>>>>>>>>> ??? ? args = /etc/dovecot/dovecot-ldap.conf
>>>>>>>>> ??? ? driver = ldap
>>>>>>>>> }
>>>>>>>>> verbose_proctitle = yes
>>>>>>>>> protocol imap {
>>>>>>>>> ??? ? mail_max_userip_connections = 50
>>>>>>>>> ??? ? mail_plugins = quota zlib notify
replication imap_quota zlib
>>>>>>>>> }
>>>>>>>>> protocol pop3 {
>>>>>>>>> ??? ? mail_plugins = quota zlib notify
replication
>>>>>>>>> }
>>>>>>>>> protocol sieve {
>>>>>>>>> ??? ? mail_max_userip_connections = 10
>>>>>>>>> }
>>>>>>>>> protocol lda {
>>>>>>>>> ??? ? mail_plugins = quota zlib notify
replication sieve zlib
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Arnaud Ab?lard
>>>>>>>> Responsable p?le Syst?me et Stockage
>>>>>>>> Service Infrastructures
>>>>>>>> DSIN Universit? de Nantes
>>>>>>>> -
>>>>>>
>>>>>> --
>>>>>> Arnaud Ab?lard
>>>>>> Responsable p?le Syst?me et Stockage
>>>>>> Service Infrastructures
>>>>>> DSIN Universit? de Nantes
>>>>>> -
>>>>
>>>> --
>>>> Arnaud Ab?lard
>>>> Responsable p?le Syst?me et Stockage
>>>> Service Infrastructures
>>>> DSIN Universit? de Nantes
>>>> -
>>
>
--
Arnaud Ab?lard
Responsable p?le Syst?me et Stockage
Service Infrastructures
DSIN Universit? de Nantes
-