On 08/11/2021 10:39, Aki Tuomi wrote:> Try doveadm replicator remove dummy-c-1 at univ-nantes.frI did so on the source server, checked indeed that dummy-c-1 at univ-nantes.fr wasn't showing up in the doveadm replicator status list and it was indeed the case, so far so good. On the destination server, dummy-c-1's mailbox was still showing a 115% quota so I tried to make dovecot recalculate the quota but it did not change anything. I ended up deleting the user's mailbox and forced a replication and, to my surprise, the new mailbox still has a 115% quota. Arnaud> > Aki > >> On 08/11/2021 11:32 Arnaud Ab?lard <arnaud.abelard at univ-nantes.fr> wrote: >> >> >> Hello Aki, >> >> Thanks for the tip. I changed postfix configuration so it will not use >> the domain part anymore and added "auth_username_format = %Ln" just in >> case (and reloaded dovecot). So I shouldn't have any more new >> user at domain users but for all the existing users, doveadm replicator >> status still show user at domain clone and on the new server mails are >> still counted twice. >> >> Our userdb and passdb don't have the @domain defined anywhere so I don't >> think it's coming from there. >> >> Is there a local cache that still has a list of those user at domain >> usernames that I could reset or something? >> >> Thanks, >> >> Arnaud >> >> >> >> On 08/11/2021 09:48, Aki Tuomi wrote: >>> This sounds like you are not normalizing usernames properly. >>> >>> Either use >>> >>> auth_username_format = %Ln >>> >>> as global setting, or return `user` attribute in both userdb and passdb lookups. >>> >>> Aki >>> >>>> On 07/11/2021 20:31 Arnaud Ab?lard <arnaud.abelard at univ-nantes.fr> wrote: >>>> >>>> >>>> Hello again, >>>> >>>> I've found out that some mailboxes are actually duplicated. Doveadm >>>> replicator status on the production server gives this: >>>> >>>> ~# doveadm replicator status 'dummy-c-1*' >>>> username priority fast sync full sync >>>> success sync failed >>>> dummy-c-1 none 01:13:19 01:13:19 >>>> 01:13:19 - >>>> dummy-c-1 at univ-nantes.fr none 00:15:28 00:15:28 >>>> 00:15:28 - >>>> >>>> That'd explain why mails are counted twice when replicated on the new >>>> server but where does this come from since I don't have this quota >>>> problem on the production server? >>>> >>>> From the logs, it seems that postfix uses username at univ-nantes.fr when >>>> calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr' >>>> mailboxes are the erroneous ones. >>>> >>>> The users aren't duplicated in our ldap user backend and aren't using >>>> the @univ-nantes.fr part. >>>> >>>> ~# doveadm user 'dummy-c-1*' >>>> dummy-c-1 >>>> >>>> Any ideas? >>>> >>>> Thanks, >>>> >>>> Arnaud >>>> >>>> >>>> >>>> >>>> >>>> On 05/11/2021 16:21, Arnaud Ab?lard wrote: >>>>> Hello, >>>>> >>>>> We are very long time happy dovecot users (since 2008 at least). We have >>>>> around 90k mailboxes and since we had to move away from our NAS storage >>>>> to a ceph storage I jumped on the opportunity to enable compression with >>>>> the zlib plugin and dovecot's replication mecanism. We are using >>>>> debian's dovecot 2.2.27 packages on production and our new server is >>>>> running dovecot's own ce-2.3.17 packages. >>>>> >>>>> On the production server everything works fine but on the new server, >>>>> replicated mailboxes' quota is all wrong: >>>>> >>>>> on production: >>>>> # doveadm quota get -u dummy-c-1 >>>>> Quota name??????? Type???? Value? Limit???????????????????????????? % >>>>> Quota Utilisateur STORAGE 660026 976563??????????????????????????? 67 >>>>> Quota Utilisateur MESSAGE?? 8651????? -???????????????????????????? 0 >>>>> >>>>> on new server: >>>>> doveadm quota get -u dummy-c-1 >>>>> Quota name??????? Type????? Value? Limit?????????????????????????? % >>>>> Quota Utilisateur STORAGE 1125251 976563???????????????????????? 115 >>>>> Quota Utilisateur MESSAGE?? 16646????? -?????????????????????????? 0 >>>>> >>>>> If I add all the S flag from the filenames n both servers I get exactly >>>>> the same usage, which is coherent with the quota on the production server: >>>>> >>>>> # find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F',' >>>>> '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l >>>>> 675865938 >>>>> >>>>> And I have exactly the same amountof mails on the two server, the >>>>> replication works as expected, no unwanted duplication of mails occurs. >>>>> >>>>> Of course, I've tried to ask dovecot to recalculate quotas with doveadm >>>>> quota recalc -u <username>, but it doesn't fix the problem. >>>>> >>>>> What am I missing? >>>>> >>>>> Thanks, >>>>> >>>>> Arnaud >>>>> >>>>> PS: Here is my doveconf -n output: >>>>> >>>>> # 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf >>>>> # Pigeonhole version 0.5.17 (054dddfa) >>>>> # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11 >>>>> # Hostname: gromel-test >>>>> auth_cache_size = 10 k >>>>> auth_verbose = yes >>>>> disable_plaintext_auth = no >>>>> doveadm_password = # hidden, use -P to show it >>>>> hostname = gromel1.univ-nantes.prive >>>>> lda_mailbox_autosubscribe = yes >>>>> listen = * >>>>> log_timestamp = "%Y-%m-%d %H:%M:%S " >>>>> login_trusted_networks = (...) >>>>> mail_gid = 5000 >>>>> mail_location = maildir:%h >>>>> mail_plugins = quota zlib notify replication >>>>> mail_privileged_group = vmail >>>>> mail_uid = 5000 >>>>> maildir_stat_dirs = yes >>>>> maildir_very_dirty_syncs = yes >>>>> managesieve_notify_capability = mailto >>>>> managesieve_sieve_capability = fileinto reject envelope >>>>> encoded-character subaddress comparator-i;ascii-numeric relational regex >>>>> imap4flags copy include variables body enotify environment mailbox date >>>>> index ihave duplicate mime foreverypart extracttext >>>>> namespace { >>>>> ? inbox = yes >>>>> ? location >>>>> ? prefix = INBOX. >>>>> ? separator = . >>>>> ? type = private >>>>> } >>>>> passdb { >>>>> ? args = /etc/dovecot/dovecot-ldap.conf >>>>> ? driver = ldap >>>>> } >>>>> plugin { >>>>> ? quota = maildir:Quota Utilisateur >>>>> ? quota_exceeded_message = Cet utilisateur a d?pass? son quota, votre >>>>> message n'a pu lui ?tre livr?. >>>>> ? quota_full_tempfail = yes >>>>> ? quota_rule = *:storage=1000M >>>>> ? quota_rule2 = INBOX.Trash:storage=+100M >>>>> ? sieve = ~/dovecot.sieve >>>>> ? sieve_dir = ~/sieve >>>>> ? sieve_extensions = -vacation >>>>> ? sieve_global_dir = /var/lib/dovecot/sieve/global/ >>>>> ? sieve_max_redirects = 1 >>>>> ? zlib_save = gz >>>>> ? zlib_save_level = 6 >>>>> } >>>>> postmaster_address = postmaster@<snip> >>>>> protocols = imap pop3 sieve >>>>> replication_max_conns = 50 >>>>> service auth { >>>>> ? client_limit = 49452 >>>>> ? unix_listener auth-userdb { >>>>> ??? group = vmail >>>>> ??? mode = 0600 >>>>> ??? user = vmail >>>>> ? } >>>>> ? user = root >>>>> } >>>>> service dict { >>>>> ? unix_listener dict { >>>>> ??? mode = 0600 >>>>> ??? user = vmail >>>>> ? } >>>>> } >>>>> service doveadm { >>>>> ? inet_listener { >>>>> ??? port = 12345 >>>>> ? } >>>>> } >>>>> service imap-login { >>>>> ? process_min_avail = 8 >>>>> ? service_count = 0 >>>>> ? user = mail >>>>> } >>>>> service imap { >>>>> ? executable = imap >>>>> ? process_limit = 16384 >>>>> ? vsz_limit = 2 G >>>>> } >>>>> service managesieve-login { >>>>> ? inet_listener sieve { >>>>> ??? port = 4190 >>>>> ? } >>>>> ? process_min_avail = 8 >>>>> ? service_count = 0 >>>>> ? user = mail >>>>> ? vsz_limit = 2 G >>>>> } >>>>> service managesieve { >>>>> ? drop_priv_before_exec = yes >>>>> ? process_limit = 16384 >>>>> } >>>>> service pop3-login { >>>>> ? process_min_avail = 8 >>>>> ? service_count = 0 >>>>> ? user = mail >>>>> } >>>>> service pop3 { >>>>> ? drop_priv_before_exec = yes >>>>> ? process_limit = 16384 >>>>> } >>>>> ssl = no >>>>> userdb { >>>>> ? args = /etc/dovecot/dovecot-ldap.conf >>>>> ? driver = ldap >>>>> } >>>>> verbose_proctitle = yes >>>>> protocol imap { >>>>> ? mail_max_userip_connections = 50 >>>>> ? mail_plugins = quota zlib notify replication imap_quota zlib >>>>> } >>>>> protocol pop3 { >>>>> ? mail_plugins = quota zlib notify replication >>>>> } >>>>> protocol sieve { >>>>> ? mail_max_userip_connections = 10 >>>>> } >>>>> protocol lda { >>>>> ? mail_plugins = quota zlib notify replication sieve zlib >>>>> } >>>>> >>>>> >>>>> >>>>> >>>> >>>> -- >>>> Arnaud Ab?lard >>>> Responsable p?le Syst?me et Stockage >>>> Service Infrastructures >>>> DSIN Universit? de Nantes >>>> - >> >> -- >> Arnaud Ab?lard >> Responsable p?le Syst?me et Stockage >> Service Infrastructures >> DSIN Universit? de Nantes >> --- Arnaud Ab?lard Responsable p?le Syst?me et Stockage Service Infrastructures DSIN Universit? de Nantes -
Recalculation won't fix replication mistakes. Did you ensure on the *target* server that it has only dummy-c-1 in replication and that the file count on the target server matches source server? You might also get something useful out of `doveadm -D quota recalc|get -u user` Aki> On 08/11/2021 12:22 Arnaud Ab?lard <arnaud.abelard at univ-nantes.fr> wrote: > > > On 08/11/2021 10:39, Aki Tuomi wrote: > > Try doveadm replicator remove dummy-c-1 at univ-nantes.fr > > I did so on the source server, checked indeed that > dummy-c-1 at univ-nantes.fr wasn't showing up in the doveadm replicator > status list and it was indeed the case, so far so good. On the > destination server, dummy-c-1's mailbox was still showing a 115% quota > so I tried to make dovecot recalculate the quota but it did not change > anything. I ended up deleting the user's mailbox and forced a > replication and, to my surprise, the new mailbox still has a 115% quota. > > Arnaud > > > > > Aki > > > >> On 08/11/2021 11:32 Arnaud Ab?lard <arnaud.abelard at univ-nantes.fr> wrote: > >> > >> > >> Hello Aki, > >> > >> Thanks for the tip. I changed postfix configuration so it will not use > >> the domain part anymore and added "auth_username_format = %Ln" just in > >> case (and reloaded dovecot). So I shouldn't have any more new > >> user at domain users but for all the existing users, doveadm replicator > >> status still show user at domain clone and on the new server mails are > >> still counted twice. > >> > >> Our userdb and passdb don't have the @domain defined anywhere so I don't > >> think it's coming from there. > >> > >> Is there a local cache that still has a list of those user at domain > >> usernames that I could reset or something? > >> > >> Thanks, > >> > >> Arnaud > >> > >> > >> > >> On 08/11/2021 09:48, Aki Tuomi wrote: > >>> This sounds like you are not normalizing usernames properly. > >>> > >>> Either use > >>> > >>> auth_username_format = %Ln > >>> > >>> as global setting, or return `user` attribute in both userdb and passdb lookups. > >>> > >>> Aki > >>> > >>>> On 07/11/2021 20:31 Arnaud Ab?lard <arnaud.abelard at univ-nantes.fr> wrote: > >>>> > >>>> > >>>> Hello again, > >>>> > >>>> I've found out that some mailboxes are actually duplicated. Doveadm > >>>> replicator status on the production server gives this: > >>>> > >>>> ~# doveadm replicator status 'dummy-c-1*' > >>>> username priority fast sync full sync > >>>> success sync failed > >>>> dummy-c-1 none 01:13:19 01:13:19 > >>>> 01:13:19 - > >>>> dummy-c-1 at univ-nantes.fr none 00:15:28 00:15:28 > >>>> 00:15:28 - > >>>> > >>>> That'd explain why mails are counted twice when replicated on the new > >>>> server but where does this come from since I don't have this quota > >>>> problem on the production server? > >>>> > >>>> From the logs, it seems that postfix uses username at univ-nantes.fr when > >>>> calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr' > >>>> mailboxes are the erroneous ones. > >>>> > >>>> The users aren't duplicated in our ldap user backend and aren't using > >>>> the @univ-nantes.fr part. > >>>> > >>>> ~# doveadm user 'dummy-c-1*' > >>>> dummy-c-1 > >>>> > >>>> Any ideas? > >>>> > >>>> Thanks, > >>>> > >>>> Arnaud > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On 05/11/2021 16:21, Arnaud Ab?lard wrote: > >>>>> Hello, > >>>>> > >>>>> We are very long time happy dovecot users (since 2008 at least). We have > >>>>> around 90k mailboxes and since we had to move away from our NAS storage > >>>>> to a ceph storage I jumped on the opportunity to enable compression with > >>>>> the zlib plugin and dovecot's replication mecanism. We are using > >>>>> debian's dovecot 2.2.27 packages on production and our new server is > >>>>> running dovecot's own ce-2.3.17 packages. > >>>>> > >>>>> On the production server everything works fine but on the new server, > >>>>> replicated mailboxes' quota is all wrong: > >>>>> > >>>>> on production: > >>>>> # doveadm quota get -u dummy-c-1 > >>>>> Quota name??????? Type???? Value? Limit???????????????????????????? % > >>>>> Quota Utilisateur STORAGE 660026 976563??????????????????????????? 67 > >>>>> Quota Utilisateur MESSAGE?? 8651????? -???????????????????????????? 0 > >>>>> > >>>>> on new server: > >>>>> doveadm quota get -u dummy-c-1 > >>>>> Quota name??????? Type????? Value? Limit?????????????????????????? % > >>>>> Quota Utilisateur STORAGE 1125251 976563???????????????????????? 115 > >>>>> Quota Utilisateur MESSAGE?? 16646????? -?????????????????????????? 0 > >>>>> > >>>>> If I add all the S flag from the filenames n both servers I get exactly > >>>>> the same usage, which is coherent with the quota on the production server: > >>>>> > >>>>> # find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F',' > >>>>> '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l > >>>>> 675865938 > >>>>> > >>>>> And I have exactly the same amountof mails on the two server, the > >>>>> replication works as expected, no unwanted duplication of mails occurs. > >>>>> > >>>>> Of course, I've tried to ask dovecot to recalculate quotas with doveadm > >>>>> quota recalc -u <username>, but it doesn't fix the problem. > >>>>> > >>>>> What am I missing? > >>>>> > >>>>> Thanks, > >>>>> > >>>>> Arnaud > >>>>> > >>>>> PS: Here is my doveconf -n output: > >>>>> > >>>>> # 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf > >>>>> # Pigeonhole version 0.5.17 (054dddfa) > >>>>> # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11 > >>>>> # Hostname: gromel-test > >>>>> auth_cache_size = 10 k > >>>>> auth_verbose = yes > >>>>> disable_plaintext_auth = no > >>>>> doveadm_password = # hidden, use -P to show it > >>>>> hostname = gromel1.univ-nantes.prive > >>>>> lda_mailbox_autosubscribe = yes > >>>>> listen = * > >>>>> log_timestamp = "%Y-%m-%d %H:%M:%S " > >>>>> login_trusted_networks = (...) > >>>>> mail_gid = 5000 > >>>>> mail_location = maildir:%h > >>>>> mail_plugins = quota zlib notify replication > >>>>> mail_privileged_group = vmail > >>>>> mail_uid = 5000 > >>>>> maildir_stat_dirs = yes > >>>>> maildir_very_dirty_syncs = yes > >>>>> managesieve_notify_capability = mailto > >>>>> managesieve_sieve_capability = fileinto reject envelope > >>>>> encoded-character subaddress comparator-i;ascii-numeric relational regex > >>>>> imap4flags copy include variables body enotify environment mailbox date > >>>>> index ihave duplicate mime foreverypart extracttext > >>>>> namespace { > >>>>> ? inbox = yes > >>>>> ? location > >>>>> ? prefix = INBOX. > >>>>> ? separator = . > >>>>> ? type = private > >>>>> } > >>>>> passdb { > >>>>> ? args = /etc/dovecot/dovecot-ldap.conf > >>>>> ? driver = ldap > >>>>> } > >>>>> plugin { > >>>>> ? quota = maildir:Quota Utilisateur > >>>>> ? quota_exceeded_message = Cet utilisateur a d?pass? son quota, votre > >>>>> message n'a pu lui ?tre livr?. > >>>>> ? quota_full_tempfail = yes > >>>>> ? quota_rule = *:storage=1000M > >>>>> ? quota_rule2 = INBOX.Trash:storage=+100M > >>>>> ? sieve = ~/dovecot.sieve > >>>>> ? sieve_dir = ~/sieve > >>>>> ? sieve_extensions = -vacation > >>>>> ? sieve_global_dir = /var/lib/dovecot/sieve/global/ > >>>>> ? sieve_max_redirects = 1 > >>>>> ? zlib_save = gz > >>>>> ? zlib_save_level = 6 > >>>>> } > >>>>> postmaster_address = postmaster@<snip> > >>>>> protocols = imap pop3 sieve > >>>>> replication_max_conns = 50 > >>>>> service auth { > >>>>> ? client_limit = 49452 > >>>>> ? unix_listener auth-userdb { > >>>>> ??? group = vmail > >>>>> ??? mode = 0600 > >>>>> ??? user = vmail > >>>>> ? } > >>>>> ? user = root > >>>>> } > >>>>> service dict { > >>>>> ? unix_listener dict { > >>>>> ??? mode = 0600 > >>>>> ??? user = vmail > >>>>> ? } > >>>>> } > >>>>> service doveadm { > >>>>> ? inet_listener { > >>>>> ??? port = 12345 > >>>>> ? } > >>>>> } > >>>>> service imap-login { > >>>>> ? process_min_avail = 8 > >>>>> ? service_count = 0 > >>>>> ? user = mail > >>>>> } > >>>>> service imap { > >>>>> ? executable = imap > >>>>> ? process_limit = 16384 > >>>>> ? vsz_limit = 2 G > >>>>> } > >>>>> service managesieve-login { > >>>>> ? inet_listener sieve { > >>>>> ??? port = 4190 > >>>>> ? } > >>>>> ? process_min_avail = 8 > >>>>> ? service_count = 0 > >>>>> ? user = mail > >>>>> ? vsz_limit = 2 G > >>>>> } > >>>>> service managesieve { > >>>>> ? drop_priv_before_exec = yes > >>>>> ? process_limit = 16384 > >>>>> } > >>>>> service pop3-login { > >>>>> ? process_min_avail = 8 > >>>>> ? service_count = 0 > >>>>> ? user = mail > >>>>> } > >>>>> service pop3 { > >>>>> ? drop_priv_before_exec = yes > >>>>> ? process_limit = 16384 > >>>>> } > >>>>> ssl = no > >>>>> userdb { > >>>>> ? args = /etc/dovecot/dovecot-ldap.conf > >>>>> ? driver = ldap > >>>>> } > >>>>> verbose_proctitle = yes > >>>>> protocol imap { > >>>>> ? mail_max_userip_connections = 50 > >>>>> ? mail_plugins = quota zlib notify replication imap_quota zlib > >>>>> } > >>>>> protocol pop3 { > >>>>> ? mail_plugins = quota zlib notify replication > >>>>> } > >>>>> protocol sieve { > >>>>> ? mail_max_userip_connections = 10 > >>>>> } > >>>>> protocol lda { > >>>>> ? mail_plugins = quota zlib notify replication sieve zlib > >>>>> } > >>>>> > >>>>> > >>>>> > >>>>> > >>>> > >>>> -- > >>>> Arnaud Ab?lard > >>>> Responsable p?le Syst?me et Stockage > >>>> Service Infrastructures > >>>> DSIN Universit? de Nantes > >>>> - > >> > >> -- > >> Arnaud Ab?lard > >> Responsable p?le Syst?me et Stockage > >> Service Infrastructures > >> DSIN Universit? de Nantes > >> - > > -- > Arnaud Ab?lard > Responsable p?le Syst?me et Stockage > Service Infrastructures > DSIN Universit? de Nantes > -