dovecot - Jul 2018 - Inbox quota usage doubled when mailbox_list_index enabled, under some circumstances

Hello,

I searched through the list archives for anything that appeared to be 
similar to this but I didn't find any good matches.? I apologize if this 
has been brought up before.

Beginning with Dovecot 2.2.34, reported quota usage of a user's inbox 
can be doubled when the following criteria are met:

1) quota plugin is enabled
2) mailbox_list_index=yes
3) A sub-folder of the inbox exists whose name also matches the prefix 
(for example, "INBOX.Test"), even if that folder has no contents.? 
(Apparently, some mail clients such as K-9 Mail on Android can create 
these folders automatically)

Downgrading to 2.2.33.2 allows this configuration to work normally. I 
looked through the changes in git between 2.2.33.2 and 2.2.34 and they 
appeared extensive, so unfortunately I wasn't able to pinpoint a 
specific change that caused this.

Reproduced on a CentOS 7 server with Dovecot 2.2.34 built from git 
sources according to 
https://wiki2.dovecot.org/CompilingSource#Compiling_Dovecot_From_Git 
using the following minimal configuration:

=========root at centos7 [~]# dovecot -n
# 2.2.34 (874deae): /usr/local/etc/dovecot/dovecot.conf
# OS: Linux 3.10.0-862.9.1.el7.x86_64 x86_64 CentOS Linux release 
7.5.1804 (Core)
# Hostname: centos7.local
mail_location = maildir:~/Maildir
mail_plugins = quota
mailbox_list_index = yes
namespace inbox {
 ? inbox = yes
 ? location  ? prefix = INBOX.
 ? separator = .
}
passdb {
 ? driver = pam
}
plugin {
 ? quota = maildir:Mailbox
}
userdb {
 ? driver = passwd
}
=========
With the system user "myuser" and a minimal Maildir setup containing 1
email in the inbox, and one sub-folder named "INBOX.Test":

=========root at centos7 [~]# ls -lAR /home/myuser/Maildir
/home/myuser/Maildir:
total 4
drwxr-xr-x. 2 myuser myuser 77 Jul 25 09:36 cur
drwxr-xr-x. 5 myuser myuser 56 Jul 25 15:19 .INBOX.Test
-rw-r--r--. 1 myuser myuser 14 Jul 25 15:19 maildirsize
drwxr-xr-x. 2 myuser myuser? 6 Jul 25 09:36 new
drwxr-xr-x. 2 myuser myuser? 6 Jul 25 09:36 tmp

/home/myuser/Maildir/cur:
total 12416
-rw-r--r--. 1 myuser myuser 12712627 Jul 25 09:07 
1532529376.M543965P58007.centos7.local,S=12712627,W=12877782:2,S

/home/myuser/Maildir/.INBOX.Test:
total 0
drwxr-xr-x. 2 myuser myuser 6 Jul 25 09:36 cur
-rw-r--r--. 1 myuser myuser 0 Jul 25 09:36 maildirfolder
drwxr-xr-x. 2 myuser myuser 6 Jul 25 09:36 new
drwxr-xr-x. 2 myuser myuser 6 Jul 25 09:36 tmp

/home/myuser/Maildir/.INBOX.Test/cur:
total 0

/home/myuser/Maildir/.INBOX.Test/new:
total 0

/home/myuser/Maildir/.INBOX.Test/tmp:
total 0

/home/myuser/Maildir/new:
total 0

/home/myuser/Maildir/tmp:
total 0
=========
Given this configuration, when "mailbox_list_index" is enabled, after 
quota recalculation, quota reports that there are 2 messages and that 
double the amount of storage is used:

=========root at centos7 [~]# doveadm quota recalc -u myuser; doveadm quota get
-u
myuser
Quota name Type??? Value 
Limit?????????????????????????????????????????????????? %
Mailbox??? STORAGE 24830 
-?????????????????????????????????????????????????? 0
Mailbox??? MESSAGE???? 2 
-?????????????????????????????????????????????????? 0
=========
When "mailbox_list_index" is disabled and quota is recalculated, it
will
report the message and storage counts correctly:

=========root at centos7 [~]# doveadm -o 'mailbox_list_index=no' quota
recalc -u
myuser; doveadm quota get -u myuser
Quota name Type??? Value 
Limit?????????????????????????????????????????????????? %
Mailbox??? STORAGE 12415 
-?????????????????????????????????????????????????? 0
Mailbox??? MESSAGE???? 1 
-?????????????????????????????????????????????????? 0
=========
I am hoping that someone with much more familiarity with the code path 
involved here could take a look at this issue and possibly submit a fix 
for it -- that would be greatly appreciated.? Please let me know if you 
need any more details regarding the replication steps, though I hope 
that I didn't miss anything.

Thank you!

-- 

Chris Dillon
Technical Analyst III
Migrations Specialist
cPanel, Inc.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3998 bytes
Desc: S/MIME Cryptographic Signature
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20180726/e44f2235/attachment.p7s>

I failed to mention previously that the behavior also exists in latest 
2.2.36 built from git sources, and I believe also exists in 2.3.2.? 
Though, I wasn't able to successfully build release-2.3.2 or master-2.3 
branches from git in the minimal test environment due to an odd 
compilation error that I have not yet attempted to resolve, but I used 
an internal binary build of 2.3.2 which has a few (though probably not 
affecting this particular issue) patches applied to it.

-- 

Chris Dillon
Technical Analyst III
Migrations Specialist
cPanel, Inc.

Register Now for cPanel Conference 2018
Oct 1 - 3, 2018, Houston, Texas
https://conference.cpanel.com


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3998 bytes
Desc: S/MIME Cryptographic Signature
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20180726/dc9d35fe/attachment.p7s>

I?ve had the opportunity to test the same configuration with a fresh build of
the git master branch (2.4.devel) and the issue also occurs there.  I see that
"mailbox_list_index = yes" is now enabled by default.  It can still be
disabled via "mailbox_list_index = no" which allows the quota to be
calculated correctly.

=========root at ubuntu1804:~# dovecot -n
# 2.4.devel (44282aeeb): /usr/local/etc/dovecot/dovecot.conf
# OS: Linux 4.15.0-30-generic x86_64 Ubuntu 18.04.1 LTS 
# Hostname: ubuntu1804
mail_location = maildir:~/Maildir
mail_plugins = quota
namespace inbox {
  inbox = yes
  location = 
  prefix = INBOX.
  separator = .
}
passdb {
  driver = pam
}
plugin {
  quota = maildir:Mailbox
}
userdb {
  driver = passwd
}
=========
(To summarize from my previous message -- other than "mailbox_list_index =
yes", second most important part of replication is that there is at least
one email in the real inbox and at least one sub-folder named "INBOX"
in maildir format)

root at ubuntu1804:~# ls -ld
/home/myuser/Maildir/cur/1532529376.M543965P58007.centos7.local\,S\=12712627\,W\=12877782\:2\,S
/home/myuser/Maildir/.INBOX.Test/
-rw-rw-r-- 1 myuser myuser 12712627 Aug 14 18:28
'/home/myuser/Maildir/cur/1532529376.M543965P58007.centos7.local,S=12712627,W=12877782:2,S'
drwxrwxr-x 5 myuser myuser       87 Aug 14 18:56 
/home/myuser/Maildir/.INBOX.Test/
========
(In the following example usage is doubled, there is only one email)

root at ubuntu1804:~# doveadm quota recalc -u myuser; doveadm quota get -u
myuser
Quota name Type    Value Limit                                                  
%
Mailbox    STORAGE 24830     -                                                  
0
Mailbox    MESSAGE     2     -                                                  
0
=========
(In the following example it works correctly with mailbox_list_index disabled)

root at ubuntu1804:~# doveadm -o 'mailbox_list_index=no' quota recalc -u
myuser; doveadm quota get -u myuser
Quota name Type    Value Limit                                                  
%
Mailbox    STORAGE 12415     -                                                  
0
Mailbox    MESSAGE     1     -                                                  
0
=========
Best Regards

On Tue, Aug 14, 2018 at 12:31 PM Chris Dillon <chris.dillon at cpanel.net>
wrote:
> I?ve had the opportunity to test the same configuration with a fresh build
> of the git master branch (2.4.devel) and the issue also occurs there.  I
> see that "mailbox_list_index = yes" is now enabled by default. 
It can
> still be disabled via "mailbox_list_index = no" which allows the
quota to
> be calculated correctly.
>
> =========> root at ubuntu1804:~# dovecot -n
> # 2.4.devel (44282aeeb): /usr/local/etc/dovecot/dovecot.conf
> # OS: Linux 4.15.0-30-generic x86_64 Ubuntu 18.04.1 LTS
> # Hostname: ubuntu1804
> mail_location = maildir:~/Maildir
> mail_plugins = quota
> namespace inbox {
>   inbox = yes
>   location >   prefix = INBOX.
>   separator = .
> }
> passdb {
>   driver = pam
> }
> plugin {
>   quota = maildir:Mailbox
> }
> userdb {
>   driver = passwd
> }
> =========>
> (To summarize from my previous message -- other than
"mailbox_list_index > yes", second most important part of
replication is that there is at least
> one email in the real inbox and at least one sub-folder named
"INBOX" in
> maildir format)
>
> root at ubuntu1804:~# ls -ld
>
/home/myuser/Maildir/cur/1532529376.M543965P58007.centos7.local\,S\=12712627\,W\=12877782\:2\,S
> /home/myuser/Maildir/.INBOX.Test/
> -rw-rw-r-- 1 myuser myuser 12712627 Aug 14 18:28
>
'/home/myuser/Maildir/cur/1532529376.M543965P58007.centos7.local,S=12712627,W=12877782:2,S'
> drwxrwxr-x 5 myuser myuser       87 Aug 14 18:56
> /home/myuser/Maildir/.INBOX.Test/
> ========>
> (In the following example usage is doubled, there is only one email)
>
> root at ubuntu1804:~# doveadm quota recalc -u myuser; doveadm quota get -u
> myuser
> Quota name Type    Value Limit
>                   %
> Mailbox    STORAGE 24830     -
>                   0
> Mailbox    MESSAGE     2     -
>                   0
> =========>
> (In the following example it works correctly with mailbox_list_index
> disabled)
>
> root at ubuntu1804:~# doveadm -o 'mailbox_list_index=no' quota
recalc -u
> myuser; doveadm quota get -u myuser
> Quota name Type    Value Limit
>                   %
> Mailbox    STORAGE 12415     -
>                   0
> Mailbox    MESSAGE     1     -
>                   0
> =========>
> Best Regards



We recently upgraded from 2.2.32 to 2.2.36.1 and ran into the same issue as
above. I was about to start compiling all the intermediate versions to
pinpoint where it started and happened upon this post first. We are in the
same boat where some users have *sub*folders starting with INBOX, resulting
in directory names like ".INBOX.Event" (and our namespace root is
INBOX)
and quota calculation double-counts INBOX's vsize.

Just to be clear too: At least in my case, it's *not* double counting,
e.g., INBOX.Event. But if the above conditions are met, it's
double-counting *INBOX*, because it now sees a folder called INBOX.INBOX
(which does *not* exist on the filesystem).

I hadn't gotten as far as Chris did (this just bubbled up today), but his
solution works here too, i.e. passing -o 'mailbox_list_index=no'  to
doveadm quota recalc.

Also, if I rename the directories to something else (e.g. from above,
rename ".INBOX.Event" to ".notINBOX.Event"), a quota recalc
works just
fine. The presence of a directory called .INBOX.<something> is triggering
this.

I'm able to create new subfolders called INBOX.<something> with
clients
like Apple Mail, which was a bit surprising (Apple Mail however choked when
I tried to just create a subfolder called 'INBOX'). We've got
millions of
mailboxes, so educating users is a non-starter :)

Any fix for this from the dovecot devs?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20190312/4cadcf11/attachment-0001.html>