dovecot - Mar 2019 - [BUG?] Double quota calulation when special folder is present

Just hoping to get some dev eyes on this. I'm incredibly reluctant to throw
the word 'bug' around
(since 99 times out of 100, it's not -- it's almost always the config),
but
I can't think of any way
that this could be a config issue, esp when the pre-2.2.34 version works as
expected.

I noticed during troubleshooting that dovecot errors out if I try to create
a subfolder called
'INBOX' but it'll happily create a subfolder called
INBOX.SomethingElse
(i.e. a folder called
INBOX.INBOX.SomethingElse - resulting in a directory called
.INBOX.SomethingElse on the
filesystem, and leading to the problem described below). Is that
sub-subfolder creation (where
the top level subfolder matches the namespace name) supposed to be allowed?
It seems
odd that 'INBOX' (as a subfolder of INBOX) would be blocked but
INBOX.SomethingElse (as
a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse
(i.e.
INBOX.INBOX.SomethingElse) would be blocked as well.


On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot <dovecot at
dovecot.org>
wrote:
> Hello,
>
> we're operating dovecot on a small server. Some years ago, we migrated
> from courier IMAP to dovecot. Therefore, we defined our default
> Namespace "inbox" with prefix "INBOX." to have this
compatible. I found
> this in some migration docs those days. Generally, everything worked as
> expected.
>
> Our only namespace is configured like this:
>
> namespace inbox {
>  separator = .
>   prefix = INBOX.
>   inbox = yes
> }
>
> Regularly, there is no folder named INBOX or .INBOX in the file system,
> I suppose this is correct.  But I found a special corner case today when
> it comes to quota calculation.
>
> When - for whatever reason - a folder .INBOX.foo (for arbitrary values
> of foo) exists, the whole mailbox is counted twice in quota
> recalculation. Just creating .INBOX does nothing but a subfolder
> triggers the problem.
>
> This is my shell view (replaced username and file path and deleted
> unnecessary debug output)
>
> $ cat maildirsize
> 268435456S
> 14697 17
> $ maildirmake .INBOX.foo
> $ sudo doveadm -D quota recalc -u <user>
> [...]
> doveadm(<user>): Debug: Namespace inbox: type=private, prefix=INBOX.,
> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
> location=maildir:/home/.../test
> doveadm(<user>): Debug: maildir++: root=/home/.../test, index=,
> indexpvt=, control=, inbox=/home/.../test, alt> doveadm(<user>):
Debug: Namespace : type=private, prefix=, sep=,
> inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none
> doveadm(<user>): Debug: none: root=, index=, indexpvt=, control=,
> inbox=, alt> doveadm(<user>): Debug: quota: quota_over_flag check:
quota_over_script
> unset - skipping
> doveadm(<user>): Debug: Quota root User quota: Recalculated relative
> rules with bytes=268435456 count=0. Now grace=26843545
> doveadm(<user>): Debug: Namespace INBOX.: Using permissions from
> /home/.../test: mode=0700 gid=default
>
> $ cat maildirsize
> 268435456S
> 29394 34
>
>
> So the used quota has exactly been doubled by just creating an empty
> subfolder.
>
> Do you have any pointers for fixing my configuration or is this a bug in
> dovecot?
>
>
I coincidentally resurrected a months-old thread with this same issue a few
days ago. I'm seeing the exact same after upgrading from 2.2.32 to 2.2.36.

The original poster (who also narrowed it down to something in 2.2.34)
mentioned a workaround that does indeed work, namely setting
mailbox_list_index=no:
> doveadm -o 'mailbox_list_index=no' quota recalc -u myuser
I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping
out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c
or src/lib-storage/list/mailbox-list-fs-iter.c or
src/lib-storage/list/mailbox-list-index-iter.c
or src/lib-storage/list/mailbox-list-index.c?

The latter few have some added strcmp's against "INBOX". Then
again,
there's a lot of new code in the diffs under src/lib-storage that
references INBOX specifically.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20190320/1a7600dd/attachment-0001.html>

On Wed, Mar 20, 2019 at 2:13 PM Mark Moseley <moseleymark at gmail.com>
wrote:
> Just hoping to get some dev eyes on this. I'm incredibly reluctant to
> throw the word 'bug' around
> (since 99 times out of 100, it's not -- it's almost always the
config),
> but I can't think of any way
> that this could be a config issue, esp when the pre-2.2.34 version works
> as expected.
>
> I noticed during troubleshooting that dovecot errors out if I try to
> create a subfolder called
> 'INBOX' but it'll happily create a subfolder called
INBOX.SomethingElse
> (i.e. a folder called
> INBOX.INBOX.SomethingElse - resulting in a directory called
> .INBOX.SomethingElse on the
> filesystem, and leading to the problem described below). Is that
> sub-subfolder creation (where
> the top level subfolder matches the namespace name) supposed to be
> allowed? It seems
> odd that 'INBOX' (as a subfolder of INBOX) would be blocked but
> INBOX.SomethingElse (as
> a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse
> (i.e.
> INBOX.INBOX.SomethingElse) would be blocked as well.
>
>
> On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot <
> dovecot at dovecot.org> wrote:
>
>> Hello,
>>
>> we're operating dovecot on a small server. Some years ago, we
migrated
>> from courier IMAP to dovecot. Therefore, we defined our default
>> Namespace "inbox" with prefix "INBOX." to have this
compatible. I found
>> this in some migration docs those days. Generally, everything worked as
>> expected.
>>
>> Our only namespace is configured like this:
>>
>> namespace inbox {
>>  separator = .
>>   prefix = INBOX.
>>   inbox = yes
>> }
>>
>> Regularly, there is no folder named INBOX or .INBOX in the file system,
>> I suppose this is correct.  But I found a special corner case today
when
>> it comes to quota calculation.
>>
>> When - for whatever reason - a folder .INBOX.foo (for arbitrary values
>> of foo) exists, the whole mailbox is counted twice in quota
>> recalculation. Just creating .INBOX does nothing but a subfolder
>> triggers the problem.
>>
>> This is my shell view (replaced username and file path and deleted
>> unnecessary debug output)
>>
>> $ cat maildirsize
>> 268435456S
>> 14697 17
>> $ maildirmake .INBOX.foo
>> $ sudo doveadm -D quota recalc -u <user>
>> [...]
>> doveadm(<user>): Debug: Namespace inbox: type=private,
prefix=INBOX.,
>> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
>> location=maildir:/home/.../test
>> doveadm(<user>): Debug: maildir++: root=/home/.../test, index=,
>> indexpvt=, control=, inbox=/home/.../test, alt>>
doveadm(<user>): Debug: Namespace : type=private, prefix=, sep=,
>> inbox=no, hidden=yes, list=no, subscriptions=no
location=fail::LAYOUT=none
>> doveadm(<user>): Debug: none: root=, index=, indexpvt=, control=,
>> inbox=, alt>> doveadm(<user>): Debug: quota:
quota_over_flag check: quota_over_script
>> unset - skipping
>> doveadm(<user>): Debug: Quota root User quota: Recalculated
relative
>> rules with bytes=268435456 count=0. Now grace=26843545
>> doveadm(<user>): Debug: Namespace INBOX.: Using permissions from
>> /home/.../test: mode=0700 gid=default
>>
>> $ cat maildirsize
>> 268435456S
>> 29394 34
>>
>>
>> So the used quota has exactly been doubled by just creating an empty
>> subfolder.
>>
>> Do you have any pointers for fixing my configuration or is this a bug
in
>> dovecot?
>>
>>
> I coincidentally resurrected a months-old thread with this same issue a
> few days ago. I'm seeing the exact same after upgrading from 2.2.32 to
> 2.2.36.
>
> The original poster (who also narrowed it down to something in 2.2.34)
> mentioned a workaround that does indeed work, namely setting
> mailbox_list_index=no:
>
> > doveadm -o 'mailbox_list_index=no' quota recalc -u myuser
>
> I've been staring at diffs of 2.2.33 and 2.2.34 without anything
jumping
> out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c
> or src/lib-storage/list/mailbox-list-fs-iter.c or
> src/lib-storage/list/mailbox-list-index-iter.c
> or src/lib-storage/list/mailbox-list-index.c?
>
> The latter few have some added strcmp's against "INBOX". Then
again,
> there's a lot of new code in the diffs under src/lib-storage that
> references INBOX specifically.
>
Can the Dovecot team confirm whether this is indeed a bug or not?  I've not
yet been able to test 2.3.x to see if the problem exists there as well.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20190403/5021da1e/attachment.html>

On Wed, Apr 3, 2019 at 9:37 PM Mark Moseley <moseleymark at gmail.com>
wrote:
>
> On Wed, Mar 20, 2019 at 2:13 PM Mark Moseley <moseleymark at
gmail.com>
> wrote:
>
>> Just hoping to get some dev eyes on this. I'm incredibly reluctant
to
>> throw the word 'bug' around
>> (since 99 times out of 100, it's not -- it's almost always the
config),
>> but I can't think of any way
>> that this could be a config issue, esp when the pre-2.2.34 version
works
>> as expected.
>>
>> I noticed during troubleshooting that dovecot errors out if I try to
>> create a subfolder called
>> 'INBOX' but it'll happily create a subfolder called
INBOX.SomethingElse
>> (i.e. a folder called
>> INBOX.INBOX.SomethingElse - resulting in a directory called
>> .INBOX.SomethingElse on the
>> filesystem, and leading to the problem described below). Is that
>> sub-subfolder creation (where
>> the top level subfolder matches the namespace name) supposed to be
>> allowed? It seems
>> odd that 'INBOX' (as a subfolder of INBOX) would be blocked but
>> INBOX.SomethingElse (as
>> a subfolder of INBOX) would be allowed. I'd expect
INBOX.SomethingElse
>> (i.e.
>> INBOX.INBOX.SomethingElse) would be blocked as well.
>>
>>
>> On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot <
>> dovecot at dovecot.org> wrote:
>>
>>> Hello,
>>>
>>> we're operating dovecot on a small server. Some years ago, we
migrated
>>> from courier IMAP to dovecot. Therefore, we defined our default
>>> Namespace "inbox" with prefix "INBOX." to have
this compatible. I found
>>> this in some migration docs those days. Generally, everything
worked as
>>> expected.
>>>
>>> Our only namespace is configured like this:
>>>
>>> namespace inbox {
>>>  separator = .
>>>   prefix = INBOX.
>>>   inbox = yes
>>> }
>>>
>>> Regularly, there is no folder named INBOX or .INBOX in the file
system,
>>> I suppose this is correct.  But I found a special corner case today
when
>>> it comes to quota calculation.
>>>
>>> When - for whatever reason - a folder .INBOX.foo (for arbitrary
values
>>> of foo) exists, the whole mailbox is counted twice in quota
>>> recalculation. Just creating .INBOX does nothing but a subfolder
>>> triggers the problem.
>>>
>>> This is my shell view (replaced username and file path and deleted
>>> unnecessary debug output)
>>>
>>> $ cat maildirsize
>>> 268435456S
>>> 14697 17
>>> $ maildirmake .INBOX.foo
>>> $ sudo doveadm -D quota recalc -u <user>
>>> [...]
>>> doveadm(<user>): Debug: Namespace inbox: type=private,
prefix=INBOX.,
>>> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
>>> location=maildir:/home/.../test
>>> doveadm(<user>): Debug: maildir++: root=/home/.../test,
index=,
>>> indexpvt=, control=, inbox=/home/.../test, alt>>>
doveadm(<user>): Debug: Namespace : type=private, prefix=, sep=,
>>> inbox=no, hidden=yes, list=no, subscriptions=no
>>> location=fail::LAYOUT=none
>>> doveadm(<user>): Debug: none: root=, index=, indexpvt=,
control=,
>>> inbox=, alt>>> doveadm(<user>): Debug: quota:
quota_over_flag check: quota_over_script
>>> unset - skipping
>>> doveadm(<user>): Debug: Quota root User quota: Recalculated
relative
>>> rules with bytes=268435456 count=0. Now grace=26843545
>>> doveadm(<user>): Debug: Namespace INBOX.: Using permissions
from
>>> /home/.../test: mode=0700 gid=default
>>>
>>> $ cat maildirsize
>>> 268435456S
>>> 29394 34
>>>
>>>
>>> So the used quota has exactly been doubled by just creating an
empty
>>> subfolder.
>>>
>>> Do you have any pointers for fixing my configuration or is this a
bug in
>>> dovecot?
>>>
>>>
>> I coincidentally resurrected a months-old thread with this same issue a
>> few days ago. I'm seeing the exact same after upgrading from 2.2.32
to
>> 2.2.36.
>>
>> The original poster (who also narrowed it down to something in 2.2.34)
>> mentioned a workaround that does indeed work, namely setting
>> mailbox_list_index=no:
>>
>> > doveadm -o 'mailbox_list_index=no' quota recalc -u myuser
>>
>> I've been staring at diffs of 2.2.33 and 2.2.34 without anything
jumping
>> out at me (not a C guy, sadly). Maybe
src/lib-storage/index/index-storage.c
>> or src/lib-storage/list/mailbox-list-fs-iter.c or
>> src/lib-storage/list/mailbox-list-index-iter.c
>> or src/lib-storage/list/mailbox-list-index.c?
>>
>> The latter few have some added strcmp's against "INBOX".
Then again,
>> there's a lot of new code in the diffs under src/lib-storage that
>> references INBOX specifically.
>>
>
> Can the Dovecot team confirm whether this is indeed a bug or not?  I've
> not yet been able to test 2.3.x to see if the problem exists there as well.
>
I've bisected this down to this commit:

git diff
7620195ceeea805137cbd1bae104e385eee474a9..97473a513feb2bbd763051869c8b7b83e24b37fa

diff --git a/src/lib-storage/list/mailbox-list-index-iter.c
b/src/lib-storage/list/mailbox-list-index-iter.c
index c9afc7a..49cd941 100644
--- a/src/lib-storage/list/mailbox-list-index-iter.c
+++ b/src/lib-storage/list/mailbox-list-index-iter.c
@@ -90,13 +90,18 @@ mailbox_list_index_update_info(struct
mailbox_list_index_iterate_context *ctx)
        if (strcmp(ctx->info.vname, "INBOX") != 0) {
                /* non-INBOX */
                ctx->info.vname = p_strdup(ctx->info_pool,
ctx->info.vname);
-       } else {
+       } else if (!ctx->prefix_inbox_list) {
                /* listing INBOX itself */
                ctx->info.vname = "INBOX";
                if (mail_namespace_is_inbox_noinferiors(ctx->info.ns)) {
                        ctx->info.flags
&~(MAILBOX_CHILDREN|MAILBOX_NOCHILDREN);
                        ctx->info.flags |= MAILBOX_NOINFERIORS;
                }
+       } else {
+               /* listing INBOX/INBOX */
+               ctx->info.vname = p_strconcat(ctx->info_pool,
+                       ctx->ctx.list->ns->prefix, "INBOX",
NULL);
+               ctx->info.flags |= MAILBOX_NONEXISTENT;
        }
        if ((node->flags & MAILBOX_LIST_INDEX_FLAG_NONEXISTENT) != 0)
                ctx->info.flags |= MAILBOX_NONEXISTENT;
@@ -126,6 +131,14 @@ mailbox_list_index_update_next(struct
mailbox_list_index_iterate_context *ctx,
 {
        struct mailbox_list_index_node *node = ctx->next_node;

+       if (!ctx->prefix_inbox_list &&
ctx->ctx.list->ns->prefix_len > 0 &&
+           strcmp(node->name, "INBOX") == 0 &&
node->parent == NULL &&
+           node->children != NULL) {
+               /* prefix/INBOX has children */
+               ctx->prefix_inbox_list = TRUE;
+               return;
+       }
+
        if (node->children != NULL && follow_children) {
                ctx->parent_len = str_len(ctx->path);
                ctx->next_node = node->children;
diff --git a/src/lib-storage/list/mailbox-list-index.h
b/src/lib-storage/list/mailbox-list-index.h
index 9dc1c48..e4b042a 100644
--- a/src/lib-storage/list/mailbox-list-index.h
+++ b/src/lib-storage/list/mailbox-list-index.h
@@ -138,6 +138,7 @@ struct mailbox_list_index_iterate_context {
        struct mailbox_list_index_node *next_node;

        unsigned int failed:1;
+       unsigned int prefix_inbox_list:1;
 };

 extern MODULE_CONTEXT_DEFINE(mailbox_list_index_module,


Prior to this commit, anything updating the quota would do the right thing
for any .INBOX.<something> folders (i.e. not double count the contents of
"INBOX" against the quota). After this commit, anything updating quota
(new
mail, quota recalc, etc) does the double counting of INBOX.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20190409/65a7cc6c/attachment-0001.html>