Mark Moseley
2019-Mar-20 21:13 UTC
[BUG?] Double quota calulation when special folder is present
Just hoping to get some dev eyes on this. I'm incredibly reluctant to throw the word 'bug' around (since 99 times out of 100, it's not -- it's almost always the config), but I can't think of any way that this could be a config issue, esp when the pre-2.2.34 version works as expected. I noticed during troubleshooting that dovecot errors out if I try to create a subfolder called 'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse (i.e. a folder called INBOX.INBOX.SomethingElse - resulting in a directory called .INBOX.SomethingElse on the filesystem, and leading to the problem described below). Is that sub-subfolder creation (where the top level subfolder matches the namespace name) supposed to be allowed? It seems odd that 'INBOX' (as a subfolder of INBOX) would be blocked but INBOX.SomethingElse (as a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse (i.e. INBOX.INBOX.SomethingElse) would be blocked as well. On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot <dovecot at dovecot.org> wrote:> Hello, > > we're operating dovecot on a small server. Some years ago, we migrated > from courier IMAP to dovecot. Therefore, we defined our default > Namespace "inbox" with prefix "INBOX." to have this compatible. I found > this in some migration docs those days. Generally, everything worked as > expected. > > Our only namespace is configured like this: > > namespace inbox { > separator = . > prefix = INBOX. > inbox = yes > } > > Regularly, there is no folder named INBOX or .INBOX in the file system, > I suppose this is correct. But I found a special corner case today when > it comes to quota calculation. > > When - for whatever reason - a folder .INBOX.foo (for arbitrary values > of foo) exists, the whole mailbox is counted twice in quota > recalculation. Just creating .INBOX does nothing but a subfolder > triggers the problem. > > This is my shell view (replaced username and file path and deleted > unnecessary debug output) > > $ cat maildirsize > 268435456S > 14697 17 > $ maildirmake .INBOX.foo > $ sudo doveadm -D quota recalc -u <user> > [...] > doveadm(<user>): Debug: Namespace inbox: type=private, prefix=INBOX., > sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes > location=maildir:/home/.../test > doveadm(<user>): Debug: maildir++: root=/home/.../test, index=, > indexpvt=, control=, inbox=/home/.../test, alt> doveadm(<user>): Debug: Namespace : type=private, prefix=, sep=, > inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none > doveadm(<user>): Debug: none: root=, index=, indexpvt=, control=, > inbox=, alt> doveadm(<user>): Debug: quota: quota_over_flag check: quota_over_script > unset - skipping > doveadm(<user>): Debug: Quota root User quota: Recalculated relative > rules with bytes=268435456 count=0. Now grace=26843545 > doveadm(<user>): Debug: Namespace INBOX.: Using permissions from > /home/.../test: mode=0700 gid=default > > $ cat maildirsize > 268435456S > 29394 34 > > > So the used quota has exactly been doubled by just creating an empty > subfolder. > > Do you have any pointers for fixing my configuration or is this a bug in > dovecot? > >I coincidentally resurrected a months-old thread with this same issue a few days ago. I'm seeing the exact same after upgrading from 2.2.32 to 2.2.36. The original poster (who also narrowed it down to something in 2.2.34) mentioned a workaround that does indeed work, namely setting mailbox_list_index=no:> doveadm -o 'mailbox_list_index=no' quota recalc -u myuserI've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c or src/lib-storage/list/mailbox-list-fs-iter.c or src/lib-storage/list/mailbox-list-index-iter.c or src/lib-storage/list/mailbox-list-index.c? The latter few have some added strcmp's against "INBOX". Then again, there's a lot of new code in the diffs under src/lib-storage that references INBOX specifically. -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20190320/1a7600dd/attachment-0001.html>
Mark Moseley
2019-Apr-04 04:37 UTC
[BUG?] Double quota calulation when special folder is present
On Wed, Mar 20, 2019 at 2:13 PM Mark Moseley <moseleymark at gmail.com> wrote:> Just hoping to get some dev eyes on this. I'm incredibly reluctant to > throw the word 'bug' around > (since 99 times out of 100, it's not -- it's almost always the config), > but I can't think of any way > that this could be a config issue, esp when the pre-2.2.34 version works > as expected. > > I noticed during troubleshooting that dovecot errors out if I try to > create a subfolder called > 'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse > (i.e. a folder called > INBOX.INBOX.SomethingElse - resulting in a directory called > .INBOX.SomethingElse on the > filesystem, and leading to the problem described below). Is that > sub-subfolder creation (where > the top level subfolder matches the namespace name) supposed to be > allowed? It seems > odd that 'INBOX' (as a subfolder of INBOX) would be blocked but > INBOX.SomethingElse (as > a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse > (i.e. > INBOX.INBOX.SomethingElse) would be blocked as well. > > > On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot < > dovecot at dovecot.org> wrote: > >> Hello, >> >> we're operating dovecot on a small server. Some years ago, we migrated >> from courier IMAP to dovecot. Therefore, we defined our default >> Namespace "inbox" with prefix "INBOX." to have this compatible. I found >> this in some migration docs those days. Generally, everything worked as >> expected. >> >> Our only namespace is configured like this: >> >> namespace inbox { >> separator = . >> prefix = INBOX. >> inbox = yes >> } >> >> Regularly, there is no folder named INBOX or .INBOX in the file system, >> I suppose this is correct. But I found a special corner case today when >> it comes to quota calculation. >> >> When - for whatever reason - a folder .INBOX.foo (for arbitrary values >> of foo) exists, the whole mailbox is counted twice in quota >> recalculation. Just creating .INBOX does nothing but a subfolder >> triggers the problem. >> >> This is my shell view (replaced username and file path and deleted >> unnecessary debug output) >> >> $ cat maildirsize >> 268435456S >> 14697 17 >> $ maildirmake .INBOX.foo >> $ sudo doveadm -D quota recalc -u <user> >> [...] >> doveadm(<user>): Debug: Namespace inbox: type=private, prefix=INBOX., >> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes >> location=maildir:/home/.../test >> doveadm(<user>): Debug: maildir++: root=/home/.../test, index=, >> indexpvt=, control=, inbox=/home/.../test, alt>> doveadm(<user>): Debug: Namespace : type=private, prefix=, sep=, >> inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none >> doveadm(<user>): Debug: none: root=, index=, indexpvt=, control=, >> inbox=, alt>> doveadm(<user>): Debug: quota: quota_over_flag check: quota_over_script >> unset - skipping >> doveadm(<user>): Debug: Quota root User quota: Recalculated relative >> rules with bytes=268435456 count=0. Now grace=26843545 >> doveadm(<user>): Debug: Namespace INBOX.: Using permissions from >> /home/.../test: mode=0700 gid=default >> >> $ cat maildirsize >> 268435456S >> 29394 34 >> >> >> So the used quota has exactly been doubled by just creating an empty >> subfolder. >> >> Do you have any pointers for fixing my configuration or is this a bug in >> dovecot? >> >> > I coincidentally resurrected a months-old thread with this same issue a > few days ago. I'm seeing the exact same after upgrading from 2.2.32 to > 2.2.36. > > The original poster (who also narrowed it down to something in 2.2.34) > mentioned a workaround that does indeed work, namely setting > mailbox_list_index=no: > > > doveadm -o 'mailbox_list_index=no' quota recalc -u myuser > > I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping > out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c > or src/lib-storage/list/mailbox-list-fs-iter.c or > src/lib-storage/list/mailbox-list-index-iter.c > or src/lib-storage/list/mailbox-list-index.c? > > The latter few have some added strcmp's against "INBOX". Then again, > there's a lot of new code in the diffs under src/lib-storage that > references INBOX specifically. >Can the Dovecot team confirm whether this is indeed a bug or not? I've not yet been able to test 2.3.x to see if the problem exists there as well. -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20190403/5021da1e/attachment.html>
Mark Moseley
2019-Apr-10 02:00 UTC
[BUG?] Double quota calulation when special folder is present
On Wed, Apr 3, 2019 at 9:37 PM Mark Moseley <moseleymark at gmail.com> wrote:> > On Wed, Mar 20, 2019 at 2:13 PM Mark Moseley <moseleymark at gmail.com> > wrote: > >> Just hoping to get some dev eyes on this. I'm incredibly reluctant to >> throw the word 'bug' around >> (since 99 times out of 100, it's not -- it's almost always the config), >> but I can't think of any way >> that this could be a config issue, esp when the pre-2.2.34 version works >> as expected. >> >> I noticed during troubleshooting that dovecot errors out if I try to >> create a subfolder called >> 'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse >> (i.e. a folder called >> INBOX.INBOX.SomethingElse - resulting in a directory called >> .INBOX.SomethingElse on the >> filesystem, and leading to the problem described below). Is that >> sub-subfolder creation (where >> the top level subfolder matches the namespace name) supposed to be >> allowed? It seems >> odd that 'INBOX' (as a subfolder of INBOX) would be blocked but >> INBOX.SomethingElse (as >> a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse >> (i.e. >> INBOX.INBOX.SomethingElse) would be blocked as well. >> >> >> On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot < >> dovecot at dovecot.org> wrote: >> >>> Hello, >>> >>> we're operating dovecot on a small server. Some years ago, we migrated >>> from courier IMAP to dovecot. Therefore, we defined our default >>> Namespace "inbox" with prefix "INBOX." to have this compatible. I found >>> this in some migration docs those days. Generally, everything worked as >>> expected. >>> >>> Our only namespace is configured like this: >>> >>> namespace inbox { >>> separator = . >>> prefix = INBOX. >>> inbox = yes >>> } >>> >>> Regularly, there is no folder named INBOX or .INBOX in the file system, >>> I suppose this is correct. But I found a special corner case today when >>> it comes to quota calculation. >>> >>> When - for whatever reason - a folder .INBOX.foo (for arbitrary values >>> of foo) exists, the whole mailbox is counted twice in quota >>> recalculation. Just creating .INBOX does nothing but a subfolder >>> triggers the problem. >>> >>> This is my shell view (replaced username and file path and deleted >>> unnecessary debug output) >>> >>> $ cat maildirsize >>> 268435456S >>> 14697 17 >>> $ maildirmake .INBOX.foo >>> $ sudo doveadm -D quota recalc -u <user> >>> [...] >>> doveadm(<user>): Debug: Namespace inbox: type=private, prefix=INBOX., >>> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes >>> location=maildir:/home/.../test >>> doveadm(<user>): Debug: maildir++: root=/home/.../test, index=, >>> indexpvt=, control=, inbox=/home/.../test, alt>>> doveadm(<user>): Debug: Namespace : type=private, prefix=, sep=, >>> inbox=no, hidden=yes, list=no, subscriptions=no >>> location=fail::LAYOUT=none >>> doveadm(<user>): Debug: none: root=, index=, indexpvt=, control=, >>> inbox=, alt>>> doveadm(<user>): Debug: quota: quota_over_flag check: quota_over_script >>> unset - skipping >>> doveadm(<user>): Debug: Quota root User quota: Recalculated relative >>> rules with bytes=268435456 count=0. Now grace=26843545 >>> doveadm(<user>): Debug: Namespace INBOX.: Using permissions from >>> /home/.../test: mode=0700 gid=default >>> >>> $ cat maildirsize >>> 268435456S >>> 29394 34 >>> >>> >>> So the used quota has exactly been doubled by just creating an empty >>> subfolder. >>> >>> Do you have any pointers for fixing my configuration or is this a bug in >>> dovecot? >>> >>> >> I coincidentally resurrected a months-old thread with this same issue a >> few days ago. I'm seeing the exact same after upgrading from 2.2.32 to >> 2.2.36. >> >> The original poster (who also narrowed it down to something in 2.2.34) >> mentioned a workaround that does indeed work, namely setting >> mailbox_list_index=no: >> >> > doveadm -o 'mailbox_list_index=no' quota recalc -u myuser >> >> I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping >> out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c >> or src/lib-storage/list/mailbox-list-fs-iter.c or >> src/lib-storage/list/mailbox-list-index-iter.c >> or src/lib-storage/list/mailbox-list-index.c? >> >> The latter few have some added strcmp's against "INBOX". Then again, >> there's a lot of new code in the diffs under src/lib-storage that >> references INBOX specifically. >> > > Can the Dovecot team confirm whether this is indeed a bug or not? I've > not yet been able to test 2.3.x to see if the problem exists there as well. >I've bisected this down to this commit: git diff 7620195ceeea805137cbd1bae104e385eee474a9..97473a513feb2bbd763051869c8b7b83e24b37fa diff --git a/src/lib-storage/list/mailbox-list-index-iter.c b/src/lib-storage/list/mailbox-list-index-iter.c index c9afc7a..49cd941 100644 --- a/src/lib-storage/list/mailbox-list-index-iter.c +++ b/src/lib-storage/list/mailbox-list-index-iter.c @@ -90,13 +90,18 @@ mailbox_list_index_update_info(struct mailbox_list_index_iterate_context *ctx) if (strcmp(ctx->info.vname, "INBOX") != 0) { /* non-INBOX */ ctx->info.vname = p_strdup(ctx->info_pool, ctx->info.vname); - } else { + } else if (!ctx->prefix_inbox_list) { /* listing INBOX itself */ ctx->info.vname = "INBOX"; if (mail_namespace_is_inbox_noinferiors(ctx->info.ns)) { ctx->info.flags &~(MAILBOX_CHILDREN|MAILBOX_NOCHILDREN); ctx->info.flags |= MAILBOX_NOINFERIORS; } + } else { + /* listing INBOX/INBOX */ + ctx->info.vname = p_strconcat(ctx->info_pool, + ctx->ctx.list->ns->prefix, "INBOX", NULL); + ctx->info.flags |= MAILBOX_NONEXISTENT; } if ((node->flags & MAILBOX_LIST_INDEX_FLAG_NONEXISTENT) != 0) ctx->info.flags |= MAILBOX_NONEXISTENT; @@ -126,6 +131,14 @@ mailbox_list_index_update_next(struct mailbox_list_index_iterate_context *ctx, { struct mailbox_list_index_node *node = ctx->next_node; + if (!ctx->prefix_inbox_list && ctx->ctx.list->ns->prefix_len > 0 && + strcmp(node->name, "INBOX") == 0 && node->parent == NULL && + node->children != NULL) { + /* prefix/INBOX has children */ + ctx->prefix_inbox_list = TRUE; + return; + } + if (node->children != NULL && follow_children) { ctx->parent_len = str_len(ctx->path); ctx->next_node = node->children; diff --git a/src/lib-storage/list/mailbox-list-index.h b/src/lib-storage/list/mailbox-list-index.h index 9dc1c48..e4b042a 100644 --- a/src/lib-storage/list/mailbox-list-index.h +++ b/src/lib-storage/list/mailbox-list-index.h @@ -138,6 +138,7 @@ struct mailbox_list_index_iterate_context { struct mailbox_list_index_node *next_node; unsigned int failed:1; + unsigned int prefix_inbox_list:1; }; extern MODULE_CONTEXT_DEFINE(mailbox_list_index_module, Prior to this commit, anything updating the quota would do the right thing for any .INBOX.<something> folders (i.e. not double count the contents of "INBOX" against the quota). After this commit, anything updating quota (new mail, quota recalc, etc) does the double counting of INBOX. -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20190409/65a7cc6c/attachment-0001.html>