dovecot - May 2021 - Dovecot v2.3.13 reporting (very) incorrect vsize for some maildir folders

Hello!

Dovecot v2.3.13. Full, but anonymized, `doveconf -n` attached.

We are in the process of migrating maildir-backed users from filesystem quotes,
to using dovecot's "count" quota backend.

This is not reflected in the doveconf output because we're overriding
`quota`, `quota_rule` and `quota_vsizes` in userdb:

  {"quota":"count:User
quota","quota_rule":"*:storage=15G","quota_vsizes":"yes"}

Afterwards, for a couple users, we received reports that mail delivery had
failed because they were over quota. When looking into it for a specific user,
we noticed that the vsize reported for a particular folder (with 47k messages)
was reported as being nearly 50 times larger than it's on-disk size:

root at mail02:~# doveadm mailbox status -u anonymized_user 'messages recent
unseen vsize' 'anonymized/folder/name'
[...]
anonymized/folder/name messages=47338 recent=0 unseen=0 vsize=14335366070

However, the filesystem itself reports a much smaller size (but correct message
count):

root at mail02:~# du -hd1 /home/anonymized_user/Maildir/.anonymized.folder.name/
313M    /home/anonymized_user/Maildir/.anonymized.folder.name/cur
36K     /home/anonymized_user/Maildir/.anonymized.folder.name/tmp
4.0K    /home/anonymized_user/Maildir/.anonymized.folder.name/new
320M    /home/anonymized_user/Maildir/.anonymized.folder.name/

root at mail02:~# ls /home/anonymized_user/Maildir/.anonymized.folder.name/cur |
wc -l
47338

I have tried:

  - `doveadm force-resync -u anonymized_user`
  - deleting the index files in the specific folder, and running `doveadm index
-u anonymized_user '*'` as well as `doveadm mailbox status -u
anonymized_user vsize '*'`
  - deleting all `*index*` files in maildir, and running `doveadm index -u
anonymized_user '*'` as well as `doveadm mailbox status -u
anonymized_user vsize '*'`
  - comparing all maildir-sizes (S=) with their actual sizes to see if there are
discrepancies. There are none.
  - regardless of the above, set `maildir_broken_filename_sizes = yes`, deleted
indexes and reindexed.

Even after deleting the list index (dovecot.list.index) as well as the mailbox
indexes, recalculating the vsize seems very quick so I feel like the incorrect
vsize is being fetched from a cache somewhere instead of being recalculated..?

Any idea what is causing dovecot to report this wildly incorrect mailbox size?
Are there any other files (not matching `*index*`) that are responsible? I'd
rather not touch the user's control files, for obvious reasons.

On a side note: We're also experiencing some issues with
`mail_vsize_bg_after_count`. It seems to work correctly by returning a temporary
error and deferring to a background job, but the indexer-worker job often times
never appears to start (or do anything) at all:

dovecot  29622  0.0  0.0   4120  1124 ?        S    15:22   0:00 dovecot/indexer
[0 clients, 0 requests]
266248   29623  0.0  0.0   6008  4604 ?        S    15:22   0:00
dovecot/indexer-worker [idling]

Waiting a while, and attempting to refetch the quota returns the same temperror
with no indexer-worker being started. However, this is a separate issue from the
above and can be ignored for now.

Best regards,
Eirik Rye

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: doveconf.txt
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20210505/201c2b14/attachment.txt>

On 5. May 2021, at 15.42, Eirik Rye <rye at trojka.no>
wrote:
> 
> Afterwards, for a couple users, we received reports that mail delivery had
failed because they were over quota. When looking into it for a specific user,
we noticed that the vsize reported for a particular folder (with 47k messages)
was reported as being nearly 50 times larger than it's on-disk size:
> 
> root at mail02:~# doveadm mailbox status -u anonymized_user 'messages
recent unseen vsize' 'anonymized/folder/name'
> [...]
> anonymized/folder/name messages=47338 recent=0 unseen=0 vsize=14335366070
> 
> However, the filesystem itself reports a much smaller size (but correct
message count):
> 
> root at mail02:~# du -hd1
/home/anonymized_user/Maildir/.anonymized.folder.name/
> 313M    /home/anonymized_user/Maildir/.anonymized.folder.name/cur
> 36K     /home/anonymized_user/Maildir/.anonymized.folder.name/tmp
> 4.0K    /home/anonymized_user/Maildir/.anonymized.folder.name/new
> 320M    /home/anonymized_user/Maildir/.anonymized.folder.name/
> 
> root at mail02:~# ls
/home/anonymized_user/Maildir/.anonymized.folder.name/cur | wc -l
> 47338
> 
> I have tried:
> 
>  - `doveadm force-resync -u anonymized_user`
>  - deleting the index files in the specific folder, and running `doveadm
index -u anonymized_user '*'` as well as `doveadm mailbox status -u
anonymized_user vsize '*'`
>  - deleting all `*index*` files in maildir, and running `doveadm index -u
anonymized_user '*'` as well as `doveadm mailbox status -u
anonymized_user vsize '*'`
>  - comparing all maildir-sizes (S=) with their actual sizes to see if there
are discrepancies. There are none.
S= is the "physical size", W= is the "virtual size".
quota=count / vsize calculations should be using the W= value, not the S= value.
> Even after deleting the list index (dovecot.list.index) as well as the
mailbox indexes, recalculating the vsize seems very quick so I feel like the
incorrect vsize is being fetched from a cache somewhere instead of being
recalculated..?
The sizes can also be stored in dovecot-uidlist.