dovecot - May 2021 - Dovecot v2.3.13 reporting (very) incorrect vsize for some maildir folders

On 5. May 2021, at 15.42, Eirik Rye <rye at trojka.no>
wrote:
> 
> Afterwards, for a couple users, we received reports that mail delivery had
failed because they were over quota. When looking into it for a specific user,
we noticed that the vsize reported for a particular folder (with 47k messages)
was reported as being nearly 50 times larger than it's on-disk size:
> 
> root at mail02:~# doveadm mailbox status -u anonymized_user 'messages
recent unseen vsize' 'anonymized/folder/name'
> [...]
> anonymized/folder/name messages=47338 recent=0 unseen=0 vsize=14335366070
> 
> However, the filesystem itself reports a much smaller size (but correct
message count):
> 
> root at mail02:~# du -hd1
/home/anonymized_user/Maildir/.anonymized.folder.name/
> 313M    /home/anonymized_user/Maildir/.anonymized.folder.name/cur
> 36K     /home/anonymized_user/Maildir/.anonymized.folder.name/tmp
> 4.0K    /home/anonymized_user/Maildir/.anonymized.folder.name/new
> 320M    /home/anonymized_user/Maildir/.anonymized.folder.name/
> 
> root at mail02:~# ls
/home/anonymized_user/Maildir/.anonymized.folder.name/cur | wc -l
> 47338
> 
> I have tried:
> 
>  - `doveadm force-resync -u anonymized_user`
>  - deleting the index files in the specific folder, and running `doveadm
index -u anonymized_user '*'` as well as `doveadm mailbox status -u
anonymized_user vsize '*'`
>  - deleting all `*index*` files in maildir, and running `doveadm index -u
anonymized_user '*'` as well as `doveadm mailbox status -u
anonymized_user vsize '*'`
>  - comparing all maildir-sizes (S=) with their actual sizes to see if there
are discrepancies. There are none.
S= is the "physical size", W= is the "virtual size".
quota=count / vsize calculations should be using the W= value, not the S= value.
> Even after deleting the list index (dovecot.list.index) as well as the
mailbox indexes, recalculating the vsize seems very quick so I feel like the
incorrect vsize is being fetched from a cache somewhere instead of being
recalculated..?
The sizes can also be stored in dovecot-uidlist.

> On 10 May 2021, at 11:52, Timo Sirainen <timo at sirainen.com> wrote:
> 
> S= is the "physical size", W= is the "virtual size".
quota=count / vsize calculations should be using the W= value, not the S= value.

I renamed all messages containing S= and W= values in a user's mailbox
(Trash) which had the incorrect quota calculation, stripping these from the
filenames to ensure they were not causing incorrect calculations.

Then, I deleted every single dovecot* file in the user's mailbox (all
indexes and control files, including dovecot-uidlist):

  root at server:~# find /mail/<username>/Maildir/ -type f -name
'dovecot*' -delete

Running `doveadm mailbox status -u <username> 'vsize'
'Trash'` -still-, even after all this, returns a vsize calculation that
is off by a factor of about two:

root at server:~# du -bs /mail/<username>/Maildir/.Trash
7200481589      /mail/<username>/Maildir/.Trash
root at server:~# doveadm mailbox status -u <username> 'vsize'
'Trash'
Trash vsize=14584428026

(The user was not logged in during any of this testing)

I am at a bit of a loss at what is causing this issue now. I have tested against
2.3.13 and 2.3.14 now. Any chance I may have hit a bug in these versions?

It is worth mentioning that I have only encountered a small handful of users
(out of about 100k) where the quota calculation is wildly incorrect. For the
most part it appears to be right.

- Eirik