dovecot - May 2021 - Dovecot v2.3.13 reporting (very) incorrect vsize for some maildir folders

On 20. May 2021, at 14.12, Eirik Rye <rye at trojka.no>
wrote:
> 
> 
>> On 20 May 2021, at 13:44, Timo Sirainen <timo at sirainen.com>
wrote:
>> 
>> You can also look at the folder-level vsizes to see which one is
causing the differences (or are they all doubled?)
> 
> In this user's case, it is only the Trash-folder that has the wrong
vsize calculation:
> 
> # doveadm mailbox status -u <username> 'messages vsize'
'*'
> Drafts messages=0 vsize=0
> Sent messages=0 vsize=0
> Trash messages=14870 vsize=14584428026
> Spam messages=3 vsize=227701
> INBOX messages=1866 vsize=1640766021
> 
> The other mailboxes (INBOX and Spam) are both within what I consider
reasonable in terms of differences in virtual/physical sizes:
> 
> # du -bs /mail/<username>/Maildir/cur
> 1603071610      /mail/<username>/Maildir/cur
> # du -bs /mail/<username>/Maildir/.Spam
> 241489  /mail/<username>/Maildir/.Spam
> 
> But the Trash mailbox is physically HALF the size of what Dovecot reports:
> 
> # du -bs /mail/<username>/Maildir/.Trash
> 7200481589      /mail/<username>/Maildir/.Trash
> 
> The message count reported by dovecot (14870) is correct, however:
> 
> # ls /mail/<username>/Maildir/.Trash/cur | wc -l
> 14870
> 
> With the other users I have noticed the same issue, it's a different
mailbox and not Trash.
Well, next step could be to compare individual mail sizes. It would require
writing some kind of a script to do the comparison though. But for example you
could look at both physical & virtual sizes in dovecot first to see if
there's a big difference:

doveadm fetch -u user 'guid storageid size.physical size.virtual'
mailbox Trash

I think either guid or storageid or both have the Maildir base filename. You
could also compare those to the ls -l output.

> On 20 May 2021, at 14:59, Timo Sirainen <timo at sirainen.com> wrote:
> 
> Well, next step could be to compare individual mail sizes. It would require
writing some kind of a script to do the comparison though.
Hello again,

Apologies, I think dovecot is innocent in all of this.

I noticed that `ls -s` reported a completely different size to `du`, but similar
to what dovecot reports:

# ls -s | head -1
total 14099016
# du
7050436 .

I assume there are some sparseness or block size related shenanigans going on
here instead, causing differences in reported physical usage by `du` (syscall
`newfstatat()`) compared to `ls` (syscall `lstat()`) and dovecot.

The filesystem quota system in Linux, which is what we're migrating from,
apparently uses the same calculation method as `du`, which adds to the
confusion.

- Eirik