dovecot - Jul 2016 - maildirsize calculation gives higher value than a real disk utilization

Hi,

hope someone can help also if this problem is difficult to debug

I have a mail account that has on file system 32GB however
quotacalculation gives a higher value 41262146

I've already tried to:

- doveadm quota recalc -u user at domain

- removing maildirsize from /users/vmail/domain/user/Maildir

- checking all emails messages if they have the size in the name
(S=xxxx) different from the real size. I used the script here in attach
giving it in input the output of << find /users/vmail/domain/user -name
*S=* -type f>>


I don't have mail compression


the mail account have a lot of mailboxes and at now 78319 messages

doveadm quota get -u user at domain
Quota name Type       Value   
Limit
%
User quota STORAGE 41262146
42991616
95
User quota MESSAGE    78319       
-
0


this is the same command run with -Dv


doveadm(root): Debug: Loading modules from directory: /usr/lib64/dovecot
doveadm(root): Debug: Module loaded:
/usr/lib64/dovecot/lib10_quota_plugin.so
doveadm(root): Debug: Loading modules from directory:
/usr/lib64/dovecot/doveadm
doveadm(root): Debug: Skipping module doveadm_acl_plugin, because
dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_acl_plugin.so:
undefined symbol: acl_user_module (this is usually intentional, so just
ignore this message)
doveadm(root): Debug: Skipping module doveadm_expire_plugin, because
dlopen() failed:
/usr/lib64/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined
symbol: expire_set_deinit (this is usually intentional, so just ignore
this message)
doveadm(root): Debug: Module loaded:
/usr/lib64/dovecot/doveadm/lib10_doveadm_quota_plugin.so
doveadm(root): Debug: Module loaded:
/usr/lib64/dovecot/doveadm/lib10_doveadm_sieve_plugin.so
doveadm(root): Debug: Skipping module doveadm_fts_lucene_plugin, because
dlopen() failed:
/usr/lib64/dovecot/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined
symbol: lucene_index_iter_deinit (this is usually intentional, so just
ignore this message)
doveadm(root): Debug: Skipping module doveadm_fts_plugin, because
dlopen() failed: /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so:
undefined symbol: fts_backend_rescan (this is usually intentional, so
just ignore this message)
doveadm(user at domain): Debug: Added userdb setting:
mail_location=maildir:/users/vmail/domain/user/Maildir
doveadm(user at domain): Debug: Added userdb setting:
plugin/quota_rule=*:storage=41G
doveadm(user at domain): Debug: Effective uid=5000, gid=5000,
home=/users/vmail/domain/user
doveadm(user at domain): Debug: Quota root: name=User quota backend=maildir
argsdoveadm(user at domain): Debug: Quota rule: root=User quota mailbox=*
bytes=44023414784 messages=0
doveadm(user at domain): Debug: Quota warning: bytes=41822244044 (95%)
messages=0 reverse=no command=quota-warning 95 user at domain
doveadm(user at domain): Debug: Quota grace: root=User quota bytes=20971520
doveadm(user at domain): Debug: Namespace inbox: type=private, prefix=,
sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes
location=maildir:/users/vmail/domain/user/Maildir
doveadm(user at domain): Debug: maildir++:
root=/users/vmail/domain/user/Maildir, index=, indexpvt=, control=,
inbox=/users/vmail/domain/user/Maildir, altQuota name Type       Value    Limit
                                    %
User quota STORAGE 41202926 42991616                                    
                                   95
User quota MESSAGE    78247        -

hope someone can help

I'm running dovecot-2.2.10-5.el7 on centos7


-------------- next part --------------
A non-text attachment was scrubbed...
Name: messagesize_check_mismatch.pl
Type: application/x-perl
Size: 485 bytes
Desc: not available
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20160721/db4be929/attachment-0001.pl>

I found a similar error to this posted on the dovecot list from 2013, but no
solution.

I'm using the avelsieve squirrelmail plugin with dovecot. It throws an error
when it connects to managesieve.

dovecot version is 2.0.19

ngrep of the connection shows....

T 127.0.0.1:4190 -> 127.0.0.1:55753 [AP]
  "IMPLEMENTATION" "Dovecot Pigeonhole".."SIEVE"
"fileinto reject envelope en
  coded-character vacation subaddress comparator-i;ascii-numeric relational r
  egex imap4flags copy include variables body enotify environment mailbox dat
  e ihave".."NOTIFY" "mailto".."SASL"
"PLAIN LOGIN CRAM-MD5".."VERSION" "1.0"
  ..OK "Dovecot ready."..
##
T 127.0.0.1:55753 -> 127.0.0.1:4190 [AP]
  AUTHENTICATE "PLAIN" \{80+}..
##
T 127.0.0.1:55753 -> 127.0.0.1:4190 [AP]
  bWF5c0A0c2Vhc29uc3Jlc29ydC5jb20AbWF5c0A0c2Vhc29uc3Jlc29ydC5jb20AZmxvdmVyMDk
  xNw==..
##
T 127.0.0.1:4190 -> 127.0.0.1:55753 [AP]
  NO "Invalid characters in atom"..NO "Error in MANAGESIEVE
command received
  by server."..

On 21 Jul 2016, at 12:54, Andrea Lisci <andrelisci at gmail.com>
wrote:
> 
> Hi,
> 
> hope someone can help also if this problem is difficult to debug
> 
> I have a mail account that has on file system 32GB however
> quotacalculation gives a higher value 41262146
> 
> I've already tried to:
> 
> - doveadm quota recalc -u user at domain
If after quota recalc the quota value is higher than "du" value, try
with "du -l" instead. My guess it that the same emails have been
copied to multiple folders with hard links. Dovecot counts each instance as its
own mail, even though on filesystem they are deduplicated.

Yes, you are right

du -skl /users/vmail/domain/user shows 41565640

su -sk /users/vmail/domain/user shows 33361728

doveadm quota get -u user at domain shows 41354370

so considering checking 'du' by '-l' shows 211270kb of
difference from
'quota get'.

the mail account was imported by imapsync and the user has made
modification to the folder structure by microsoft outlook 2010.

What can cause this ?




On 07/25/2016 12:11 AM, Timo Sirainen wrote:
> On 21 Jul 2016, at 12:54, Andrea Lisci <andrelisci at gmail.com>
wrote:
>> Hi,
>>
>> hope someone can help also if this problem is difficult to debug
>>
>> I have a mail account that has on file system 32GB however
>> quotacalculation gives a higher value 41262146
>>
>> I've already tried to:
>>
>> - doveadm quota recalc -u user at domain
> If after quota recalc the quota value is higher than "du" value,
try with "du -l" instead. My guess it that the same emails have been
copied to multiple folders with hard links. Dovecot counts each instance as its
own mail, even though on filesystem they are deduplicated.
>