dovecot - Nov 2008 - (1.0.13) fsync failed: Disk quota exceeded for some accounts

Hello all,

We've recently migrated the mail server used by our 5000 students, from 
Tru64/UW-IMAP/Procmail/Postfix/mbox to Debian
Etch/Dovecot/Deliver/Postfix/Maildir

E-mails are not stored directly on the server (except for index and control
files), but on an NAS
that exports the students' homedirs on NFS. Each student has a 100MB quota
on the NAS.

Here is the configuration :
vega:~# dovecot -n
# 1.0.13: /etc/dovecot/dovecot.conf
log_timestamp: %Y-%m-%d %H:%M:%S
protocols: imap pop3
listen: [::]
ssl_disable: yes
disable_plaintext_auth: no
verbose_ssl: yes
login_dir: /var/run/dovecot/login
login_executable(default): /usr/lib/dovecot/imap-login
login_executable(imap): /usr/lib/dovecot/imap-login
login_executable(pop3): /usr/lib/dovecot/pop3-login
login_greeting_capability(default): yes
login_greeting_capability(imap): yes
login_greeting_capability(pop3): no
login_process_size: 32
login_max_processes_count: 1024
mail_privileged_group: mail
mail_location:
maildir:~/Maildir:INDEX=/var/mail/index/%u:CONTROL=/var/mail/control/%u
maildir_copy_with_hardlinks: yes
maildir_copy_preserve_filename: yes
mail_executable(default): /usr/lib/dovecot/imap
mail_executable(imap): /usr/lib/dovecot/imap
mail_executable(pop3): /usr/lib/dovecot/pop3
mail_plugin_dir(default): /usr/lib/dovecot/modules/imap
mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap
mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3
pop3_uidl_format(default):
pop3_uidl_format(imap):
pop3_uidl_format(pop3): %08Xu%08Xv
namespace:
   type: private
   separator: /
   inbox: yes
namespace:
   type: private
   separator: /
   prefix: mail/
   hidden: yes
auth default:
   cache_size: 1024
   passdb:
     driver: pam
     args: blocking=yes cache_key=%u dovecot
   userdb:
     driver: passwd
     args: blocking=yes



When one mails a student that has filled his quota, one immediately receives
this message :
"Your message to <studentx> was automatically rejected:
Not enough disk space

Reporting-UA: vega; Dovecot Mail Delivery Agent
Final-Recipient: rfc822; studentx
Original-Message-ID: <490F0A9A.8030109 at utc.fr>
Disposition: automatic-action/MDN-sent-automatically; deleted"

And in /var/log/mail.log :
(...) save failed to INBOX: Not enough disk space
(...) Rejected: Not enough disk space


So far, so good, except that for *some* users (and I can't seem to find
anything unusual about their
account), one gets this in /var/log/mail.log :
Nov  6 10:42:24 vega deliver(studentx): 
fsync(/voletu/users/studentx/Maildir/tmp/1225964544.P25878Q0M80993.vega) failed:
Disk quota exceeded
Nov  6 10:42:24 vega deliver(studentx):
msgid=<2254E7BC-665D-4599-B719-825E3A58DC1D at utc.fr>: save
failed to INBOX: Internal error occurred. Refer to server log for more
information. [2008-11-06
10:42:24]
Nov  6 10:42:24 vega postfix/local[25827]: 010F45EC18: to=<studentx at
vega.utc.fr>,
orig_to=<studentx at etu.utc.fr>, relay=local, delay=0.2,
delays=0.02/0/0/0.17, dsn=4.3.0,
status=deferred (temporary failure)

So for *some* users, it seems that Deliver doesn't detect that there
isn't enough space, it tries to
write the e-mail and of course fails, then reports an error, and Postfix
interprets this as a
temporary error and retries later.

After about a week :
vega:~# qshape deferred
                                          T  5 10 20 40 80 160 320 640 1280
1280+
                                  TOTAL 500  1  4  3  9 12   6   8  17  140  
300
                             etu.utc.fr 499  1  4  3  9 12   6   8  17  139  
300
                            vega.utc.fr   1  0  0  0  0  0   0   0   0    1    
0



For the students that have this problem (about 30 for now), this happens *even*
if I delete
~studentx/Maildir, /var/mail/index/studentx and /var/mail/control/studentx, let
Deliver recreate the
necessary files (by sending the user a mail) so that everything is clean,
artificially fill the
quota with a big file in the homedir, then send a mail again. It gets
deferred...


I have looked at the changelog for Dovecot, but have not found a clear clue that
is a bug (if it is
!) that has already been corrected in a more recent version...
Though if possible, I'd like to keep the Debian package (from
etch-backports) instead of compiling
it from source.

Also, while I've found Dovecot's wiki to be surprisingly informative,
there are still a few points
that did not seem very clear :
- should I use the quota:fs plugin in this case (it seems to work out well
without it for most users) ?
- would it work in v1.0.13 on NFS ?
- if it worked, and I enabled it, what would change for me ?
- what is *supposed* to happen without the quota plugin when a user has filled
his quota : mail is
rejected, or mail is deferred ?
- and what about imap_quota ?


Best regards,
Eric

Hi,

El Jueves, 6 de Noviembre de 2008 a las 11:30, Eric Marin
escribi?:
> - should I use the quota:fs plugin in this case (it seems to work out well
> without it for most users) ? 
quota:fs is only about reporting quota status using IMAP. So, it should do no 
diference in this case.
> - would it work in v1.0.13 on NFS ?
No, you have to apply a patch to get 1.0 quota over NFS
> - if it worked, and I enabled it, what would change for me ?
You can offer that information to your imap users. I think it's good
offering
this info to my users, but this is another issue.

Maybe the problem is with your NAS quotas. Have you tried deleting one of 
these users' quota, and reapplying it?

HTH
-- 
Joseba Torre. CIDIR Bizkaia.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part.
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20081106/6373caba/attachment-0002.bin>

Eric Marin wrote:
> So for *some* users, it seems that Deliver doesn't detect that there
> isn't enough space, it tries to write the e-mail and of course fails,
> then reports an error, and Postfix interprets this as a temporary
> error and retries later.
>

Without knowing anything about the problem I would guess this is a
rounding issue.  Probably you are off by the number of bytes in your
short message saying you are over-quota.  Sometimes you are just inside
and sometimes just outside?

Ed

Hi,

Joseba Torre a ?crit :
> Hi,
> 
> El Jueves, 6 de Noviembre de 2008 a las 11:30, Eric Marin escribi?:
>> - should I use the quota:fs plugin in this case (it seems to work out
well
>> without it for most users) ? 
> 
> quota:fs is only about reporting quota status using IMAP. So, it should do
no
> diference in this case.
> 
>> - would it work in v1.0.13 on NFS ?
> 
> No, you have to apply a patch to get 1.0 quota over NFS
> 
>> - if it worked, and I enabled it, what would change for me ?
> 
> You can offer that information to your imap users. I think it's good
offering
> this info to my users, but this is another issue.
> 
Yes that would be great, i'll see about that, especially for Horde/IMP. But
then, what does
imap_quota do that quota itself doesn't ?
> Maybe the problem is with your NAS quotas. Have you tried deleting one of 
> these users' quota, and reapplying it?
> 
> HTH
Unfortunately, I don't think this can be the cause because we defined quotas
globally on the NAS :
only one line defines the quota for all students...

Eric

On Thu, 2008-11-06 at 11:30 +0100, Eric Marin wrote:
> So far, so good, except that for *some* users (and I can't seem to find
anything unusual about their
> account), one gets this in /var/log/mail.log :
> Nov  6 10:42:24 vega deliver(studentx): 
> fsync(/voletu/users/studentx/Maildir/tmp/1225964544.P25878Q0M80993.vega)
failed: Disk quota exceeded
Strange that no-one had complained about this before. The problem is
that your kernel is caching writes and it doesn't know the user is over
quota. So then when fsync() finally gets called the kernel writes the
data to NFS server, which replies that quota exceeded.

This fixes it: http://hg.dovecot.org/dovecot-1.0/rev/9879434a6339

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20081106/63ba82f8/attachment-0002.bin>

It works, thank you very much !

What can I say, this is a shining example of what's good with open-source.

Eric

Timo Sirainen a ?crit :
> On Thu, 2008-11-06 at 11:30 +0100, Eric Marin wrote:
>> So far, so good, except that for *some* users (and I can't seem to
find anything unusual about their
>> account), one gets this in /var/log/mail.log :
>> Nov  6 10:42:24 vega deliver(studentx): 
>>
fsync(/voletu/users/studentx/Maildir/tmp/1225964544.P25878Q0M80993.vega) failed:
Disk quota exceeded
> 
> Strange that no-one had complained about this before. The problem is
> that your kernel is caching writes and it doesn't know the user is over
> quota. So then when fsync() finally gets called the kernel writes the
> data to NFS server, which replies that quota exceeded.
> 
> This fixes it: http://hg.dovecot.org/dovecot-1.0/rev/9879434a6339
>