dovecot - May 2008 - Vanishing maildirsize files with 1.0.7 using 'deliver' via postfix

Aloha,

I've searched the list and have found no similar reports, so I'm asking
here.

We have 3 RH5.1 boxes running the stock Postfix 2.3.3, but due to issues
with the stock Dovecot 1.0rc15, Redhat supplied us with the version of
Dovecot 1.0.7 that will be in Redhat 5.2.

The mail store is automounted per-user from a Sun box via NFS.

We were using it with no problems until last Friday, when we attempted to
implement quotas.  I had done quite a lot of testing on a single system, so
I didn't anticipate any issues here.  I created maildirsize files for each
user in the NFS-mounted /var/spool/mail/$USER and everything seemed fine.
I changed mailbox_command to /usr/libexec/dovecot/deliver  in postfix
main.cf.  Mail was delivered, quota files updated, all good.

However almost immediately a maildirsize folder vanished.  Soon after
another, then another, and now we're missing nearly 40 of them.

Any ideas as to why they're vanishing?  Something to do with NFS?   I've
tried to trigger it by massively sending to the same user on all three
systems but have yet to reproduce it at will.

Any clues would be appreciated.  For reasons of support I cannot upgrade to
a later dovecot at this time.

Best Regards,

- jason

Jason Forester wrote:
> We have 3 RH5.1 boxes running the stock Postfix 2.3.3, but due to issues
> with the stock Dovecot 1.0rc15, Redhat supplied us with the version of
> Dovecot 1.0.7 that will be in Redhat 5.2.
 >
> The mail store is automounted per-user from a Sun box via NFS.
Timo strongly recommends to use 1.1 if you are using NFS...

1.1 is available via atrpms.net

Charles Marcus wrote:
>> Jason Forester wrote:
>> We have 3 RH5.1 boxes running the stock Postfix 2.3.3, but due to
issues
>> with the stock Dovecot 1.0rc15, Redhat supplied us with the version of
>> Dovecot 1.0.7 that will be in Redhat 5.2.
>>
>> The mail store is automounted per-user from a Sun box via NFS.
>
>Timo strongly recommends to use 1.1 if you are using NFS...
>
>1.1 is available via atrpms.net
I do understand, but as I stated fairly clearly I thought "For reasons
of support I cannot upgrade to a later dovecot at this time."

Of course I have a ticket open with RedHat, but I thought I might come
to the source, find out if it's some form of a known bug, and then
perhaps I *could* use that as leverage to get another version
installed.  So the question still stands:

Is this a known bug?  Is maildirsize not NFS safe?

Knowing that would be a huge first step  for me, so thanks in advance
for any assistance.

Best Regards,

-jason


On Wed, May 21, 2008 at 12:21 PM, Jason Forester <hilobird at gmail.com>
wrote:
>
> Aloha,
>
> I've searched the list and have found no similar reports, so I'm
asking here.
>
> We have 3 RH5.1 boxes running the stock Postfix 2.3.3, but due to issues
with the stock Dovecot 1.0rc15, Redhat supplied us with the version of Dovecot
1.0.7 that will be in Redhat 5.2.
>
> The mail store is automounted per-user from a Sun box via NFS.
>
> We were using it with no problems until last Friday, when we attempted to
implement quotas.  I had done quite a lot of testing on a single system, so I
didn't anticipate any issues here.  I created maildirsize files for each
user in the NFS-mounted /var/spool/mail/$USER and everything seemed fine.   I
changed mailbox_command to /usr/libexec/dovecot/deliver  in postfix main.cf. 
Mail was delivered, quota files updated, all good.
>
> However almost immediately a maildirsize folder vanished.  Soon after
another, then another, and now we're missing nearly 40 of them.
>
> Any ideas as to why they're vanishing?  Something to do with NFS?  
I've tried to trigger it by massively sending to the same user on all three
systems but have yet to reproduce it at will.
>
> Any clues would be appreciated.  For reasons of support I cannot upgrade to
a later dovecot at this time.
>
> Best Regards,
>
> - jason

On Wed, 2008-05-21 at 12:21 -1000, Jason Forester wrote:
> We have 3 RH5.1 boxes running the stock Postfix 2.3.3, but due to issues
> with the stock Dovecot 1.0rc15, Redhat supplied us with the version of
> Dovecot 1.0.7 that will be in Redhat 5.2.
There is one change to Maildir++ quota since v1.0.7:
http://hg.dovecot.org/dovecot-1.0/rev/288fd70ac258

But that's mostly an optimization and doesn't cause files to be deleted.
> However almost immediately a maildirsize folder vanished.  Soon after
> another, then another, and now we're missing nearly 40 of them.
Maildir++ quota recalculation works like:

Go through all files and sum up their sizes. Keep track of the highest
directory mtime. Once everything is calculated, again go through all
directories and find the highest mtime. If the highest mtime had
increased, something had changed, so delete maildirsize file.

So could it be simply that the recalculation is slow enough that the
maildir changes during it and causes the file to be deleted? Another
possibility is that some error happens, but then Dovecot should have
logged something.

Do you keep your quota limits stored only in the maildirsize file? If
not, having the files deleted shouldn't affect functionality, but of
course the performance is worse.

The rebuild performance could be improved if maildir filenames contain
",S=xxx" containing the file's size so stat() calls can be
avoided. But
you can't really change existing filenames without having them show up
as new mails.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20080525/e24daccd/attachment-0002.bin>

Hi Timo, thanks for replying.  My comments are inline:

On Sun, May 25, 2008 at 2:25 AM, Timo Sirainen <tss at iki.fi> wrote:

<snip>
>> However almost immediately a maildirsize folder vanished.  Soon after
>> another, then another, and now we're missing nearly 40 of them.
>
> Maildir++ quota recalculation works like:
>
> Go through all files and sum up their sizes. Keep track of the highest
> directory mtime. Once everything is calculated, again go through all
> directories and find the highest mtime. If the highest mtime had
> increased, something had changed, so delete maildirsize file.
>
> So could it be simply that the recalculation is slow enough that the
> maildir changes during it and causes the file to be deleted? Another
> possibility is that some error happens, but then Dovecot should have
> logged something.
I'm not seeing anything in my logs.  It's certainly possible that
multiple mails are delivered during a recalculation.  I'm not quite
sure I understand the mtime explanation above, are you saying that if
dovecot thinks that the maildirsize file is inaccurate it deletes it
and goes on?
> Do you keep your quota limits stored only in the maildirsize file? If
> not, having the files deleted shouldn't affect functionality, but of
> course the performance is worse.
Basically yes, we have many users with different quotas and as such we
cannot use a single quota in the config file.

So, it sounds like I'm the only person who has encountered this
problem?  If it's really just a consequence of the maildir changing,
wouldn't it be more common?

Thanks again Timo,

- jason