dovecot - Apr 2015 - Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads

very_dirty_syncs helped a lot - the problems still happen but only about every
30 minutes.  It seems this flag prevents re-indexing on whatever was constantly
inducing it (with every client-check, if there was a change written to the
directory, is my best guess watching the logs go).

I still get all the same symptoms, but far less frequently.

I'm inclined to believe, as trivial as it may be to enumerate, that:

Something is triggering dovecot to believe the indexes need to be rebuilt.

When checking mail during the rebuild, clients get confused by UIDs in
transition.

Today I had to quit TB and restart because duplicates filled my preview pane. 
Mulberry has been better, when it gets confused now, it is usually sufficient to
initiate a manual check or expunge to get the list to update.

I would think that sdbox would alleviate these issues, no?

-------- Original Message --------
Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries,
duplicated messages, excessive full re-downloads
From: Timo Sirainen <tss at iki.fi>
To: David Gessel <gessel at blackrosetech.com>
Date: Sat Apr 18 2015 17:38:02 GMT+0300 (Arabic Standard Time)
> On 18 Apr 2015, at 16:00, David Gessel <gessel at blackrosetech.com>
wrote:
>>
>>> No. My best guess is that (your) ZFS+FreeBSD is simply not behaving
the way Dovecot wants filesystems to behave (readdir() sometimes skipping over
files). I don't have any other suggestion than trying a different filesystem
or OS. I haven't heard ZFS having these problems before though.
>>
>>
>> Or migrating to sdbox.  Absent any further suggestions, that will be my
next step.  My commitment to Maildir is far lower than my commitment to
FreeBSD/ZFS.  It is just a historical preference coming from a decade+ of
courier IMAP.
> 
> Actually as a workaround you could try maildir_very_dirty_syncs=yes which
gets rid of most of the readdir()s.
>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, 23 Apr 2015, David Gessel wrote:
> I'm inclined to believe, as trivial as it may be to enumerate, that:
>
> Something is triggering dovecot to believe the indexes need to be rebuilt.
> When checking mail during the rebuild, clients get confused by UIDs in
transition.
>
> I would think that sdbox would alleviate these issues, no?
The real problem is that you do not know _why_ "Something is triggering 
dovecot to believe the indexes need to be rebuilt".

This is the same for sdbox and mdbox, IMHO.

That's why I asked about if some external process is trying to change the 
mail storage. Is there something except Dovecot that changes the mtime of 
the directories "new", "cur" or Maildir base? Do you deliver
messages
without Dovecot LDA/LMTP? Do you store different information in the 
Maildir? Do you (not) have separate mail storage and user home 
directories? Do you run a virus checker on file system level?

Do you run two Dovecot instances on the same server, maybe as left over 
from some testing or crash?

- -- 
Steffen Kaiser
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEVAwUBVTnw6nz1H7kL/d9rAQKz+AgAuD0pyVOSQILqA6kLYgGOcUU0Wa8ca91H
3UB2t18AxVXx3/elCguTdNFpd0AB2dMfjkWaob/9ywr60i8wjNIZPQcWAI4hmeoV
2YHm+YJ8ndfSdYKQ6mN34mDBmD1+K69h8v5GVfQ5jOakcwPSqemBWY8RjfLtmSpU
wgfRZtkG209BZxaAi3fj2J6xrpgE4BYG0g3KwafVRGLZMcBFujO0UJgPqeMmEMJY
NwSgSgRoU1Aau+nTQGYrFzsSgtuG8R2LnF2l4nYWTL12PAiiqgbuAQCQ/WXf3jzQ
OWtzcKM7laO9nc5iXuaAiO59TrV9VtevcTrE8usK2FdJke2C2bEVWw==3KjE
-----END PGP SIGNATURE-----

The update to dovecot2-2.2.18/dovecot-pigeonhole-0.4.8 seems to have made
problems worse.  Whole directories don't load at all - or they may load a
weird segment - like mail from a few months ago back and mail from the last two
days, but nothing in between.

I did a 
# find /mail -name "dovecot.index*" -type f -delete

and then on the client side (K9) things clear up, on TB a Right click on folder
-> properties -> repair folder cleans things up.  Of course all messages
have to be redownloaded so it isn't exactly a performant option.

Is there a way to completely disable caching on dovecot?  I know it is sort of
the raison d'?tre, but if I can't solve the problems with dovecot,
I'll have to try a different, non-caching/indexing server anyway and at
least it would help narrow down the issues.


-------- Original Message --------
Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries,
duplicated messages, excessive full re-downloads
From: David Gessel <gessel at blackrosetech.com>
To: Timo Sirainen <tss at iki.fi>
Date: Thu Apr 23 2015 18:39:02 GMT+0400 (Arabian Standard Time)
> very_dirty_syncs helped a lot - the problems still happen but only about
every 30 minutes.  It seems this flag prevents re-indexing on whatever was
constantly inducing it (with every client-check, if there was a change written
to the directory, is my best guess watching the logs go).
> 
> I still get all the same symptoms, but far less frequently.
> 
> I'm inclined to believe, as trivial as it may be to enumerate, that:
> 
> Something is triggering dovecot to believe the indexes need to be rebuilt.
> 
> When checking mail during the rebuild, clients get confused by UIDs in
transition.
> 
> Today I had to quit TB and restart because duplicates filled my preview
pane.  Mulberry has been better, when it gets confused now, it is usually
sufficient to initiate a manual check or expunge to get the list to update.
> 
> I would think that sdbox would alleviate these issues, no?
> 
> -------- Original Message --------
> Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries,
duplicated messages, excessive full re-downloads
> From: Timo Sirainen <tss at iki.fi>
> To: David Gessel <gessel at blackrosetech.com>
> Date: Sat Apr 18 2015 17:38:02 GMT+0300 (Arabic Standard Time)
> 
>> On 18 Apr 2015, at 16:00, David Gessel <gessel at
blackrosetech.com> wrote:
>>>
>>>> No. My best guess is that (your) ZFS+FreeBSD is simply not
behaving the way Dovecot wants filesystems to behave (readdir() sometimes
skipping over files). I don't have any other suggestion than trying a
different filesystem or OS. I haven't heard ZFS having these problems before
though.
>>>
>>>
>>> Or migrating to sdbox.  Absent any further suggestions, that will
be my next step.  My commitment to Maildir is far lower than my commitment to
FreeBSD/ZFS.  It is just a historical preference coming from a decade+ of
courier IMAP.
>>
>> Actually as a workaround you could try maildir_very_dirty_syncs=yes
which gets rid of most of the readdir()s.
>>

Steffen,

I apologize for missing your helpful message - I wasn't intentionally
ignoring it. One of the artifacts of my mail issues is that mail disappears,
this included.

-------- Original Message --------
Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries,
duplicated messages, excessive full re-downloads
From: Steffen Kaiser <skdovecot at smail.inf.fh-brs.de>
To: David Gessel <gessel at blackrosetech.com>
Date: Fri Apr 24 2015 10:29:46 GMT+0300 (Arabic Standard Time)
> On Thu, 23 Apr 2015, David Gessel wrote:
> 
>> I'm inclined to believe, as trivial as it may be to enumerate,
that:
> 
>> Something is triggering dovecot to believe the indexes need to be
rebuilt.
>> When checking mail during the rebuild, clients get confused by UIDs in
transition.
> 
>> I would think that sdbox would alleviate these issues, no?
> 
> The real problem is that you do not know _why_ "Something is
triggering dovecot to believe the indexes need to be rebuilt".
I'm not 100% sure that's what is happening.
> 
> This is the same for sdbox and mdbox, IMHO.
that would be sad - but I haven't tried that yet.
> 
> That's why I asked about if some external process is trying to change
the mail storage. Is there something except Dovecot that changes the mtime of
the directories "new", "cur" or Maildir base?
Not that I am aware of.  this is a jail only running mail.
> Do you deliver messages without Dovecot LDA/LMTP? 
Dovecot LDA only.
>Do you store different information in the Maildir? 
nothing not part of Dovecot's processes - but the full text index files are
also stored there.
> Do you (not) have separate mail storage and user home directories? 
Virtual mail configuration - home directories are completely isolated.  Nothing
happens in /mail that isn't mail related.
> Do you run a virus checker on file system level?
no, but amavisd calls clamAV on inbound messages.
> 
> Do you run two Dovecot instances on the same server, maybe as left over
from some testing or crash?
certainly not intentionally and I'm pretty confident not actually; these
issues survive many reboots without much variation, so not from testing.
> 
> -- Steffen Kaiser

Some updates to the process:

I experimented with all variations of Mail processes values with no real
improvement - some perhaps but likely as not just observational variations
rather than meaningful data.  Specifically in 10-mail.conf:
 mmap_disable = yes
 mail_fsync = always
 lock_method = flock  

I'm probably suffering from confirmation bias, but I think things got a
little bit better.

However, I still got double messages in TB and K9 on some checks.  Then I looked
at the IMAP capability string returned from telnet localhost 993.  It didn't
include NAMESPACE.  The post-login enumeration did, but not the pre-login.

adding
imap_capability = +NAMESPACE
to 20-imap.conf
seems to have cleared up the appearance of double entries in clients.  I had one
message's header/display get confused (another symptom of the issues) but
given the problems my local client database is pretty scrambled.  when I have a
decent network connection for a few days, I'll try recreating my client
database and see if that helps.

Il 20/05/2015 10:44, David Gessel ha scritto:
> Is there a way to completely disable caching on dovecot?
Yes: mail_location = maildir:~/Maildir:INDEX=MEMORY

-- 
Alessio Cecchi
http://www.linkedin.com/in/alessice