dovecot - Jun 2015 - Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads

Il 20/05/2015 10:44, David Gessel ha scritto:
> Is there a way to completely disable caching on dovecot?
Yes: mail_location = maildir:~/Maildir:INDEX=MEMORY

-- 
Alessio Cecchi
http://www.linkedin.com/in/alessice

Alessio Cecchi skrev den 2015-06-01 15:29:
> Il 20/05/2015 10:44, David Gessel ha scritto:
>> Is there a way to completely disable caching on dovecot?
> 
> Yes: mail_location = maildir:~/Maildir:INDEX=MEMORY
dont know if that really solve it, imho it just move the problem from 
disk cache to memory cache, it does not imho disable it

-------- Original Message --------
Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries,
duplicated messages, excessive full re-downloads
From: Benny Pedersen <me at junc.eu>
To: dovecot at dovecot.org
Date: Mon Jun 01 2015 16:47:48 GMT+0300 (Arabic Standard Time)
> Alessio Cecchi skrev den 2015-06-01 15:29:
>> Il 20/05/2015 10:44, David Gessel ha scritto:
>>> Is there a way to completely disable caching on dovecot?
>>
>> Yes: mail_location = maildir:~/Maildir:INDEX=MEMORY
> 
> dont know if that really solve it, imho it just move the problem from disk
cache to memory cache, it does not imho disable it

It seemed to do the trick.  Behavior finally seems normal.  The best evidence: 
At 10:00:00-ish I made the change in the config file and at 10:02:29 I restarted
the server.

It is now after 17:14 in Oakland and the logs are quiet.

 09:00:06 host newsyslog[55292]: logfile turned over due to size>1000K
 09:00:08 imap(user1): Warning: Maildir /mail/dom1.com/user1/: Expunged message
reappeared, giving a new UID (old uid=3558585,
file=1434383291.M430136P55004.host,S=2380,W=2423:2,)
 09:00:08 imap(user1): Warning: Maildir /mail/dom1.com/user1/: Expunged message
reappeared, giving a new UID (old uid=3558586,
file=1434383253.M121801P54997.host,S=27419,W=28395:2,)
 09:01:28 indexer-worker(user1): Warning: Maildir /mail/dom1.com/user1/:
Expunged message reappeared, giving a new UID (old uid=3558587,
file=1434383997.M731644P55286.host,S=7334,W=7513:2,)
 09:07:49 indexer-worker(user1): Warning: Maildir /mail/dom1.com/user1/:
Expunged message reappeared, giving a new UID (old uid=3561318,
file=1434383997.M731644P55286.host,S=7334,W=7513:2,)
 09:08:49 imap(user2): Warning: Maildir /mail/dom2/dmag/: Expunged message
reappeared, giving a new UID (old uid=1846654,
file=1434381278.M451318P54372.host,S=5013,W=5116:2,)
 09:09:48 imap(user1): Warning: Maildir /mail/dom1.com/user1/: Expunged message
reappeared, giving a new UID (old uid=3561319,
file=1405470941.M117568P19536.host.dom1.com,S=8266,W=8382:2,)
 09:21:02 imap(user1): Warning: Maildir /mail/dom1.com/user1/: Expunged message
reappeared, giving a new UID (old uid=3564050,
file=1434384469.M876726P55434.host,S=7297,W=7418:2,)
 09:24:49 imap(user2): Warning: Maildir /mail/dom2/dmag/: Expunged message
reappeared, giving a new UID (old uid=1847030,
file=1434383414.M615990P55039.host,S=9890,W=10269:2,)
 09:24:49 imap(user2): Warning: Maildir /mail/dom2/dmag/: Expunged message
reappeared, giving a new UID (old uid=1847031,
file=1434384445.M934994P55428.host,S=3405,W=3487:2,)
 09:24:49 imap(user2): Warning: Maildir /mail/dom2/dmag/: Expunged message
reappeared, giving a new UID (old uid=1847032,
file=1434381278.M451318P54372.host,S=5013,W=5116:2,)
 09:27:18 indexer-worker(user2): Warning: Maildir /mail/dom2/dmag/: Expunged
message reappeared, giving a new UID (old uid=1847033,
file=1434384562.M535709P55456.host,S=3403,W=3485:2,)
 09:42:41 indexer-worker(user1): Warning: Maildir /mail/dom1.com/user1/:
Expunged message reappeared, giving a new UID (old uid=3565416,
file=1434385257.M437928P55657.host,S=176241,W=178722:2,)
 09:47:18 indexer-worker(user1): Warning: Maildir /mail/dom1.com/user1/:
Expunged message reappeared, giving a new UID (old uid=3568150,
file=1434386561.M197796P56113.host,S=905854,W=917647:2,)
 09:47:18 indexer-worker(user1): Warning: Maildir /mail/dom1.com/user1/:
Expunged message reappeared, giving a new UID (old uid=3568151,
file=1434385257.M437928P55657.host,S=176241,W=178722:2,)
 09:47:23 imap(user1): Warning: Maildir /mail/dom1.com/user1/: Expunged message
reappeared, giving a new UID (old uid=3568152,
file=1434386561.M197796P56113.host,S=905854,W=917647:2,)
 09:53:34 imap(user2): Warning: Maildir /mail/dom2/dmag/: Expunged message
reappeared, giving a new UID (old uid=1847228,
file=1434384562.M535709P55456.host,S=3403,W=3485:2,)
 09:59:22 master: Warning: Killed with signal 15 (by pid=56486 uid=0 code=kill)
 10:00:57 lda(user3): Warning: Maildir /mail/dom1.com/user3/: UIDVALIDITY
changed (1404858902 -> 1433446125)
 10:00:57 indexer-worker(user3): Warning: fts-lucene: Settings have changed,
rebuilding index for mailbox
 10:00:57 indexer-worker(user1): Warning: Maildir /mail/dom1.com/user1/:
Expunged message reappeared, giving a new UID (old uid=3569520,
file=1434386838.M74634P56231.host,S=905827,W=917620:2,)
 10:02:29 master: Warning: Killed with signal 15 (by pid=66696 uid=0 code=kill)
 10:05:13 indexer-worker(user1): Warning: fts-lucene: Settings have changed,
rebuilding index for mailbox
 10:05:13 indexer-worker(user1): Error:
stat(/mail/dom1.com/user1//.dovecot.sieve/tmp) failed: Not a directory
 10:08:09 indexer-worker(user3): Warning: fts-lucene: Settings have changed,
rebuilding index for mailbox
 10:17:00 indexer-worker(user2): Warning: fts-lucene: Settings have changed,
rebuilding index for mailbox
 15:24:16 imap(user1): Error: stat(/mail/dom1.com/user1//.dovecot.sieve/tmp)
failed: Not a directory

The last log file was turned over when it got greater than 1000K.  It covered
1312 minutes and 3726 "expunged Message" warnings or one every 21
seconds.  There hasn't been a single one in the 7+ hours since I moved the
index to MEMORY.

Whatever is confusing the index state on disk, it is not happening in RAM.