David Gessel
2015-Apr-18 13:00 UTC
Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads
-------- Original Message -------- Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads From: Timo Sirainen <tss at iki.fi> To: David Gessel <gessel at blackrosetech.com> Date: Sat Apr 18 2015 15:48:28 GMT+0300 (Arabic Standard Time)> No. My best guess is that (your) ZFS+FreeBSD is simply not behaving the way Dovecot wants filesystems to behave (readdir() sometimes skipping over files). I don't have any other suggestion than trying a different filesystem or OS. I haven't heard ZFS having these problems before though.Or migrating to sdbox. Absent any further suggestions, that will be my next step. My commitment to Maildir is far lower than my commitment to FreeBSD/ZFS. It is just a historical preference coming from a decade+ of courier IMAP.
Edgar Pettijohn III
2015-Apr-18 13:30 UTC
Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads
On Apr 18, 2015, at 8:00 AM, David Gessel wrote:> > > -------- Original Message -------- > Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads > From: Timo Sirainen <tss at iki.fi> > To: David Gessel <gessel at blackrosetech.com> > Date: Sat Apr 18 2015 15:48:28 GMT+0300 (Arabic Standard Time) > >> No. My best guess is that (your) ZFS+FreeBSD is simply not behaving the way Dovecot wants filesystems to behave (readdir() sometimes skipping over files). I don't have any other suggestion than trying a different filesystem or OS. I haven't heard ZFS having these problems before though. > > > Or migrating to sdbox. Absent any further suggestions, that will be my next step. My commitment to Maildir is far lower than my commitment to FreeBSD/ZFS. It is just a historical preference coming from a decade+ of courier IMAP.What is the output for: # zpool status perhaps you have a hardware issue?
David Gessel
2015-Apr-18 14:09 UTC
Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads
Edgar, thanks for the help! -------- Original Message -------- Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads From: Edgar Pettijohn III <edgar at pettijohn-web.com> To: David Gessel <gessel at blackrosetech.com> Date: Sat Apr 18 2015 16:30:07 GMT+0300 (Arabic Standard Time)> > On Apr 18, 2015, at 8:00 AM, David Gessel wrote: > >> >> >> -------- Original Message -------- >> Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads >> From: Timo Sirainen <tss at iki.fi> >> To: David Gessel <gessel at blackrosetech.com> >> Date: Sat Apr 18 2015 15:48:28 GMT+0300 (Arabic Standard Time) >> >>> No. My best guess is that (your) ZFS+FreeBSD is simply not behaving the way Dovecot wants filesystems to behave (readdir() sometimes skipping over files). I don't have any other suggestion than trying a different filesystem or OS. I haven't heard ZFS having these problems before though. >> >> >> Or migrating to sdbox. Absent any further suggestions, that will be my next step. My commitment to Maildir is far lower than my commitment to FreeBSD/ZFS. It is just a historical preference coming from a decade+ of courier IMAP. > > > > What is the output for: > > # zpool status > > perhaps you have a hardware issue? >All good - quite reliable hardware: IBM x3655 dual AMD quad core, 40GB of ECC RAM, 8x IBM 2.5" SAS on battery-backed ServeRAID, RAID 6 and (somewhat pointlessly) ZFS2. All IBM server diagnostics check, full RAM check run without errors, no ECC corrections reported, no uncorrectable errors reported. # zpool status pool: bootpool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM bootpool ONLINE 0 0 0 aacd0s1a ONLINE 0 0 0 errors: No known data errors pool: zroot state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 aacd0s1d.eli ONLINE 0 0 0 errors: No known data errors
Timo Sirainen
2015-Apr-18 14:38 UTC
Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads
On 18 Apr 2015, at 16:00, David Gessel <gessel at blackrosetech.com> wrote:> >> No. My best guess is that (your) ZFS+FreeBSD is simply not behaving the way Dovecot wants filesystems to behave (readdir() sometimes skipping over files). I don't have any other suggestion than trying a different filesystem or OS. I haven't heard ZFS having these problems before though. > > > Or migrating to sdbox. Absent any further suggestions, that will be my next step. My commitment to Maildir is far lower than my commitment to FreeBSD/ZFS. It is just a historical preference coming from a decade+ of courier IMAP.Actually as a workaround you could try maildir_very_dirty_syncs=yes which gets rid of most of the readdir()s.
David Gessel
2015-Apr-18 17:35 UTC
Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads
-------- Original Message -------- Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads From: Timo Sirainen <tss at iki.fi> To: David Gessel <gessel at blackrosetech.com> Date: Sat Apr 18 2015 17:38:02 GMT+0300 (Arabic Standard Time)> Actually as a workaround you could try maildir_very_dirty_syncs=yes which gets rid of most of the readdir()s.Enabled. I can't read that option without hearing Bubbles say it. Might this also help: maildir_broken_filename_sizes = yes I was a little suspicious of sizes and had considered running your maildir-size-fix.pl script. They did come from Courier and after 10 years or more of operation, the risk of cruftiness is high. However, I thought it was a long shot as I IMAP-Synced to the new directory so the directory was built by dovecot... right? I mean it wasn't a # mv operation. -David
David Gessel
2015-Apr-18 17:35 UTC
Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads
THANKS TIMO! -------- Original Message -------- Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads From: Timo Sirainen <tss at iki.fi> To: David Gessel <gessel at blackrosetech.com> Date: Sat Apr 18 2015 17:38:02 GMT+0300 (Arabic Standard Time)> Actually as a workaround you could try maildir_very_dirty_syncs=yes which gets rid of most of the readdir()s.I will try that right now and report any differences in behavior!
David Gessel
2015-Apr-23 14:39 UTC
Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads
very_dirty_syncs helped a lot - the problems still happen but only about every 30 minutes. It seems this flag prevents re-indexing on whatever was constantly inducing it (with every client-check, if there was a change written to the directory, is my best guess watching the logs go). I still get all the same symptoms, but far less frequently. I'm inclined to believe, as trivial as it may be to enumerate, that: Something is triggering dovecot to believe the indexes need to be rebuilt. When checking mail during the rebuild, clients get confused by UIDs in transition. Today I had to quit TB and restart because duplicates filled my preview pane. Mulberry has been better, when it gets confused now, it is usually sufficient to initiate a manual check or expunge to get the list to update. I would think that sdbox would alleviate these issues, no? -------- Original Message -------- Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads From: Timo Sirainen <tss at iki.fi> To: David Gessel <gessel at blackrosetech.com> Date: Sat Apr 18 2015 17:38:02 GMT+0300 (Arabic Standard Time)> On 18 Apr 2015, at 16:00, David Gessel <gessel at blackrosetech.com> wrote: >> >>> No. My best guess is that (your) ZFS+FreeBSD is simply not behaving the way Dovecot wants filesystems to behave (readdir() sometimes skipping over files). I don't have any other suggestion than trying a different filesystem or OS. I haven't heard ZFS having these problems before though. >> >> >> Or migrating to sdbox. Absent any further suggestions, that will be my next step. My commitment to Maildir is far lower than my commitment to FreeBSD/ZFS. It is just a historical preference coming from a decade+ of courier IMAP. > > Actually as a workaround you could try maildir_very_dirty_syncs=yes which gets rid of most of the readdir()s. >
Reasonably Related Threads
- Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads
- Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads
- Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads
- Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads
- Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads