dovecot - Apr 2015 - Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads

-------- Original Message --------
Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries,
duplicated messages, excessive full re-downloads
From: Timo Sirainen <tss at iki.fi>
To: David Gessel <gessel at blackrosetech.com>
Date: Sat Apr 18 2015 15:48:28 GMT+0300 (Arabic Standard Time)
> No. My best guess is that (your) ZFS+FreeBSD is simply not behaving the way
Dovecot wants filesystems to behave (readdir() sometimes skipping over files). I
don't have any other suggestion than trying a different filesystem or OS. I
haven't heard ZFS having these problems before though.

Or migrating to sdbox.  Absent any further suggestions, that will be my next
step.  My commitment to Maildir is far lower than my commitment to FreeBSD/ZFS. 
It is just a historical preference coming from a decade+ of courier IMAP.

On Apr 18, 2015, at 8:00 AM, David Gessel wrote:
> 
> 
> -------- Original Message --------
> Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries,
duplicated messages, excessive full re-downloads
> From: Timo Sirainen <tss at iki.fi>
> To: David Gessel <gessel at blackrosetech.com>
> Date: Sat Apr 18 2015 15:48:28 GMT+0300 (Arabic Standard Time)
> 
>> No. My best guess is that (your) ZFS+FreeBSD is simply not behaving the
way Dovecot wants filesystems to behave (readdir() sometimes skipping over
files). I don't have any other suggestion than trying a different filesystem
or OS. I haven't heard ZFS having these problems before though.
> 
> 
> Or migrating to sdbox.  Absent any further suggestions, that will be my
next step.  My commitment to Maildir is far lower than my commitment to
FreeBSD/ZFS.  It is just a historical preference coming from a decade+ of
courier IMAP.


What is the output for:

# zpool status

perhaps you have a hardware issue?

Edgar, thanks for the help!

-------- Original Message --------
Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries,
duplicated messages, excessive full re-downloads
From: Edgar Pettijohn III <edgar at pettijohn-web.com>
To: David Gessel <gessel at blackrosetech.com>
Date: Sat Apr 18 2015 16:30:07 GMT+0300 (Arabic Standard Time)
> 
> On Apr 18, 2015, at 8:00 AM, David Gessel wrote:
> 
>>
>>
>> -------- Original Message --------
>> Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched
summaries, duplicated messages, excessive full re-downloads
>> From: Timo Sirainen <tss at iki.fi>
>> To: David Gessel <gessel at blackrosetech.com>
>> Date: Sat Apr 18 2015 15:48:28 GMT+0300 (Arabic Standard Time)
>>
>>> No. My best guess is that (your) ZFS+FreeBSD is simply not behaving
the way Dovecot wants filesystems to behave (readdir() sometimes skipping over
files). I don't have any other suggestion than trying a different filesystem
or OS. I haven't heard ZFS having these problems before though.
>>
>>
>> Or migrating to sdbox.  Absent any further suggestions, that will be my
next step.  My commitment to Maildir is far lower than my commitment to
FreeBSD/ZFS.  It is just a historical preference coming from a decade+ of
courier IMAP.
> 
> 
> 
> What is the output for:
> 
> # zpool status
> 
> perhaps you have a hardware issue?
> 
All good - quite reliable hardware:  IBM x3655 dual AMD quad core, 40GB of ECC
RAM, 8x IBM 2.5" SAS on battery-backed ServeRAID, RAID 6 and (somewhat
pointlessly) ZFS2.  All IBM server diagnostics check, full RAM check run without
errors, no ECC corrections reported, no uncorrectable errors reported.



 # zpool status
  pool: bootpool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        bootpool    ONLINE       0     0     0
          aacd0s1a  ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
  scan: none requested
config:

        NAME            STATE     READ WRITE CKSUM
        zroot           ONLINE       0     0     0
          aacd0s1d.eli  ONLINE       0     0     0

errors: No known data errors

On 18 Apr 2015, at 16:00, David Gessel <gessel at blackrosetech.com>
wrote:
> 
>> No. My best guess is that (your) ZFS+FreeBSD is simply not behaving the
way Dovecot wants filesystems to behave (readdir() sometimes skipping over
files). I don't have any other suggestion than trying a different filesystem
or OS. I haven't heard ZFS having these problems before though.
> 
> 
> Or migrating to sdbox.  Absent any further suggestions, that will be my
next step.  My commitment to Maildir is far lower than my commitment to
FreeBSD/ZFS.  It is just a historical preference coming from a decade+ of
courier IMAP.
Actually as a workaround you could try maildir_very_dirty_syncs=yes which gets
rid of most of the readdir()s.

-------- Original Message --------
Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries,
duplicated messages, excessive full re-downloads
From: Timo Sirainen <tss at iki.fi>
To: David Gessel <gessel at blackrosetech.com>
Date: Sat Apr 18 2015 17:38:02 GMT+0300 (Arabic Standard Time)
> Actually as a workaround you could try maildir_very_dirty_syncs=yes which
gets rid of most of the readdir()s.
Enabled. 

I can't read that option without hearing Bubbles say it.

Might this also help: maildir_broken_filename_sizes = yes

I was a little suspicious of sizes and had considered running your
maildir-size-fix.pl script.  They did come from Courier and after 10 years or
more of operation, the risk of cruftiness is high.  However, I thought it was a
long shot as I IMAP-Synced to the new directory so the directory was built by
dovecot... right?  I mean it wasn't a # mv operation.

-David

THANKS TIMO!

-------- Original Message --------
Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries,
duplicated messages, excessive full re-downloads
From: Timo Sirainen <tss at iki.fi>
To: David Gessel <gessel at blackrosetech.com>
Date: Sat Apr 18 2015 17:38:02 GMT+0300 (Arabic Standard Time)
> Actually as a workaround you could try maildir_very_dirty_syncs=yes which
gets rid of most of the readdir()s.
I will try that right now and report any differences in behavior!

very_dirty_syncs helped a lot - the problems still happen but only about every
30 minutes.  It seems this flag prevents re-indexing on whatever was constantly
inducing it (with every client-check, if there was a change written to the
directory, is my best guess watching the logs go).

I still get all the same symptoms, but far less frequently.

I'm inclined to believe, as trivial as it may be to enumerate, that:

Something is triggering dovecot to believe the indexes need to be rebuilt.

When checking mail during the rebuild, clients get confused by UIDs in
transition.

Today I had to quit TB and restart because duplicates filled my preview pane. 
Mulberry has been better, when it gets confused now, it is usually sufficient to
initiate a manual check or expunge to get the list to update.

I would think that sdbox would alleviate these issues, no?

-------- Original Message --------
Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries,
duplicated messages, excessive full re-downloads
From: Timo Sirainen <tss at iki.fi>
To: David Gessel <gessel at blackrosetech.com>
Date: Sat Apr 18 2015 17:38:02 GMT+0300 (Arabic Standard Time)
> On 18 Apr 2015, at 16:00, David Gessel <gessel at blackrosetech.com>
wrote:
>>
>>> No. My best guess is that (your) ZFS+FreeBSD is simply not behaving
the way Dovecot wants filesystems to behave (readdir() sometimes skipping over
files). I don't have any other suggestion than trying a different filesystem
or OS. I haven't heard ZFS having these problems before though.
>>
>>
>> Or migrating to sdbox.  Absent any further suggestions, that will be my
next step.  My commitment to Maildir is far lower than my commitment to
FreeBSD/ZFS.  It is just a historical preference coming from a decade+ of
courier IMAP.
> 
> Actually as a workaround you could try maildir_very_dirty_syncs=yes which
gets rid of most of the readdir()s.
>