dovecot - Dec 2015 - Looks like a bug to me: Dovecot ignores Maildir/new timestamp

Hello!

I run Debian jessie and have an issue with the interaction between
Dovecot (2.2.13), Mutt (1.5.23) and the Android Gmail (5.8.105868218)
IMAP client, when using Maildir:

I use Dovecot LMTP for local delivery into Maildir and access these
files both through Dovecot IMAP from the Android Gmail IMAP client and
also directly through the filesystem by running Mutt locally on the mail
server.

When a new mail arrives, Dovecot LMTP will add a new file in Maildir/new
and also update the Dovecot index cache. If I then open Mutt and just
delete this new message, then Mutt will just delete this file from
Maildir/new and the Dovecot index cache will be out-of-sync.

(For testing, Mutt can be cut out of the equation entirely by instead
just running "rm Maildir/new/*" from the command line to mimic what
Mutt
does when deleting a new message, but I mention the use of Mutt here to
show why this is a real use-case for me.)

If I after this open the Gmail app and pull down the message index to
refresh from the Dovecot IMAP server, it'll get the cached version of
the index, in which the new message is still present (but only the
headers, no body will be loaded and the app will show no body snippet).
Pulling down the message index again to force a second refresh will
cause the new/deleted message to disappear again.

This seems to be caused by Dovecot not looking at the Maildir/new
timestamp to determine whether the index cache is up-to-date, but
instead only looking at the Maildir/cur timestamp.

If I instead repeat all these testing steps from the beginning, sending
a new mail message, deleting it with "rm Maildir/new/*" but then also
running "touch Maildir/cur" before opening the Gmail app and doing the
refresh, then Dovecot does update the cache and does not send any ghost
messages to the IMAP client.

It does seem reasonable to me that Mutt doesn't touch Maildir/cur when
just deleting new messages and I suspect that it might be a bug in
Dovecot that it timestamp of Maildir/new isn't checked and the index
cache updated if that timestamp is newer than the last index update done
by the Dovecot LMTP service.

Does this analysis seem sound? Did I miss anything?

(Output of "doveconf -n" attached.)

Cheers // Fredrik Roubert

-- 
Forsterstrasse 64  |  +41 78 8170377
CH-8044 Z?rich     |  https://roubert.name/fredrik/
-------------- next part --------------
# 2.2.13: /etc/dovecot/dovecot.conf
# OS: Linux 3.16.0-4-amd64 x86_64 Debian 8.2 
auth_username_chars =
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@/
auth_username_format = %Ln
first_valid_uid = 1000
last_valid_uid = 65533
listen = 5.148.172.24, 2a02:418:1003:7::1
mail_location = maildir:~/Maildir
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy
include variables body enotify environment mailbox date ihave editheader
vnd.dovecot.duplicate
mbox_read_locks = flock
mbox_write_locks = flock
namespace inbox {
  inbox = yes
  location = 
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix = 
}
passdb {
  args = scheme=SHA512-CRYPT /etc/dovecot/passwd-imap
  driver = passwd-file
}
passdb {
  args = scheme=SHA512-CRYPT /etc/dovecot/passwd-smtp
  driver = passwd-file
}
plugin {
  sieve = ~/.dovecot.sieve
  sieve_before = /var/lib/dovecot/sieve/spam.sieve
  sieve_dir = ~/.sieve
  sieve_extensions = +editheader +vnd.dovecot.duplicate
}
protocols = " imap lmtp sieve"
service auth {
  unix_listener /var/spool/postfix/private/dovecot-auth {
    group = postfix
    mode = 0666
    user = postfix
  }
}
service imap-login {
  inet_listener imap {
    port = 143
  }
}
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    group = postfix
    mode = 0666
    user = postfix
  }
}
service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
}
ssl = required
ssl_cert = </etc/ssl/startssl/certs/roubert.net.crt
ssl_cipher_list =
EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:+CAMELLIA256:+AES256:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!ECDSA:CAMELLIA256-SHA:AES256-SHA:CAMELLIA128-SHA:AES128-SHA
ssl_dh_parameters_length = 2048
ssl_key = </etc/ssl/startssl/private/roubert.net.key
ssl_prefer_server_ciphers = yes
ssl_protocols = !SSLv3 !SSLv2
userdb {
  args = home=/home/%n allow_all_users=yes
  driver = static
  result_failure = return-fail
  result_internalfail = return-fail
  result_success = continue
}
userdb {
  args = blocking=no
  driver = passwd
}
protocol lmtp {
  mail_plugins = " sieve"
  postmaster_address = postmaster at roubert.net
}
protocol imap {
  passdb {
    args = scheme=SHA512-CRYPT /etc/dovecot/passwd-smtp
    deny = yes
    driver = passwd-file
    name = 
  }
}

On 2015-12-01 11:30:10 +0100, Fredrik Roubert wrote:
> I run Debian jessie and have an issue with the interaction between
> Dovecot (2.2.13), Mutt (1.5.23) and the Android Gmail (5.8.105868218)
> IMAP client, when using Maildir:
As a fellow mutt user. I use mutt with dovecot. mutt headercache was
much slower for me than dovecot's caching. if you dont want to go via
the tcp port, you can set /usr/lib/dovecot/imap as a tunnel in your
muttrc.

hth

    darix

-- 
           openSUSE - SUSE Linux is my linux
               openSUSE is good for you
                   www.opensuse.org

> On 01 Dec 2015, at 12:30, Fredrik Roubert <fredrik at roubert.name>
wrote:
> 
> Hello!
> 
> I run Debian jessie and have an issue with the interaction between
> Dovecot (2.2.13), Mutt (1.5.23) and the Android Gmail (5.8.105868218)
> IMAP client, when using Maildir:
> 
> I use Dovecot LMTP for local delivery into Maildir and access these
> files both through Dovecot IMAP from the Android Gmail IMAP client and
> also directly through the filesystem by running Mutt locally on the mail
> server.
> 
> When a new mail arrives, Dovecot LMTP will add a new file in Maildir/new
> and also update the Dovecot index cache. If I then open Mutt and just
> delete this new message, then Mutt will just delete this file from
> Maildir/new and the Dovecot index cache will be out-of-sync.
> 
> (For testing, Mutt can be cut out of the equation entirely by instead
> just running "rm Maildir/new/*" from the command line to mimic
what Mutt
> does when deleting a new message, but I mention the use of Mutt here to
> show why this is a real use-case for me.)
> 
> If I after this open the Gmail app and pull down the message index to
> refresh from the Dovecot IMAP server, it'll get the cached version of
> the index, in which the new message is still present (but only the
> headers, no body will be loaded and the app will show no body snippet).
> Pulling down the message index again to force a second refresh will
> cause the new/deleted message to disappear again.
> 
> This seems to be caused by Dovecot not looking at the Maildir/new
> timestamp to determine whether the index cache is up-to-date, but
> instead only looking at the Maildir/cur timestamp.
Dovecot does check Maildir/new timestamp, but it'll only perform a
"partial sync" then, which means it adds any newly found mails but it
doesn't delete any mails because it can't be sure that they weren't
moved to cur/. One simple fix for this would be to scan cur/ also whenever new/
changes, but that would unnecessarily make the performance worse for most
people. Another fix would be to remember that the mail was in new/ and when
cur's timestamp hadn't changed and the mail wasn't in new/ anymore
it would be removed from index. But this would require remembering that the mail
is in new/, which isn't currently done and adding it would probably cause
extra disk writes.

So I don't really see a way of fixing this in any nice way.