David Gessel
2014-Jul-15 01:12 UTC
Dovecot/K9/Thunderbird IMAP interaction issues: replicated messages, constant reloading (TB) & missing new messages (K-9)
Version Info: FreeBSD 9.2-RELEASE #0 r266046 AMD64, Dovecot 2.2.13,
dovecot-pigeonhole-0.4.3, amavisd-new-2.8.0_2,1, clamav-0.98.4,
postfix29-2.9.9_1,1, postfixadmin-2.3.6, spamassassin-3.4.0_12 (zpool
data and output of dovecot -n at bottom of post)
I suspect these are related problems and further that they are related
to some trivial newb error I've made in my configuration, but searching
descriptions of the symptoms has not yet guided me to a solution.
I recently migrated from Courier to Dovecot on new hardware and imap
synced a 16GB mailstore (du -hs) divided up into annual directories with
about 15,000 messages each. I mention the size because it seems an
errant Remove Duplicate Messages (Alternate) 0.3.9 acting on what seem
to be false duplicates in Thunderbird reduced the inbox size from
7500-ish messages to 1300 (DOH!) and may have solved my K-9 problem
described below. My old Courier server still has the vast majority of
these messages and I will resync someday and then try to file them away
more expediently than my previously somewhat lazy management style to
keep the inbox small.
* In thunderbird, I get constant downloading of messages. Every time I
switch to the inbox from another folder, the messages start
redownloading. This sometimes, but not always, seems to create
duplicate messages, but I'm now fairly sure they are only created in the
client and not on the server. Searching through various forums, it
appears this has been a problem for Thunderbird for some time, but it
wasn't for me on Courier. As posts about similar errors seem to have
petered out over the last year without any clear guidance on how to fix
it that I could find, I expect there's a simple fix I just haven't found
yet, either in TB configuration or in Dovecot. Any hints? I tried
Thunderfix to remove the .msf files and will be redownloading my mail
over the thin pipe out to Iraq for the next 6 months (DOH!). One
artifact is that when I restart thunderbird, my inbox is reported as
having about 130 messages in it, then thunderbird starts redownloading
headers and eventually gets back up to the 1297 currently listed both in
the TB pane and by ls | wc -l in /cur. Is it possible there is a time
out that needs to be extended? In TB, I've set (among other less
apparently relevant options)
mailnews.tcptimeout;1000
mail.db.idle_limit;30000000
* Also, with Thunderbird, I get duplication of messages in the client.
I suspect this is related to the previous redownload error, but there is
some indication in some posts that it is potentially related to a dirty
flag. I've had this problem before with Thunderbird and Courier, but it
went away after a TB update. It seems to have reappeared with Dovecot.
The TB client hasn't been updated during the transition, so some
interaction issue seems more likely than a pure TB bug. Likely TB's
fault as this is a well known, but poorly diagnosed TB issue (deleting
duplicates regularly seems to be the solution, but my inbox would double
in size every few hours on the LAN. Now that I'm remote over a 2mbps
link, it has slowed way down: bonus!).
* "Expunged message reappeared, giving a new UID" errors appear a lot.
This might well be related to the last error. This seems to happen with
TB client connections, but I'm not sure what action triggers it. I am
not replicating. I tried renaming 10-director.conf to
10-director.conf.disabled and got no complaints from Dovecot (as I'm not
using replication, it shouldn't) but that indicates that this Expunged
message reappeared error is probably of a different source than, for
example, Simon Fraser's debug effort.
* K9 was only showing a subset of my messages and generally not the new
ones. They'd appear in the alert properly, but weren't visible in the
client and sometimes they'd be visible and then disappear later. I
imagined there could a correlation between TB checking mail and perhaps
moving messages from /new to /cur and K9 losing track, but the
potentially tragic "duplicate" removal may have reduced my inbox to a
size that k9 can handle as K9 inbox display seems normal now as far as I
can tell. When I resync (assuming I can), I sort of dread this problem
reappearing. K9 worked as expected against Courier with the same inbox.
These apparently difficult to diagnose problems are occasionally
suggested as related to namespace configuration issues and thunderbird
is reported to have issues with some separator options. I vaguely
suspect this, but I'm pretty sure my namespace config is default and
should be well tested. I haven't, for example, explicitly stated the
default "/" but left it commented out. I increased the connection
limit
count per user*IP to 20 as TB seems to be parallel downloading a lot of
folders. This has neither improved nor degraded performance in any way
I can see and is probably irrelevant.
Any hints greatly appreciated.
Filesystem/Zpool data:
NAME PROPERTY VALUE SOURCE
zroot size 400G -
zroot capacity 28% -
zroot altroot - default
zroot health ONLINE -
zroot guid 1061309925223582150 default
zroot version - default
zroot bootfs zroot local
zroot delegation on default
zroot autoreplace off default
zroot cachefile - default
zroot failmode wait default
zroot listsnapshots off default
zroot autoexpand off default
zroot dedupditto 0 default
zroot dedupratio 1.10x -
zroot free 286G -
zroot allocated 114G -
zroot readonly off -
zroot comment - default
zroot expandsize 0 -
zroot freeing 0 default
zroot feature at async_destroy enabled local
zroot feature at empty_bpobj active local
zroot feature at lz4_compress active local
server # dovecot -n
# 2.2.13: /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 9.2-RELEASE amd64
auth_default_realm = mydomain.com
auth_mechanisms = plain login
auth_realms = mydomain.com mydomain2.org
disable_plaintext_auth = no
first_valid_gid = 5000
first_valid_uid = 5000
hostname = mydomain.com
imap_client_workarounds = tb-extra-mailbox-sep tb-lsub-flags
imap_idle_notify_interval = 29 mins
last_valid_gid = 5000
last_valid_uid = 5000
lda_mailbox_autocreate = yes
listen = *
mail_gid = 5000
mail_location = maildir:~/:CONTROL=/var/no-quota/%u
mail_plugins = quota mail_log notify
mail_temp_dir = /var/tmp
mail_uid = 5000
maildir_broken_filename_sizes = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date ihave duplicate spamtest spamtestplus editheader
mbox_lazy_writes = no
namespace inbox {
inbox = yes
location mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix }
passdb {
args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
expire = Trash
mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename
mail_log_fields = uid box msgid size
quota = maildir:User quota
quota_exceeded_message = Storage quota for this account has been
exceeded, please try again later.
quota_rule = *:storage=50G
quota_rule2 = Trash:storage=+30%%
quota_rule3 = Sent:storage=+30%%
quota_warning = storage=90%% quota-warning 90 %u
quota_warning2 = storage=75%% quota-warning 75 %u
sieve = ~/.dovecot.sieve
sieve_before = /usr/local/etc/dovecot/sieve/
sieve_dir = ~/sieve
sieve_extensions = +spamtest +spamtestplus +relational
+comparator-i;ascii-numeric +editheader
}
postmaster_address = gessel at mydomain.com
protocols = imap lmtp sieve sieve
sendmail_path = /usr/local/sbin/sendmail
service auth {
unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0666
user = postfix
}
unix_listener auth-userdb {
group = vmail
mode = 0600
user = vmail
}
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
service_count = 1
vsz_limit = 128 M
}
service quota-warning {
executable = script /usr/local/bin/quota-warning.sh
unix_listener quota-warning {
user = vmail
}
user = vmail
}
ssl = required
ssl_cert = </usr/local/etc/ca/server-cert.pem
ssl_cipher_list
ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES256-SHA:!SSLv2
ssl_key = </usr/local/etc/ca/server-unencrypted-key.pem
ssl_prefer_server_ciphers = yes
userdb {
driver = prefetch
}
userdb {
args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
protocol lda {
mail_plugins = quota mail_log notify sieve
}
protocol imap {
mail_max_userip_connections = 20
mail_plugins = quota mail_log notify imap_quota imap_zlib
}
protocol lmtp {
mail_plugins = quota mail_log notify sieve
}
Alex JOST
2014-Jul-15 10:30 UTC
Dovecot/K9/Thunderbird IMAP interaction issues: replicated messages, constant reloading (TB) & missing new messages (K-9)
Am 15.07.2014 03:12, schrieb David Gessel:> * In thunderbird, I get constant downloading of messages. Every time I > switch to the inbox from another folder, the messages start > redownloading. This sometimes, but not always, seems to create > duplicate messages, but I'm now fairly sure they are only created in the > client and not on the server. Searching through various forums, it > appears this has been a problem for Thunderbird for some time, but it > wasn't for me on Courier. As posts about similar errors seem to have > petered out over the last year without any clear guidance on how to fix > it that I could find, I expect there's a simple fix I just haven't found > yet, either in TB configuration or in Dovecot. Any hints? I tried > Thunderfix to remove the .msf files and will be redownloading my mail > over the thin pipe out to Iraq for the next 6 months (DOH!). One > artifact is that when I restart thunderbird, my inbox is reported as > having about 130 messages in it, then thunderbird starts redownloading > headers and eventually gets back up to the 1297 currently listed both in > the TB pane and by ls | wc -l in /cur. Is it possible there is a time > out that needs to be extended? In TB, I've set (among other less > apparently relevant options) > mailnews.tcptimeout;1000 > mail.db.idle_limit;30000000How old is your Thunderbird profile. Have you tried creating a new one and see if the problem persists? Thunderfix should not be needed by the way. Just right click on the folder => Properties => Repair> server # dovecot -n > # 2.2.13: /usr/local/etc/dovecot/dovecot.conf > # OS: FreeBSD 9.2-RELEASE amd64 > auth_default_realm = mydomain.com > auth_mechanisms = plain login > auth_realms = mydomain.com mydomain2.org > disable_plaintext_auth = no > first_valid_gid = 5000 > first_valid_uid = 5000 > hostname = mydomain.com > imap_client_workarounds = tb-extra-mailbox-sep tb-lsub-flags > imap_idle_notify_interval = 29 mins > last_valid_gid = 5000 > last_valid_uid = 5000 > lda_mailbox_autocreate = yes > listen = * > mail_gid = 5000 > mail_location = maildir:~/:CONTROL=/var/no-quota/%u > mail_plugins = quota mail_log notify > mail_temp_dir = /var/tmp > mail_uid = 5000 > maildir_broken_filename_sizes = yes > managesieve_notify_capability = mailto > managesieve_sieve_capability = fileinto reject envelope > encoded-character vacation subaddress comparator-i;ascii-numeric > relational regex imap4flags copy include variables body enotify > environment mailbox date ihave duplicate spamtest spamtestplus editheader > mbox_lazy_writes = no > namespace inbox { > inbox = yes > location > mailbox Drafts { > special_use = \Drafts > } > mailbox Junk { > special_use = \Junk > } > mailbox Sent { > special_use = \Sent > } > mailbox "Sent Messages" { > special_use = \Sent > } > mailbox Trash { > special_use = \Trash > } > prefix > } > passdb { > args = /usr/local/etc/dovecot/dovecot-sql.conf.ext > driver = sql > } > plugin { > expire = Trash > mail_log_events = delete undelete expunge copy mailbox_delete > mailbox_rename > mail_log_fields = uid box msgid size > quota = maildir:User quota > quota_exceeded_message = Storage quota for this account has been > exceeded, please try again later. > quota_rule = *:storage=50G > quota_rule2 = Trash:storage=+30%% > quota_rule3 = Sent:storage=+30%% > quota_warning = storage=90%% quota-warning 90 %u > quota_warning2 = storage=75%% quota-warning 75 %u > sieve = ~/.dovecot.sieve > sieve_before = /usr/local/etc/dovecot/sieve/ > sieve_dir = ~/sieve > sieve_extensions = +spamtest +spamtestplus +relational > +comparator-i;ascii-numeric +editheader > } > postmaster_address = gessel at mydomain.com > protocols = imap lmtp sieve sieve > sendmail_path = /usr/local/sbin/sendmail > service auth { > unix_listener /var/spool/postfix/private/auth { > group = postfix > mode = 0666 > user = postfix > } > unix_listener auth-userdb { > group = vmail > mode = 0600 > user = vmail > } > } > service managesieve-login { > inet_listener sieve { > port = 4190 > } > service_count = 1 > vsz_limit = 128 M > } > service quota-warning { > executable = script /usr/local/bin/quota-warning.sh > unix_listener quota-warning { > user = vmail > } > user = vmail > } > ssl = required > ssl_cert = </usr/local/etc/ca/server-cert.pem > ssl_cipher_list > ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES256-SHA:!SSLv2 > ssl_key = </usr/local/etc/ca/server-unencrypted-key.pem > ssl_prefer_server_ciphers = yes > userdb { > driver = prefetch > } > userdb { > args = /usr/local/etc/dovecot/dovecot-sql.conf.ext > driver = sql > } > protocol lda { > mail_plugins = quota mail_log notify sieve > } > protocol imap { > mail_max_userip_connections = 20 > mail_plugins = quota mail_log notify imap_quota imap_zlib > } > protocol lmtp { > mail_plugins = quota mail_log notify sieve > } >I don't know if it makes a difference for you as you are not using "LAYOUT=fs" but try to disable those imap_client_workarounds. -- Alex JOST