dovecot - Dec 2020 - Very slow mail download/notification with dovecot 2.3.7 and Thunderbird​

Hi!

I know that there are many results if I search for "dovecot thunderbird
very slow" on Google but none of them helped me with my problem. :(? (I
already posted this on February 2020, but forgot to answer. O.o)

I am running Ubuntu 20.04 (4GB RAM, 40GB HDD with EXT4) server with the
following software:

- Dovecot 2.3.7.2
---- All accounts are using Maildir
---- Sieve filters

- Postfix 3.4.13
---- with opendkim, spamassassin, razor, pyzor, ....

And as IMAP client I am using Thunderbird 78.6 on Windows 10.
Sieve is used to sort some incoming mails into my 228 folders (56 INBOX.
folders and its subfolders (output of "find .INBOX.* -maxdepth 4 -type d
-print |wc -l")).? The total size of my Maildir directory is 5.5GB.

My problem is that it always takes very very long to synchronize the
INBOX and its subfolders. Even if there are only a few new messages to
be downloaded. Sometimes Thunderbird outputs something like "downloading
message 1 of 19" and it takes at least 60 seconds to download all of
them. None of these mails is larger than 5kB. And sometimes it seems
that the transfer is suspended for a long time and it takes >5 Minutes
to synchronize only a few mails. :(

And it seems that some folders are never updated in TB. In example I
sort all messages from this list into a
"INBOX.Mailinglists.ML-Dovecot"
subfolder by using a simple sieve rule. But even after 10 minutes TB
thinks that there are no new messages. As soon as I click on the
ML-Dovecot folder I see in the /var/log/dovecot-info.log file that TB
logs on to the server and then displays hundreds, of new messages. I
checked twice but I have subscribed to all 228 folders.

How can I find the bottleneck?
Is there a test tool available that simulates a simple mail client and
can analyse an IMAP server?

This is the output of "dovecot -n":

==============================# 2.3.7.2 (3c910f64b): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.7.2 ()
# OS: Linux 5.4.0-54-generic x86_64 Ubuntu 20.04.1 LTS
# Hostname: mail.XXXXXXX.de
auth_mechanisms = plain login
debug_log_path = /var/log/dovecot-debug.log
default_client_limit = 15653
default_process_limit = 5200
first_valid_uid = 150
info_log_path = /var/log/dovecot-info.log
last_valid_uid = 150
lda_original_recipient_header = X-Original-To
log_path = /var/log/dovecot-error.log
login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
%k
mail_gid = mail
mail_home = /var/vmail/%d/%n
mail_location = maildir:~/Maildir
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart extracttext
namespace inbox {
? inbox = yes
? location ? mailbox Archive {
??? special_use = \Archive
? }
? mailbox Drafts {
??? special_use = \Drafts
? }
? mailbox Junk {
??? special_use = \Junk
? }
? mailbox Sent {
??? special_use = \Sent
? }
? mailbox "Sent Messages" {
??? special_use = \Sent
? }
? mailbox Trash {
??? special_use = \Trash
? }
? prefix }
passdb {
? args = /etc/dovecot/dovecot-sql.conf.ext
? driver = sql
}
plugin {
? sieve = ~/.dovecot.sieve
? sieve_default = /var/vmail/global.sieve
? sieve_dir = ~/sieve
}
postmaster_address = thorsten at MY_DOMAIN.TLD
protocols = " imap sieve pop3"
service auth {
? unix_listener /var/spool/postfix/private/auth {
??? group = postfix
??? mode = 0666
??? user = postfix
? }
? unix_listener auth-userdb {
??? group = mail
??? mode = 0666
??? user = vmail
? }
}
service imap-login {
? inet_listener imap {
??? port = 143
? }
? inet_listener imaps {
??? port = 993
??? ssl = yes
? }
? process_limit = 50
? process_min_avail = 2
? service_count = 20
}
service imap {
? process_limit = 1024
}
service managesieve-login {
? inet_listener sieve {
??? port = 4190
? }
? service_count = 0
}
ssl_cert = </etc/ssl/mail/mail.XXXXXXX.de.fullchain.pem
ssl_cipher_list
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
ssl_min_protocol = TLSv1.2
syslog_facility = local5
userdb {
? args = /etc/dovecot/dovecot-sql.conf.ext
? driver = sql
}
protocol lda {
? deliver_log_format = msgid=%m: %$
? mail_plugins = " sieve"
? rejection_reason = Your message to <%t> was automatically rejected:%n%r
}
protocol imap {
? imap_client_workarounds = tb-extra-mailbox-sep
? imap_idle_notify_interval = 10 mins
? mail_max_userip_connections = 100
? mail_plugins }
protocol sieve {
? info_log_path = /var/log/dovecot-sieve.log
? mail_max_userip_connections = 100
? managesieve_implementation_string = Dovecot Pigeonhole
? managesieve_max_line_length = 65536
==============================
Output of "free -h":
==============================free -h
????????????? total??????? used??????? free????? shared? buff/cache??
available
Mem:????????? 3.8Gi?????? 1.7Gi?????? 1.1Gi??????? 10Mi??????
958Mi?????? 1.7Gi
Swap:???????? 1.0Gi?????? 375Mi?????? 648Mi
==============================
Thank you very much!
Thorsten

On 27/12/2020 15:11, ml_dovecot at thorsten-reichelt.de wrote:
> And it seems that some folders are never updated in TB. In example I
> sort all messages from this list into a
"INBOX.Mailinglists.ML-Dovecot"
> subfolder by using a simple sieve rule. But even after 10 minutes TB
> thinks that there are no new messages. As soon as I click on the
> ML-Dovecot folder I see in the /var/log/dovecot-info.log file that TB
> logs on to the server and then displays hundreds, of new messages. I
> checked twice but I have subscribed to all 228 folders.
I can't help you with your performance problem, but for Thunderbird to
check all folders, you need to set
"mail.check_all_imap_folders_for_new"
to 'true' in the Thunderbird config editor.


Regards,
  Juri