Hello, I'm testing dovecot with some setups, and one of them is with shared mailboxes. The test I wrote will create and delete mail using multiple connections to the same user and folder. Each connection makes a couple of mails, remembers the uid from APPENDUID, and will delete those emails again. At the end of the test I expect an empty folder. This is not what happens. At the end I still have several mails in the folder. I lack insight in the dovecot source to tell exactly what's going on. I've tested this with different setups: 1) local system user, connecting over localhost -> bug is present 2) local system user, connecting over internet -> bug is present, but is harder to reproduce 3) dovecot as proxy to another imap server -> bug is present In step 3, you can even setup a dovecot to be a proxy to another dovecot server.>From logging in the other imap server I've seen that a client command to theproxy like: TAG UID STORE 1:3 +FLAGS (\Deleted) TAG UID EXPUNGE 1:3 will be sent to the other imap server in 3 steps, one for each message. When running the test with multiple threads, that logging shows that some uids are never sent to the other imap server, and some uids are sent over different connections than they original were sent to. (Thread 1 deletes 1:3, Thread 2 deletes 4:6, the proxy of Thread 1 might expunge messages from Thread 2 and vice versa). Attached is a python script which tests the behavior. The script expects a file named "testmail.eml" to upload to the imap server. I used an email which was about 75 kB. I tested using version: 2.2.22 (fe789d2). Let me know if I can help in any other way too. John -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: dovecot-threading-bug.py URL: <http://dovecot.org/pipermail/dovecot/attachments/20160530/89d17fdc/attachment.ksh> -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5332 bytes Desc: not available URL: <http://dovecot.org/pipermail/dovecot/attachments/20160530/89d17fdc/attachment.p7s>
On 30.05.2016 16:41, van der Kamp, John wrote:> Hello, > > I'm testing dovecot with some setups, and one of them is with shared > mailboxes. The test I wrote will create and delete mail using multiple > connections to the same user and folder. Each connection makes a couple of > mails, remembers the uid from APPENDUID, and will delete those emails again. > At the end of the test I expect an empty folder. > > This is not what happens. At the end I still have several mails in the > folder. I lack insight in the dovecot source to tell exactly what's going > on. I've tested this with different setups: > 1) local system user, connecting over localhost -> bug is present > 2) local system user, connecting over internet -> bug is present, but is > harder to reproduce > 3) dovecot as proxy to another imap server -> bug is present > In step 3, you can even setup a dovecot to be a proxy to another dovecot > server. > > From logging in the other imap server I've seen that a client command to the > proxy like: > TAG UID STORE 1:3 +FLAGS (\Deleted) > TAG UID EXPUNGE 1:3 > will be sent to the other imap server in 3 steps, one for each message. When > running the test with multiple threads, that logging shows that some uids > are never sent to the other imap server, and some uids are sent over > different connections than they original were sent to. (Thread 1 deletes > 1:3, Thread 2 deletes 4:6, the proxy of Thread 1 might expunge messages from > Thread 2 and vice versa). > > Attached is a python script which tests the behavior. The script expects a > file named "testmail.eml" to upload to the imap server. I used an email > which was about 75 kB. > I tested using version: 2.2.22 (fe789d2). > Let me know if I can help in any other way too. > > John >Hi! We tested with 2.2.24, and were unable to reproduce the error. Can you try again with 2.2.24? Aki
On 03/06/2016 07:20, Aki Tuomi wrote:> We tested with 2.2.24, and were unable to reproduce the error. Can you > try again with 2.2.24?Apologies for butting in, but I've been seeing exactly the same issue post upgrade to 2.2.24 (from 2.2.18): [2016-06-02T10:38:28+0100] imap(xxxxx): Error: Corrupted index cache file /mnt/index/8cc/95 2952/.INBOX/dovecot.index.cache: Broken MIME parts for mail UID 13758 in mailbox INBOX: Cached MIME parts don't match message during parsing: Cached header size mismatch (parts=41000000f7020000000000000a030000000000000508000 0000000003108000000000000030000004800000018030000000000005b000000000000005f00000000000000f100000000000000f500000 000000000040000004000000086040000000000002700000000000000290000000000000064010000000000006b010000000000004400000 033060000000000001e000000000000002000000000000000860400000000000097040000000000001100000001000000410000005106000 000000000850400000000000095040000000000000000000000000000000000000000000000000000) etc. At first I assumed it was better at picking up corrupted indexes and would reduce in severity over time. However, I've also seen that attempting to force-resync or remove and rebuild indexes doesn't help - it reoccurs on the same mailboxes. Unfortunately I'm unable to reveal mails from affected mailboxes, so I'm not sure how much help this is beyond a "me too". doveconf (dovecot behind directors) # doveconf -n # 2.2.24 (a82c823): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.14 (099a97c) # OS: Linux 2.6.32-573.26.1.el6.x86_64 x86_64 CentOS release 6.7 (Final) auth_anonymous_username auth_failure_delay = 0 auth_master_user_separator = * auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_ auth_worker_max_count = 64 base_dir = /var/run/dovecot/ default_client_limit = 5120 default_process_limit = 128 default_vsz_limit = 64 M deliver_log_format = msgid=%m from=<%e> (%f) to=<%{to_envelope}>: %$ disable_plaintext_auth = no first_valid_gid = 2000 first_valid_uid = 2000 lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes lda_original_recipient_header = X-Original-To listen = 192.168.0.214 lmtp_hdr_delivery_address = none lmtp_rcpt_check_quota = yes log_path = /var/log/dovecot/dovecot.log log_timestamp = "[%Y-%m-%dT%H:%M:%S%z] " login_greeting = Mail Server Ready login_log_format_elements = pid=%e user=<%u> ip=%r login_trusted_networks = 192.168.0.223 192.168.0.224 192.168.0.225 192.168.0.226 192.168.0.227 192.168.0.228 mail_access_groups = doveuser mail_fsync = always mail_home = /mnt/mail/%3Mu/%u mail_location = maildir:~:INDEX=/mnt/index/%3Mu/%i:CONTROL=/mnt/control/%3Mu/%u mail_plugins = " stats" mailbox_idle_check_interval = 2 mins mailbox_list_index = yes maildir_very_dirty_syncs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii -numeric relational regex imap4flags copy include variables body enotify mailbox date index ihave duplicate mime foreverypart extracttext mmap_disable = yes passdb { args = /etc/dovecot/passwd.master driver = passwd-file master = yes pass = yes result_failure = return-fail result_internalfail = return-fail } passdb { args = /etc/dovecot/sql.d/user.ext driver = sql } plugin { sieve = file:~/sieve;active=~/sieve/.active.sieve sieve_extensions = -environment sieve_max_actions = 32 sieve_max_redirects = 4 sieve_max_script_size = 1M sieve_quota_max_scripts = 16 sieve_quota_max_storage = 2M sieve_redirect_envelope_from = sender sieve_user_log = /dev/null stats_refresh = 30 secs stats_track_cmds = no } protocols = lmtp imap pop3 sieve recipient_delimiter service auth-worker { user = $default_internal_user vsz_limit = 128 M } service auth { client_limit = 7664 vsz_limit = 128 M } service imap-login { inet_listener imap { port = 10143 } service_count = 0 } service imap { process_limit = 5120 process_min_avail = 24 vsz_limit = 512 M } service lmtp { executable = lmtp -L inet_listener lmtp { port = 10024 } process_limit = 128 process_min_avail = 12 vsz_limit = 512 M } service managesieve-login { inet_listener sieve { port = 14190 } service_count = 0 vsz_limit = 512 M } service managesieve { process_limit = 128 } service pop3-login { inet_listener pop3 { port = 10110 } service_count = 0 } service pop3 { process_limit = 2048 process_min_avail = 24 vsz_limit = 512 M } service stats { fifo_listener stats-mail { group = doveuser mode = 0660 } vsz_limit = 256 M } ssl = no stats_command_min_time = 0 stats_ip_min_time = 2 mins stats_memory_limit = 64 M stats_session_min_time = 2 mins stats_user_min_time = 2 mins userdb { args = /etc/dovecot/sql.d/user.ext driver = sql } verbose_proctitle = yes protocol lmtp { auth_username_chars auth_username_format = %Ln log_path = /var/log/dovecot/lmtp.log mail_plugins = " stats sieve" } protocol imap { imap_client_workarounds = delay-newmail imap_id_log = name version os os-version imap_id_send imap_idle_notify_interval = 10 mins imap_logout_format = bytes=%i/%o imap_max_line_length = 64 k mail_max_userip_connections = 16 mail_plugins = " stats imap_stats" } protocol sieve { mail_max_userip_connections = 16 managesieve_logout_format = bytes=%i/%o managesieve_max_compile_errors = 5 } protocol pop3 { mail_max_userip_connections = 16 pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_enable_last = no pop3_fast_size_lookups = no pop3_lock_session = yes pop3_logout_format = bytes=%i/%o, top=%t/%p, retr=%r/%b, del=%d/%m, size=%s pop3_no_flag_updates = no pop3_reuse_xuidl = no pop3_save_uidl = no pop3_uidl_format = %08Xv%08Xu } -- Dave
On 03-06-16 08:20, aki.tuomi at dovecot.fi (Aki Tuomi) wrote:> On 30.05.2016 16:41, van der Kamp, John wrote: >> Hello, >> >> I'm testing dovecot with some setups, and one of them is with shared >> mailboxes. The test I wrote will create and delete mail using multiple >> connections to the same user and folder. Each connection makes a couple of >> mails, remembers the uid from APPENDUID, and will delete those emails again. >> At the end of the test I expect an empty folder. >> >> This is not what happens. At the end I still have several mails in the >> folder. I lack insight in the dovecot source to tell exactly what's going >> on. I've tested this with different setups: >> 1) local system user, connecting over localhost -> bug is present >> 2) local system user, connecting over internet -> bug is present, but is >> harder to reproduce >> 3) dovecot as proxy to another imap server -> bug is present >> In step 3, you can even setup a dovecot to be a proxy to another dovecot >> server. >> >> From logging in the other imap server I've seen that a client command to the >> proxy like: >> TAG UID STORE 1:3 +FLAGS (\Deleted) >> TAG UID EXPUNGE 1:3 >> will be sent to the other imap server in 3 steps, one for each message. When >> running the test with multiple threads, that logging shows that some uids >> are never sent to the other imap server, and some uids are sent over >> different connections than they original were sent to. (Thread 1 deletes >> 1:3, Thread 2 deletes 4:6, the proxy of Thread 1 might expunge messages from >> Thread 2 and vice versa). >> >> Attached is a python script which tests the behavior. The script expects a >> file named "testmail.eml" to upload to the imap server. I used an email >> which was about 75 kB. >> I tested using version: 2.2.22 (fe789d2). >> Let me know if I can help in any other way too. >> >> John >> > Hi! > > We tested with 2.2.24, and were unable to reproduce the error. Can you > try again with 2.2.24? > > Aki >Hi, Sorry for the late reply. Didn't notice when this was picked up. I've tried again with out-of-the-box Ubuntu Xenial 16.04, which ships with 2.2.24. Here the problem is still present. I patched the packages with the commits Timo mentioned in another reply. That did not fix the problem the python script tries to reproduce. I tried 2.2.25 on a Debian testing installation. This too is just out-of-the-box local users with mailbox in homedir configuration. I had to run the script a couple of times before it showed up again. So I guess things have been made better, but not flawless. John