dovecot - Oct 2022 - Unseen field reported by imap status command returns wrong count for shared mailboxes on dovecot cluster

Hello,

I want to make a correction on my first post. We are using version 
2.3.19.1 for all our dovecot servers. Is there any update regarding this 
issue? I would be happy to provide more details if necessary.

Best Regards,

Nikos Pyrgiotis

On 8/9/22 17:47, Nikolaos Pyrgiotis wrote:
> Hello,
> We recently migrated our mail server to a dovecot cluster of 5nodes, a 
> dovecot proxy, 2directors and 2dovecot backends.
> All dovecot nodes run version 2.19.1. We use a glusterfs mounted 
> volume on the backends forthe mail storage.
> We noticed that when issuing the IMAP command to check forUNSEEN 
> messages formails on the shared namespace instead of seeing the value 
> of the unseen messages of the user that the mailbox has been shared to,
> the value of the unseen messages of the owner of the mailbox is returned.
> This behavior causes thunderbird when getting new messages forshared 
> mailboxes, to show briefly all messages of the mailbox as unseen 
> forthe user before showing the correct unseen count forthe shared 
> mailbox.
> The mail location of the shared mailboxes is defined with imapc 
> storage type as described inthe documentation.
> Is this a dovecot bug? Can we configure a different dovecot 
> configuration setting so that imap status command reports the correct 
> unseen field count when mail location is an imapc storage location?
> An examples is given below when running doveadm command from one the 2 
> dovecot directors:
> When issuing the status command we see that the unseen count is 2:
> root at doved0-rmt0-cn1:/etc/cron.d# doveadm mailbox status -u npyrgiotis 
> all shared.sysadmins doveadm(npyrgiotis): Info: 
> remote(10.101.0.71:8080): 
> doveadm(npyrgiotis at
domie02.com)<19078><9l8IJTb+GWOGSgAAEU9A+w>:
> imapc(10.101.0.75:143): Connected to 10.101.0.75:143 (local 
> 10.101.0.71:33476) shared.sysadmins messages=2 recent=0 uidnext=3 
> uidvalidity=1662640492 unseen=2 highestmodseq=3 vsize=3950 
> guid=c92f64f79f0d1ed01e6d5b314f04886c firstsaved=1662643853
> shared.sysadmins 
>
messages=2recent=0uidnext=3uidvalidity=1662640492unseen=2highestmodseq=3vsize=3950guid=c92f64f79f0d1ed01e6d5b314f04886c
> firstsaved=1662643853
> But when fetching the emails of the mailboxes we can see that the 
> \Seen flag is set for both emails
> root at doved0-rmt0-cn1:/etc/cron.d# doveadm fetch -u npyrgiotis flags 
> mailbox shared.sysadmins ALL doveadm(npyrgiotis): Info: 
> remote(10.101.0.71:8080): 
> doveadm(npyrgiotis at
domie02.com)<19074><e7rWDgj+GWOCSgAAEU9A+w>:
> imapc(10.101.0.75:143): Connected to 10.101.0.75:143 (local 
> 10.101.0.71:38750) flags: \Seen flags: \Seen
> Below i post the dovecot configuration of one of the two dovecot backends:
> # 2.3.19.1 (9b53102964): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.5.19 (4eae2f79)
> # OS: Linux 5.10.0-17-amd64 x86_64 Debian 11.4
> # Hostname: doveb0-rmt0-cn1
> auth_cache_negative_ttl = 5mins
> auth_cache_size = 50M
> auth_debug = yes
> auth_default_realm = example.com
> auth_master_user_separator = *
> auth_mechanisms = plain login
> auth_verbose = yes
> auth_worker_max_count = 16
> disable_plaintext_auth = no
> first_valid_uid = 499
> hostname = smtp.example.com
> imapc_features = fetch-bodystructure fetch-headers rfc822.size search 
> modseq acl delay-login
> imapc_host = 10.101.0.75
> imapc_password = # hidden, use -P to show it
> imapc_sasl_mechanisms = plain login
> imapc_ssl = starttls
> imapc_ssl_verify = no
> last_valid_uid = 499
> lda_mailbox_autocreate = yes
> lda_mailbox_autosubscribe = yes
> login_greeting = You have successfully logged into example.com IMAP server
> login_trusted_networks = 10.101.0.73 10.101.0.74
> mail_always_cache_fields = flags hdr.* date.received date.sent
> mail_cache_fields = flags date.received guid size.physical 
> size.virtual imap.bodystructure body.snippet
> mail_debug = yes
> mail_fsync = always
> mail_plugins = " notify mail_log zlib acl"
> mail_privileged_group = mail
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope 
> encoded-character vacation subaddress comparator-i;ascii-numeric 
> relational regex imap4flags copy include variables body enotify 
> environment mailbox date index ihave duplicate mime foreverypart 
> extracttext
> mdbox_rotate_size = 200M
> mmap_disable = yes
> namespace example {
> list = children
> location = imapc:~/shared/%%n:INDEXPVT=~/shared-pvt/%%n
> prefix = shared.%%n.
> separator = .
> subscriptions = no
> type = shared
> }
> namespace inbox {
> inbox = yes
> location > mailbox Drafts {
> special_use = \Drafts
> }
> mailbox Junk {
> special_use = \Junk
> }
> mailbox Sent {
> special_use = \Sent
> }
> mailbox "Sent Messages" {
> special_use = \Sent
> }
> mailbox Spam.Virus {
> auto = subscribe
> }
> mailbox Spam {
> auto = subscribe
> }
> mailbox Trash {
> special_use = \Trash
> }
> prefix > separator = .
> subscriptions = yes
> type = private
> }
> passdb {
> args = password=#hidden_use-P_to_show#
> driver = static
> master = yes
> override_fields = 
> allow_real_nets=127.0.0.0/8,local,10.101.0.73,10.101.0.74 
> userdb_acl_user=%{auth_user} userdb_namespace/example/disabled=yes
> result_success = continue
> }
> passdb {
> args = password=#hidden_use-P_to_show#
> driver = static
> override_fields = 
> allow_real_nets=127.0.0.0/8,local,10.101.0.73,10.101.0.74
> result_success = continue
> }
> passdb {
> args = /etc/dovecot/dovecot-ldap-users.conf.ext
> driver = ldap
> }
> passdb {
> args = /etc/dovecot/dovecot-ldap-groups.conf.ext
> driver = ldap
> }
> plugin {
> acl = vfile:/etc/dovecot/global-acls/%d/master-dovecot-acl:cache_secs=300
> acl_defaults_from_inbox = yes
> acl_globals_only = no
> acl_shared_dict = file:/var/dovecot-acl/%d/shared-mailboxes.db
> imap_compress_deflate_level = 6
> mail_log_events = flag_change
> mail_log_fields = uid box flags
> sieve = file:~/sieve;active=~/sieve/%n.sieve
> sieve_before = /var/vmail/global-sieve/global.sieve
> sieve_env_email_address = %u
> sieve_env_mailbox_prefix = %{userdb:mailbox_prefix}
> sieve_global = /var/vmail/sieve
> sieve_global_extensions = +editheader +vnd.dovecot.environment
> zlib_save = zstd
> zlib_save_level = 5
> }
> protocols = " imap lmtp sieve submission"
> service auth-worker {
> user = dovecot
> }
> service auth {
> unix_listener auth-userdb {
> group = mail
> mode = 0666
> user = vmail
> }
> }
> service doveadm {
> inet_listener dovadm-dom-mgmt {
> address = 10.100.1.71
> port = 8080
> ssl = no
> }
> inet_listener dovadm-dom {
> address = 10.101.0.71
> port = 8080
> ssl = no
> }
> vsz_limit = 256M
> }
> service imap-login {
> inet_listener imap-dom {
> address = 10.101.0.71
> port = 143
> ssl = no
> }
> inet_listener imap {
> port = 0
> }
> inet_listener imaps-dom {
> address = 10.101.0.71
> port = 993
> ssl = yes
> }
> inet_listener imaps {
> port = 0
> }
> process_limit = 400
> process_min_avail = 2
> service_count = 20
> vsz_limit = 512M
> }
> service imap {
> process_limit = 2048
> process_min_avail = 0
> service_count = 1
> vsz_limit = 4G
> }
> service lmtp {
> inet_listener lmtp-dom {
> address = 10.101.0.71
> port = 24
> ssl = no
> }
> process_limit = 0
> process_min_avail = 0
> service_count = 0
> unix_listener lmtp {
> mode = 0666
> }
> vsz_limit = 256M
> }
> service managesieve-login {
> inet_listener sieve-dom {
> address = 10.101.0.71
> port = 4190
> ssl = no
> }
> inet_listener sieve {
> port = 0
> }
> process_limit = 0
> process_min_avail = 0
> service_count = 0
> vsz_limit = 64M
> }
> service managesieve {
> process_limit = 1024
> process_min_avail = 0
> service_count = 1
> vsz_limit = 256M
> }
> service submission-login {
> inet_listener submission-dom {
> address = 10.101.0.71
> port = 587
> ssl = no
> }
> inet_listener submission {
> port = 0
> }
> inet_listener submissions-dom {
> address = 10.101.0.71
> port = 465
> ssl = yes
> }
> process_limit = 250
> process_min_avail = 2
> service_count = 1
> vsz_limit = 256M
> }
> service submission {
> process_limit = 1024
> process_min_avail = 0
> service_count = 1
> vsz_limit = 256M
> }
> ssl = required
> ssl_cert = </etc/ssl/certs/doveb0-rmt0-cn1.mail.rmt.example.com.pem
> ssl_client_ca_dir = /etc/ssl/certs
> ssl_key = # hidden, use -P to show it
> submission_max_mail_size = 30000k
> submission_relay_host = 10.101.0.7
> submission_relay_port = 587
> submission_relay_ssl = starttls
> submission_relay_ssl_verify = no
> submission_relay_trusted = yes
> userdb {
> driver = prefetch
> }
> userdb {
> args = /etc/dovecot/dovecot-ldap-users.conf.ext
> driver = ldap
> }
> userdb {
> args = /etc/dovecot/dovecot-ldap-groups.conf.ext
> driver = ldap
> }
> protocol lmtp {
> mail_plugins = " notify mail_log zlib acl sieve acl"
> }
> protocol lda {
> mail_plugins = " notify mail_log zlib acl sieve acl"
> }
> protocol imap {
> mail_max_userip_connections = 100
> mail_plugins = " notify mail_log zlib acl"
> }
> protocol sieve {
> mail_max_userip_connections = 40
> }
> protocol submission {
> mail_max_userip_connections = 70
> }
> local10.101.0.71 {
> doveadm_allowed_commands > doveadm_password = # hidden, use -P to show
it
> }
> local10.100.1.71 {
> doveadm_allowed_commands > doveadm_password = # hidden, use -P to show
it
> }
> Thank you,
> Nikolaos Pyrgiotis
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20221007/76ce2458/attachment-0001.htm>

Hi!

This seems to be a bug in imapc client, we'll look into this. Thank you for
reporting this issue. It's currently tracked as DOV-5579.

Aki
> On 07/10/2022 15:38 EEST Nikolaos Pyrgiotis <npyrgiotis at
modulus.gr> wrote:
> 
> 
> Hello, 
> 
> I want to make a correction on my first post. We are using version 2.3.19.1
for all our dovecot servers. Is there any update regarding this issue? I would
be happy to provide more details if necessary.
> 
> Best Regards,
> Nikos Pyrgiotis
> 
> On 8/9/22 17:47, Nikolaos Pyrgiotis wrote:
> 
> > Hello,
> > We recently migrated our mail server to a dovecot cluster of5nodes, a
dovecot proxy,2directors and2dovecot backends.
> > All dovecot nodes run version2.19.1. We use a glusterfs mounted volume
on the backendsforthe mail storage.
> > We noticed that when issuing the IMAP command to checkforUNSEEN
messagesformails on the shared namespace instead of seeing the value of the
unseen messages of the user that the mailbox has been shared to,
> > the value of the unseen messages of the owner of the mailbox is
returned.
> > This behavior causes thunderbird when getting new messagesforshared
mailboxes, to show briefly all messages of the mailbox as unseenforthe user
before showing the correct unseen countforthe shared mailbox.
> > The mail location of the shared mailboxes is defined with imapc
storage type as describedinthe documentation.
> > Is this a dovecot bug? Can we configure a different dovecot
configuration setting so that imap status command reports the correct unseen
field count when mail location is an imapc storage location?
> > 
> > An examples is given below when running doveadm command from one the 2
dovecot directors:
> > When issuing the status command we see that the unseen count is2:
> > root at doved0-rmt0-cn1:/etc/cron.d# doveadm mailbox status -u
npyrgiotis all shared.sysadmins doveadm(npyrgiotis): Info:
remote(10.101.0.71:8080): doveadm(npyrgiotis at
domie02.com)<19078><9l8IJTb+GWOGSgAAEU9A+w>: imapc(10.101.0.75:143):
Connected to 10.101.0.75:143 (local 10.101.0.71:33476) shared.sysadmins
messages=2 recent=0 uidnext=3 uidvalidity=1662640492 unseen=2 highestmodseq=3
vsize=3950 guid=c92f64f79f0d1ed01e6d5b314f04886c firstsaved=1662643853
> > shared.sysadmins
messages=2recent=0uidnext=3uidvalidity=1662640492unseen=2highestmodseq=3vsize=3950guid=c92f64f79f0d1ed01e6d5b314f04886c
firstsaved=1662643853
> > 
> > But when fetching the emails of the mailboxes we can see that the
\Seen flag is set for both emails
> > 
> > root at doved0-rmt0-cn1:/etc/cron.d# doveadm fetch -u npyrgiotis flags
mailbox shared.sysadmins ALL doveadm(npyrgiotis): Info:
remote(10.101.0.71:8080): doveadm(npyrgiotis at
domie02.com)<19074><e7rWDgj+GWOCSgAAEU9A+w>: imapc(10.101.0.75:143):
Connected to 10.101.0.75:143 (local 10.101.0.71:38750) flags: \Seen flags: \Seen
> > Below i post the dovecot configuration of one of the two dovecot
backends:
> > #2.3.19.1 (9b53102964): /etc/dovecot/dovecot.conf
> > # Pigeonhole version0.5.19 (4eae2f79)
> > # OS: Linux5.10.0-17-amd64 x86_64 Debian11.4
> > # Hostname: doveb0-rmt0-cn1
> > auth_cache_negative_ttl =5mins
> > auth_cache_size =50M
> > auth_debug = yes
> > auth_default_realm = example.com
> > auth_master_user_separator = *
> > auth_mechanisms = plain login
> > auth_verbose = yes
> > auth_worker_max_count =16
> > disable_plaintext_auth = no
> > first_valid_uid =499
> > hostname = smtp.example.com
> > imapc_features = fetch-bodystructure fetch-headers rfc822.size search
modseq acl delay-login
> > imapc_host =10.101.0.75
> > imapc_password = # hidden, use -P to show it
> > imapc_sasl_mechanisms = plain login
> > imapc_ssl = starttls
> > imapc_ssl_verify = no
> > last_valid_uid =499
> > lda_mailbox_autocreate = yes
> > lda_mailbox_autosubscribe = yes
> > login_greeting = You have successfully loggedinto example.com IMAP
server
> > login_trusted_networks =10.101.0.7310.101.0.74
> > mail_always_cache_fields = flags hdr.* date.received date.sent
> > mail_cache_fields = flags date.received guid size.physical
size.virtual imap.bodystructure body.snippet
> > mail_debug = yes
> > mail_fsync = always
> > mail_plugins =" notify mail_log zlib acl"
> > mail_privileged_group = mail
> > managesieve_notify_capability = mailto
> > managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric relational
regex imap4flags copy include variables body enotify environment mailbox date
index ihave duplicate mime foreverypart extracttext
> > mdbox_rotate_size =200M
> > mmap_disable = yes
> > namespace example {
> > list = children
> > location = imapc:~/shared/%%n:INDEXPVT=~/shared-pvt/%%n
> > prefix = shared.%%n.
> > separator = .
> > subscriptions = no
> > type = shared
> > }
> > namespace inbox {
> > inbox = yes
> > location > > mailbox Drafts {
> > special_use = \Drafts
> > }
> > mailbox Junk {
> > special_use = \Junk
> > }
> > mailbox Sent {
> > special_use = \Sent
> > }
> > mailbox"Sent Messages"{
> > special_use = \Sent
> > }
> > mailbox Spam.Virus {
> > auto = subscribe
> > }
> > mailbox Spam {
> > auto = subscribe
> > }
> > mailbox Trash {
> > special_use = \Trash
> > }
> > prefix > > separator = .
> > subscriptions = yes
> > type = private
> > }
> > passdb {
> > args = password=#hidden_use-P_to_show#
> > driver = static
> > master = yes
> > override_fields =
allow_real_nets=127.0.0.0/8,local,10.101.0.73,10.101.0.74
userdb_acl_user=%{auth_user} userdb_namespace/example/disabled=yes
> > result_success = continue
> > }
> > passdb {
> > args = password=#hidden_use-P_to_show#
> > driver = static
> > override_fields =
allow_real_nets=127.0.0.0/8,local,10.101.0.73,10.101.0.74
> > result_success = continue
> > }
> > passdb {
> > args = /etc/dovecot/dovecot-ldap-users.conf.ext
> > driver = ldap
> > }
> > passdb {
> > args = /etc/dovecot/dovecot-ldap-groups.conf.ext
> > driver = ldap
> > }
> > plugin {
> > acl =
vfile:/etc/dovecot/global-acls/%d/master-dovecot-acl:cache_secs=300
> > acl_defaults_from_inbox = yes
> > acl_globals_only = no
> > acl_shared_dict = file:/var/dovecot-acl/%d/shared-mailboxes.db
> > imap_compress_deflate_level =6
> > mail_log_events = flag_change
> > mail_log_fields = uid box flags
> > sieve = file:~/sieve;active=~/sieve/%n.sieve
> > sieve_before = /var/vmail/global-sieve/global.sieve
> > sieve_env_email_address = %u
> > sieve_env_mailbox_prefix = %{userdb:mailbox_prefix}
> > sieve_global = /var/vmail/sieve
> > sieve_global_extensions = +editheader +vnd.dovecot.environment
> > zlib_save = zstd
> > zlib_save_level =5
> > }
> > protocols =" imap lmtp sieve submission"
> > service auth-worker {
> > user = dovecot
> > }
> > service auth {
> > unix_listener auth-userdb {
> > group = mail
> > mode =0666
> > user = vmail
> > }
> > }
> > service doveadm {
> > inet_listener dovadm-dom-mgmt {
> > address =10.100.1.71
> > port =8080
> > ssl = no
> > }
> > inet_listener dovadm-dom {
> > address =10.101.0.71
> > port =8080
> > ssl = no
> > }
> > vsz_limit =256M
> > }
> > service imap-login {
> > inet_listener imap-dom {
> > address =10.101.0.71
> > port =143
> > ssl = no
> > }
> > inet_listener imap {
> > port =0
> > }
> > inet_listener imaps-dom {
> > address =10.101.0.71
> > port =993
> > ssl = yes
> > }
> > inet_listener imaps {
> > port =0
> > }
> > process_limit =400
> > process_min_avail =2
> > service_count =20
> > vsz_limit =512M
> > }
> > service imap {
> > process_limit =2048
> > process_min_avail =0
> > service_count =1
> > vsz_limit =4G
> > }
> > service lmtp {
> > inet_listener lmtp-dom {
> > address =10.101.0.71
> > port =24
> > ssl = no
> > }
> > process_limit =0
> > process_min_avail =0
> > service_count =0
> > unix_listener lmtp {
> > mode =0666
> > }
> > vsz_limit =256M
> > }
> > service managesieve-login {
> > inet_listener sieve-dom {
> > address =10.101.0.71
> > port =4190
> > ssl = no
> > }
> > inet_listener sieve {
> > port =0
> > }
> > process_limit =0
> > process_min_avail =0
> > service_count =0
> > vsz_limit =64M
> > }
> > service managesieve {
> > process_limit =1024
> > process_min_avail =0
> > service_count =1
> > vsz_limit =256M
> > }
> > service submission-login {
> > inet_listener submission-dom {
> > address =10.101.0.71
> > port =587
> > ssl = no
> > }
> > inet_listener submission {
> > port =0
> > }
> > inet_listener submissions-dom {
> > address =10.101.0.71
> > port =465
> > ssl = yes
> > }
> > process_limit =250
> > process_min_avail =2
> > service_count =1
> > vsz_limit =256M
> > }
> > service submission {
> > process_limit =1024
> > process_min_avail =0
> > service_count =1
> > vsz_limit =256M
> > }
> > ssl = required
> > ssl_cert = </etc/ssl/certs/doveb0-rmt0-cn1.mail.rmt.example.com.pem
> > ssl_client_ca_dir = /etc/ssl/certs
> > ssl_key = # hidden, use -P to show it
> > submission_max_mail_size =30000k
> > submission_relay_host =10.101.0.7
> > submission_relay_port =587
> > submission_relay_ssl = starttls
> > submission_relay_ssl_verify = no
> > submission_relay_trusted = yes
> > userdb {
> > driver = prefetch
> > }
> > userdb {
> > args = /etc/dovecot/dovecot-ldap-users.conf.ext
> > driver = ldap
> > }
> > userdb {
> > args = /etc/dovecot/dovecot-ldap-groups.conf.ext
> > driver = ldap
> > }
> > protocol lmtp {
> > mail_plugins =" notify mail_log zlib acl sieve acl"
> > }
> > protocol lda {
> > mail_plugins =" notify mail_log zlib acl sieve acl"
> > }
> > protocol imap {
> > mail_max_userip_connections =100
> > mail_plugins =" notify mail_log zlib acl"
> > }
> > protocol sieve {
> > mail_max_userip_connections =40
> > }
> > protocol submission {
> > mail_max_userip_connections =70
> > }
> > local10.101.0.71 {
> > doveadm_allowed_commands > > doveadm_password = # hidden, use -P
to show it
> > }
> > local10.100.1.71 {
> > doveadm_allowed_commands > > doveadm_password = # hidden, use -P
to show it
> > }
> > Thank you,
> > Nikolaos Pyrgiotis