dovecot - Feb 2019 - flags not synced correctly with dovecot sync (dsync)

I'm running dovecot 2.3.4.1 from https://repo.dovecot.org/ on Ubuntu
18.04 on three machines that I'll call server, laptop1 and laptop2.

Both laptop1 and laptop2 run dovecot sync against server to keep local
copies of my imap folders.  Even when I initially had only two machines,
laptop1 and server, I occasionally noticed that flags were lost, usually
custom flags used by Gnus, but I couldn't reliably reproduce the
problem.

Now that I have two laptops syncing against the server, the problem has
gotten worse and I figured out a way to reproduce it:

- on server: create new IMAP folder test, and put two read messages in it
- on laptop1:  doveadm sync -u user -l 10 -m test -f user at server
- on laptop2:  doveadm sync -u user -l 10 -m test -f user at server

At this point, all three machines show the two messages M1 and M2
as being read.

- on laptop1: mark message M1 unread
- on laptop2: mark message M2 unread
- on laptop1:  doveadm sync -u user -l 10 -m test -f user at server
  Both laptop1 and server have M1 unread, M2 read, as expected.
- on laptop2:  doveadm sync -u user -l 10 -m test -f user at server
  Now laptop2 and server have M1 *read*, M2 unread.
- on laptop1:  doveadm sync -u user -l 10 -m test -f user at server
  Now laptop1 and the server have both M1 and M2 *read*.
- on laptop2:  doveadm sync -u user -l 10 -m test -f user at server
  Now laptop2 has both read as well.

The two lines that say "*read*" are wrong in my opinion.  dsync
propagated a read mark to an unread message, even though that message
was marked unread more recently than it was marked read.

I usually use stateful sync, and get many related problems.
I just did a test in which M1 and M2 started out read, and I
started with empty files named dstate.test on laptop1 and laptop2.
Then I did the above procedure, using the command

doveadm sync -u user -l 10 -m test -s "`cat dstate.test`" user at
server > dstate.test

At the end, laptop2 and server had both messages unread (which is good),
but laptop1 had only M1 unread, and repeated runs of the sync command
did not correct this.  So the stateful sync failed to detect a change.

Are these bugs in dovecot?  Is there more information that I can
provide?  The output of doveconf -n on one machine is below, and
the others are almost identical.

Thanks for any help!

Dan

# 2.3.4.1 (3c0b8769e): /etc/dovecot/dovecot.conf
# OS: Linux 4.15.0-45-generic x86_64 Ubuntu 18.04.1 LTS 
# Hostname: laptop2
auth_mechanisms = plain login
listen = 127.0.0.1
mail_index_log2_max_age = 10 days
mail_index_log_rotate_min_age = 1 days
mail_index_log_rotate_min_size = 300 k
mail_location = maildir:~/Maildir
namespace inbox {
  inbox = yes
  location = 
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix = 
}
passdb {
  args = scheme=CRYPT username_format=%u /etc/dovecot/users
  driver = passwd-file
}
protocols = imap
service imap-login {
  inet_listener imap {
    address = *
    port = 143
  }
  inet_listener imaps {
    address = *
    port = 943
    ssl = yes
  }
}
service imap {
  process_limit = 25
}
ssl_cert = </etc/dovecot/private/dovecot.pem
ssl_client_ca_dir = /etc/ssl/certs
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
userdb {
  args = username_format=%u /etc/dovecot/users
  driver = passwd-file
}
protocol lda {
  postmaster_address = [elided]
}
protocol imap {
  mail_max_userip_connections = 20
}

I should add that these problems arise frequently in practice, even if
changes are only being made on a single machine, e.g. laptop1.  Flag
changes on laptop1 propagate to server, but then are reverted on server
when laptop2 syncs with server, and then are reverted on laptop1 when it
syncs again with server.  If I redo the flag changes on laptop1, the
process just repeats and they are removed again.

Suggestions for debugging are very welcome, as currently syncing is
much too unreliable for me to continue using it.

Dan

On Feb 16, 2019, Dan Christensen via dovecot <dovecot at dovecot.org>
wrote:
> I'm running dovecot 2.3.4.1 from https://repo.dovecot.org/ on Ubuntu
> 18.04 on three machines that I'll call server, laptop1 and laptop2.
>
> Both laptop1 and laptop2 run dovecot sync against server to keep local
> copies of my imap folders.  Even when I initially had only two machines,
> laptop1 and server, I occasionally noticed that flags were lost, usually
> custom flags used by Gnus, but I couldn't reliably reproduce the
> problem.
>
> Now that I have two laptops syncing against the server, the problem has
> gotten worse and I figured out a way to reproduce it:
>
> - on server: create new IMAP folder test, and put two read messages in it
> - on laptop1:  doveadm sync -u user -l 10 -m test -f user at server
> - on laptop2:  doveadm sync -u user -l 10 -m test -f user at server
>
> At this point, all three machines show the two messages M1 and M2
> as being read.
>
> - on laptop1: mark message M1 unread
> - on laptop2: mark message M2 unread
> - on laptop1:  doveadm sync -u user -l 10 -m test -f user at server
>   Both laptop1 and server have M1 unread, M2 read, as expected.
> - on laptop2:  doveadm sync -u user -l 10 -m test -f user at server
>   Now laptop2 and server have M1 *read*, M2 unread.
> - on laptop1:  doveadm sync -u user -l 10 -m test -f user at server
>   Now laptop1 and the server have both M1 and M2 *read*.
> - on laptop2:  doveadm sync -u user -l 10 -m test -f user at server
>   Now laptop2 has both read as well.
>
> The two lines that say "*read*" are wrong in my opinion.  dsync
> propagated a read mark to an unread message, even though that message
> was marked unread more recently than it was marked read.
>
> I usually use stateful sync, and get many related problems.
> I just did a test in which M1 and M2 started out read, and I
> started with empty files named dstate.test on laptop1 and laptop2.
> Then I did the above procedure, using the command
>
> doveadm sync -u user -l 10 -m test -s "`cat dstate.test`" user at
server > dstate.test
>
> At the end, laptop2 and server had both messages unread (which is good),
> but laptop1 had only M1 unread, and repeated runs of the sync command
> did not correct this.  So the stateful sync failed to detect a change.
>
> Are these bugs in dovecot?  Is there more information that I can
> provide?  The output of doveconf -n on one machine is below, and
> the others are almost identical.
>
> Thanks for any help!
>
> Dan
>
> # 2.3.4.1 (3c0b8769e): /etc/dovecot/dovecot.conf
> # OS: Linux 4.15.0-45-generic x86_64 Ubuntu 18.04.1 LTS 
> # Hostname: laptop2
> auth_mechanisms = plain login
> listen = 127.0.0.1
> mail_index_log2_max_age = 10 days
> mail_index_log_rotate_min_age = 1 days
> mail_index_log_rotate_min_size = 300 k
> mail_location = maildir:~/Maildir
> namespace inbox {
>   inbox = yes
>   location = 
>   mailbox Drafts {
>     special_use = \Drafts
>   }
>   mailbox Junk {
>     special_use = \Junk
>   }
>   mailbox Sent {
>     special_use = \Sent
>   }
>   mailbox "Sent Messages" {
>     special_use = \Sent
>   }
>   mailbox Trash {
>     special_use = \Trash
>   }
>   prefix = 
> }
> passdb {
>   args = scheme=CRYPT username_format=%u /etc/dovecot/users
>   driver = passwd-file
> }
> protocols = imap
> service imap-login {
>   inet_listener imap {
>     address = *
>     port = 143
>   }
>   inet_listener imaps {
>     address = *
>     port = 943
>     ssl = yes
>   }
> }
> service imap {
>   process_limit = 25
> }
> ssl_cert = </etc/dovecot/private/dovecot.pem
> ssl_client_ca_dir = /etc/ssl/certs
> ssl_dh = # hidden, use -P to show it
> ssl_key = # hidden, use -P to show it
> userdb {
>   args = username_format=%u /etc/dovecot/users
>   driver = passwd-file
> }
> protocol lda {
>   postmaster_address = [elided]
> }
> protocol imap {
>   mail_max_userip_connections = 20
> }

Dan Christensen via dovecot wrote:
> I should add that these problems arise frequently in practice, even if
> changes are only being made on a single machine, e.g. laptop1.  Flag
> changes on laptop1 propagate to server, but then are reverted on server
> when laptop2 syncs with server, and then are reverted on laptop1 when it
> syncs again with server.  If I redo the flag changes on laptop1, the
> process just repeats and they are removed again.
> 
> Suggestions for debugging are very welcome, as currently syncing is
> much too unreliable for me to continue using it.
> 
> Dan
I (and others) have the same issue.

A quote from Stephan Bosch on 20-1-2019:
>> We're working on it. We have reproduced the problem without
difficulty and isolated the commit that caused it.
> 
> Tracking internally as DOP-842.
> 
> Regards,
> 
> Stephan. 

Marc

> 
> On Feb 16, 2019, Dan Christensen via dovecot <dovecot at dovecot.org>
wrote:
> 
>> I'm running dovecot 2.3.4.1 from https://repo.dovecot.org/ on
Ubuntu
>> 18.04 on three machines that I'll call server, laptop1 and laptop2.
>>
>> Both laptop1 and laptop2 run dovecot sync against server to keep local
>> copies of my imap folders.  Even when I initially had only two
machines,
>> laptop1 and server, I occasionally noticed that flags were lost,
usually
>> custom flags used by Gnus, but I couldn't reliably reproduce the
>> problem.
>>
>> Now that I have two laptops syncing against the server, the problem has
>> gotten worse and I figured out a way to reproduce it:
>>
>> - on server: create new IMAP folder test, and put two read messages in
it
>> - on laptop1:  doveadm sync -u user -l 10 -m test -f user at server
>> - on laptop2:  doveadm sync -u user -l 10 -m test -f user at server
>>
>> At this point, all three machines show the two messages M1 and M2
>> as being read.
>>
>> - on laptop1: mark message M1 unread
>> - on laptop2: mark message M2 unread
>> - on laptop1:  doveadm sync -u user -l 10 -m test -f user at server
>>    Both laptop1 and server have M1 unread, M2 read, as expected.
>> - on laptop2:  doveadm sync -u user -l 10 -m test -f user at server
>>    Now laptop2 and server have M1 *read*, M2 unread.
>> - on laptop1:  doveadm sync -u user -l 10 -m test -f user at server
>>    Now laptop1 and the server have both M1 and M2 *read*.
>> - on laptop2:  doveadm sync -u user -l 10 -m test -f user at server
>>    Now laptop2 has both read as well.
>>
>> The two lines that say "*read*" are wrong in my opinion. 
dsync
>> propagated a read mark to an unread message, even though that message
>> was marked unread more recently than it was marked read.
>>
>> I usually use stateful sync, and get many related problems.
>> I just did a test in which M1 and M2 started out read, and I
>> started with empty files named dstate.test on laptop1 and laptop2.
>> Then I did the above procedure, using the command
>>
>> doveadm sync -u user -l 10 -m test -s "`cat dstate.test`"
user at server > dstate.test
>>
>> At the end, laptop2 and server had both messages unread (which is
good),
>> but laptop1 had only M1 unread, and repeated runs of the sync command
>> did not correct this.  So the stateful sync failed to detect a change.
>>
>> Are these bugs in dovecot?  Is there more information that I can
>> provide?  The output of doveconf -n on one machine is below, and
>> the others are almost identical.
>>
>> Thanks for any help!
>>
>> Dan
>>
>> # 2.3.4.1 (3c0b8769e): /etc/dovecot/dovecot.conf
>> # OS: Linux 4.15.0-45-generic x86_64 Ubuntu 18.04.1 LTS
>> # Hostname: laptop2
>> auth_mechanisms = plain login
>> listen = 127.0.0.1
>> mail_index_log2_max_age = 10 days
>> mail_index_log_rotate_min_age = 1 days
>> mail_index_log_rotate_min_size = 300 k
>> mail_location = maildir:~/Maildir
>> namespace inbox {
>>    inbox = yes
>>    location >>    mailbox Drafts {
>>      special_use = \Drafts
>>    }
>>    mailbox Junk {
>>      special_use = \Junk
>>    }
>>    mailbox Sent {
>>      special_use = \Sent
>>    }
>>    mailbox "Sent Messages" {
>>      special_use = \Sent
>>    }
>>    mailbox Trash {
>>      special_use = \Trash
>>    }
>>    prefix >> }
>> passdb {
>>    args = scheme=CRYPT username_format=%u /etc/dovecot/users
>>    driver = passwd-file
>> }
>> protocols = imap
>> service imap-login {
>>    inet_listener imap {
>>      address = *
>>      port = 143
>>    }
>>    inet_listener imaps {
>>      address = *
>>      port = 943
>>      ssl = yes
>>    }
>> }
>> service imap {
>>    process_limit = 25
>> }
>> ssl_cert = </etc/dovecot/private/dovecot.pem
>> ssl_client_ca_dir = /etc/ssl/certs
>> ssl_dh = # hidden, use -P to show it
>> ssl_key = # hidden, use -P to show it
>> userdb {
>>    args = username_format=%u /etc/dovecot/users
>>    driver = passwd-file
>> }
>> protocol lda {
>>    postmaster_address = [elided]
>> }
>> protocol imap {
>>    mail_max_userip_connections = 20
>> }
>

Over many months I've been getting frequent errors in synchronization
using dsync.  They are hard to reproduce, and aren't always of the same
nature, but there's a particular case that I can reproduce 100% of the
time.  It involves read status being synced in the wrong direction
when two machines are both synchronizing with a server.

I'm reposting below a message that gives the method for reproducing the
problem.  I can still reproduce it with Ubuntu's 2:2.3.7.2-1~bionic
packages.

I also have problems with only one machine synchronizing with a server,
but those I can't reproduce on demand.

I'm happy to try patches or test other things.

I'm using Maildir storage.

Thanks for any suggestions!

Dan

On Feb 16, 2019, Dan Christensen via dovecot <dovecot at dovecot.org>
wrote:
> I'm running dovecot 2.3.4.1 from https://repo.dovecot.org/ on Ubuntu
> 18.04 on three machines that I'll call server, laptop1 and laptop2.
>
> Both laptop1 and laptop2 run dovecot sync against server to keep local
> copies of my imap folders.  Even when I initially had only two machines,
> laptop1 and server, I occasionally noticed that flags were lost, usually
> custom flags used by Gnus, but I couldn't reliably reproduce the
> problem.
>
> Now that I have two laptops syncing against the server, the problem has
> gotten worse and I figured out a way to reproduce it:
>
> - on server: create new IMAP folder test, and put two read messages in it
> - on laptop1:  doveadm sync -u user -l 10 -m test -f user at server
> - on laptop2:  doveadm sync -u user -l 10 -m test -f user at server
>
> At this point, all three machines show the two messages M1 and M2
> as being read.
>
> - on laptop1: mark message M1 unread
> - on laptop2: mark message M2 unread
> - on laptop1:  doveadm sync -u user -l 10 -m test -f user at server
>   Both laptop1 and server have M1 unread, M2 read, as expected.
> - on laptop2:  doveadm sync -u user -l 10 -m test -f user at server
>   Now laptop2 and server have M1 *read*, M2 unread.
> - on laptop1:  doveadm sync -u user -l 10 -m test -f user at server
>   Now laptop1 and the server have both M1 and M2 *read*.
> - on laptop2:  doveadm sync -u user -l 10 -m test -f user at server
>   Now laptop2 has both read as well.
>
> The two lines that say "*read*" are wrong in my opinion.  dsync
> propagated a read mark to an unread message, even though that message
> was marked unread more recently than it was marked read.
>
> I usually use stateful sync, and get many related problems.
> I just did a test in which M1 and M2 started out read, and I
> started with empty files named dstate.test on laptop1 and laptop2.
> Then I did the above procedure, using the command
>
> doveadm sync -u user -l 10 -m test -s "`cat dstate.test`" user at
server > dstate.test
>
> At the end, laptop2 and server had both messages unread (which is good),
> but laptop1 had only M1 unread, and repeated runs of the sync command
> did not correct this.  So the stateful sync failed to detect a change.
>
> Are these bugs in dovecot?  Is there more information that I can
> provide?  The output of doveconf -n on one machine is below, and
> the others are almost identical.
>
> Thanks for any help!
>
> Dan
>
> # 2.3.4.1 (3c0b8769e): /etc/dovecot/dovecot.conf
> # OS: Linux 4.15.0-45-generic x86_64 Ubuntu 18.04.1 LTS 
> # Hostname: laptop2
> auth_mechanisms = plain login
> listen = 127.0.0.1
> mail_index_log2_max_age = 10 days
> mail_index_log_rotate_min_age = 1 days
> mail_index_log_rotate_min_size = 300 k
> mail_location = maildir:~/Maildir
> namespace inbox {
>   inbox = yes
>   location = 
>   mailbox Drafts {
>     special_use = \Drafts
>   }
>   mailbox Junk {
>     special_use = \Junk
>   }
>   mailbox Sent {
>     special_use = \Sent
>   }
>   mailbox "Sent Messages" {
>     special_use = \Sent
>   }
>   mailbox Trash {
>     special_use = \Trash
>   }
>   prefix = 
> }
> passdb {
>   args = scheme=CRYPT username_format=%u /etc/dovecot/users
>   driver = passwd-file
> }
> protocols = imap
> service imap-login {
>   inet_listener imap {
>     address = *
>     port = 143
>   }
>   inet_listener imaps {
>     address = *
>     port = 943
>     ssl = yes
>   }
> }
> service imap {
>   process_limit = 25
> }
> ssl_cert = </etc/dovecot/private/dovecot.pem
> ssl_client_ca_dir = /etc/ssl/certs
> ssl_dh = # hidden, use -P to show it
> ssl_key = # hidden, use -P to show it
> userdb {
>   args = username_format=%u /etc/dovecot/users
>   driver = passwd-file
> }
> protocol lda {
>   postmaster_address = [elided]
> }
> protocol imap {
>   mail_max_userip_connections = 20
> }