dovecot - Feb 2019 - flags not synced correctly with dovecot sync (dsync)

Dan Christensen via dovecot wrote:
> I should add that these problems arise frequently in practice, even if
> changes are only being made on a single machine, e.g. laptop1.  Flag
> changes on laptop1 propagate to server, but then are reverted on server
> when laptop2 syncs with server, and then are reverted on laptop1 when it
> syncs again with server.  If I redo the flag changes on laptop1, the
> process just repeats and they are removed again.
> 
> Suggestions for debugging are very welcome, as currently syncing is
> much too unreliable for me to continue using it.
> 
> Dan
I (and others) have the same issue.

A quote from Stephan Bosch on 20-1-2019:
>> We're working on it. We have reproduced the problem without
difficulty and isolated the commit that caused it.
> 
> Tracking internally as DOP-842.
> 
> Regards,
> 
> Stephan. 

Marc

> 
> On Feb 16, 2019, Dan Christensen via dovecot <dovecot at dovecot.org>
wrote:
> 
>> I'm running dovecot 2.3.4.1 from https://repo.dovecot.org/ on
Ubuntu
>> 18.04 on three machines that I'll call server, laptop1 and laptop2.
>>
>> Both laptop1 and laptop2 run dovecot sync against server to keep local
>> copies of my imap folders.  Even when I initially had only two
machines,
>> laptop1 and server, I occasionally noticed that flags were lost,
usually
>> custom flags used by Gnus, but I couldn't reliably reproduce the
>> problem.
>>
>> Now that I have two laptops syncing against the server, the problem has
>> gotten worse and I figured out a way to reproduce it:
>>
>> - on server: create new IMAP folder test, and put two read messages in
it
>> - on laptop1:  doveadm sync -u user -l 10 -m test -f user at server
>> - on laptop2:  doveadm sync -u user -l 10 -m test -f user at server
>>
>> At this point, all three machines show the two messages M1 and M2
>> as being read.
>>
>> - on laptop1: mark message M1 unread
>> - on laptop2: mark message M2 unread
>> - on laptop1:  doveadm sync -u user -l 10 -m test -f user at server
>>    Both laptop1 and server have M1 unread, M2 read, as expected.
>> - on laptop2:  doveadm sync -u user -l 10 -m test -f user at server
>>    Now laptop2 and server have M1 *read*, M2 unread.
>> - on laptop1:  doveadm sync -u user -l 10 -m test -f user at server
>>    Now laptop1 and the server have both M1 and M2 *read*.
>> - on laptop2:  doveadm sync -u user -l 10 -m test -f user at server
>>    Now laptop2 has both read as well.
>>
>> The two lines that say "*read*" are wrong in my opinion. 
dsync
>> propagated a read mark to an unread message, even though that message
>> was marked unread more recently than it was marked read.
>>
>> I usually use stateful sync, and get many related problems.
>> I just did a test in which M1 and M2 started out read, and I
>> started with empty files named dstate.test on laptop1 and laptop2.
>> Then I did the above procedure, using the command
>>
>> doveadm sync -u user -l 10 -m test -s "`cat dstate.test`"
user at server > dstate.test
>>
>> At the end, laptop2 and server had both messages unread (which is
good),
>> but laptop1 had only M1 unread, and repeated runs of the sync command
>> did not correct this.  So the stateful sync failed to detect a change.
>>
>> Are these bugs in dovecot?  Is there more information that I can
>> provide?  The output of doveconf -n on one machine is below, and
>> the others are almost identical.
>>
>> Thanks for any help!
>>
>> Dan
>>
>> # 2.3.4.1 (3c0b8769e): /etc/dovecot/dovecot.conf
>> # OS: Linux 4.15.0-45-generic x86_64 Ubuntu 18.04.1 LTS
>> # Hostname: laptop2
>> auth_mechanisms = plain login
>> listen = 127.0.0.1
>> mail_index_log2_max_age = 10 days
>> mail_index_log_rotate_min_age = 1 days
>> mail_index_log_rotate_min_size = 300 k
>> mail_location = maildir:~/Maildir
>> namespace inbox {
>>    inbox = yes
>>    location >>    mailbox Drafts {
>>      special_use = \Drafts
>>    }
>>    mailbox Junk {
>>      special_use = \Junk
>>    }
>>    mailbox Sent {
>>      special_use = \Sent
>>    }
>>    mailbox "Sent Messages" {
>>      special_use = \Sent
>>    }
>>    mailbox Trash {
>>      special_use = \Trash
>>    }
>>    prefix >> }
>> passdb {
>>    args = scheme=CRYPT username_format=%u /etc/dovecot/users
>>    driver = passwd-file
>> }
>> protocols = imap
>> service imap-login {
>>    inet_listener imap {
>>      address = *
>>      port = 143
>>    }
>>    inet_listener imaps {
>>      address = *
>>      port = 943
>>      ssl = yes
>>    }
>> }
>> service imap {
>>    process_limit = 25
>> }
>> ssl_cert = </etc/dovecot/private/dovecot.pem
>> ssl_client_ca_dir = /etc/ssl/certs
>> ssl_dh = # hidden, use -P to show it
>> ssl_key = # hidden, use -P to show it
>> userdb {
>>    args = username_format=%u /etc/dovecot/users
>>    driver = passwd-file
>> }
>> protocol lda {
>>    postmaster_address = [elided]
>> }
>> protocol imap {
>>    mail_max_userip_connections = 20
>> }
>

On Feb 19, 2019, Marc Weustink via dovecot <dovecot at dovecot.org> wrote:
> I (and others) have the same issue.
Thanks, but the issue described in that thread is not the same issue.
I don't believe IMAP MOVE is used in the test case I outlined.  And
I can also reproduce it with 1:2.2.33.2-1ubuntu4.2, whereas the issue
in that thread was not present in 1:2.2.36-1.

Does anyone else have input on this problem?  I used offlineimap for
years without losing flags, but I was hoping to use dovecot sync in
the future, since it is a lot faster.

Several times per day I also have the problem that sync stops working
and I have to remove the state file for new messages to be noticed.

I'm happy to help debug these issues.

Thanks,

Dan

In another thread, Timo wrote:

On Mar 12, 2019, Timo Sirainen via dovecot <dovecot at dovecot.org> wrote:
> That bug is fixed with attached patch.
Thanks!  I'm attaching the patch here, so it is in this thread as well.

A couple of questions before I test this:

- Do I need the patch on the remote end of the sync, or the local end,
  or both?

- Does it make sense to try it based on the master-2.3 branch from git?
  Following https://wiki.dovecot.org/CompilingSource  ?

I'll report back once I've tested it.

Dan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2656.patch
Type: text/x-diff
Size: 1225 bytes
Desc: not available
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20190312/e3ea33fb/attachment.bin>