On Sat, Oct 17, 2020 at 11:17:34PM +0200, John Fawcett wrote:> > I have observed my mail storage directories on both instances and have > > noticed that they diverge a little over time - the instance B not having > > all the updates. However the moment I connect another client to instance > > B and log in as a particular user, the state on instance B springs to the > > correct state and I see the same content on both clients, and the storage > > directories become identical as well. > > Are there any replication errors logged? Does doveadm replicator status or > doveadm replicator status <mailbox> give any sign of errors? > > Can you give some example or evidence of the disalignment that is happening?Thank you for response. I was asking if such difference in directories is normal or not and do I have the approach right or is there some obvious error in my configuration. I still have an example that prompted me to write this email. Instance B had three mails missing in comparison to instance A, when I listed the storage directories from the command line. They got replicated fine just after I connected a client to instance B. No difference since then, but I need to observe the servers for some more time, it seems like it happens over a few weeks or so. No errors in dovecot logs, doveadm replicator status output does not show any errors as well, but instances have been restarted, so it is not a goof indication. I am going to enable debug to see what's going on. Maybe it was some transient situation or human error on my part during setting it all up or tweaking the configuration. I will come back if I observe such thing again. Here are the listings. Hostnames etc. replaced for privacy. --- Instance A root at serverA:/var/vmail/user at example.net/mail/INBOX/.MailDir/cur# ll total 14684 drwx------ 2 vmail vmail 4096 Oct 16 13:15 . drwx------ 5 vmail vmail 4096 Oct 16 13:15 .. -rw------- 1 vmail vmail 303371 Oct 30 2019 '1572438988.M167729P10012.hostname.mydomain.net,S=303371,W=307546:2,S' -rw------- 1 vmail vmail 1789790 Oct 30 2019 '1572438998.M633142P10012.hostname.mydomain.net,S=1789790,W=1814325:2,S' -rw------- 1 vmail vmail 171191 Nov 17 2019 '1574012316.M188507P14607.hostname.mydomain.net,S=171191,W=173557:2,S' -rw------- 1 vmail vmail 3430888 Nov 29 2019 '1574956300.M308957P25339.hostname.mydomain.net,S=3430888,W=3475534:2,S' -rw------- 1 vmail vmail 7185 Mar 14 2020 '1584150674.M740366P21925.hostname.mydomain.net,S=7185,W=7294:2,S' -rw------- 1 vmail vmail 5543 Mar 25 2020 '1585148832.M609262P877.hostname.mydomain.net,S=5543,W=5674:2,S' -rw------- 1 vmail vmail 3052 Apr 19 22:23 '1587327803.M423484P19471.hostname.mydomain.net,S=3052,W=3122:2,S' -rw------- 1 vmail vmail 13690 Apr 23 14:47 '1587646046.M939301P22621.hostname.mydomain.net,S=13690,W=13886:2,S' -rw------- 1 vmail vmail 8917623 May 25 19:06 '1590426395.M585516P47979.hostname.mydomain.net,S=8917623,W=9033467:2,S' -rw------- 1 vmail vmail 5473 Jun 20 20:14 '1592676840.M810974P68188.hostname.mydomain.net,S=5473,W=5576:2,S' -rw------- 1 vmail vmail 8776 Jun 21 19:25 '1592760350.M765525P68795.hostname.mydomain.net,S=8776,W=8936:2,S' -rw------- 1 vmail vmail 4624 Jul 23 17:25 '1595517903.M310464P9504.hostname.mydomain.net,S=4624,W=4715:2,S' -rw------- 1 vmail vmail 4378 Aug 18 00:16 '1597702561.M109036P34930.hostname.mydomain.net,S=4378,W=4453:2,S' -rw------- 1 vmail vmail 14522 Sep 17 11:52 '1600336372.M982371P3862.hostname.mydomain.net,S=14522,W=14813:2,S' -rw------- 1 vmail vmail 30411 Sep 23 14:43 '1600865025.M865916P104.hostname.mydomain.net,S=30411,W=30850:2,S' -rw------- 1 vmail vmail 39928 Sep 30 08:27 '1601447231.M21429P464.hostname.mydomain.net,S=39928,W=40487:2,S' -rw------- 1 vmail vmail 8529 Oct 9 15:32 '1602250379.M823144P212.hostname.mydomain.net,S=8529,W=8701:2,S' -rw------- 1 vmail vmail 17972 Oct 15 00:59 '1602716371.M291521P5700.hostname.mydomain.net,S=17972,W=18407:2,S' -rw------- 1 vmail vmail 38528 Oct 15 08:24 '1602743049.M23930P5928.hostname.mydomain.net,S=38528,W=39224:2,S' -rw------- 1 vmail vmail 52350 Oct 15 12:05 '1602756328.M324186P6101.hostname.mydomain.net,S=52350,W=53436:2,S' -rw------- 1 vmail vmail 112458 Oct 16 13:05 '1602846323.M415371P7147.hostname.mydomain.net,S=112458,W=114768:2,S' --- Instance B root at serverB:/var/vmail/user at example.net/mail/INBOX/.MailDir/cur# ll total 14608 drwx------ 2 vmail vmail 12288 Oct 16 13:15 . drwx------ 5 vmail vmail 4096 Oct 16 13:15 .. -rw------- 1 vmail vmail 303371 Oct 30 2019 '1572438988.M167729P10012.hostname.mydomain.net,S=303371,W=307546:2,S' -rw------- 1 vmail vmail 1789790 Oct 30 2019 '1572438998.M633142P10012.hostname.mydomain.net,S=1789790,W=1814325:2,S' -rw------- 1 vmail vmail 171191 Nov 17 2019 '1574012316.M188507P14607.hostname.mydomain.net,S=171191,W=173557:2,S' -rw------- 1 vmail vmail 3430888 Nov 29 2019 '1574956300.M308957P25339.hostname.mydomain.net,S=3430888,W=3475534:2,S' -rw------- 1 vmail vmail 7185 Mar 14 2020 '1584150674.M740366P21925.hostname.mydomain.net,S=7185,W=7294:2,S' -rw------- 1 vmail vmail 5543 Mar 25 2020 '1585148832.M609262P877.hostname.mydomain.net,S=5543,W=5674:2,S' -rw------- 1 vmail vmail 3052 Apr 19 22:23 '1587327803.M423484P19471.hostname.mydomain.net,S=3052,W=3122:2,S' -rw------- 1 vmail vmail 13690 Apr 23 14:47 '1587646046.M939301P22621.hostname.mydomain.net,S=13690,W=13886:2,S' -rw------- 1 vmail vmail 8917623 May 25 19:06 '1590426395.M585516P47979.hostname.mydomain.net,S=8917623,W=9033467:2,S' -rw------- 1 vmail vmail 5473 Jun 20 20:14 '1592676840.M810974P68188.hostname.mydomain.net,S=5473,W=5576:2,S' -rw------- 1 vmail vmail 8776 Jun 21 19:25 '1592760350.M765525P68795.hostname.mydomain.net,S=8776,W=8936:2,S' -rw------- 1 vmail vmail 4624 Jul 23 17:25 '1595517903.M310464P9504.hostname.mydomain.net,S=4624,W=4715:2,S' -rw------- 1 vmail vmail 4378 Aug 18 00:16 '1597702561.M109036P34930.hostname.mydomain.net,S=4378,W=4453:2,S' -rw------- 1 vmail vmail 14522 Sep 17 11:52 '1600336372.M982371P3862.hostname.mydomain.net,S=14522,W=14813:2,S' -rw------- 1 vmail vmail 30411 Sep 23 14:43 '1600865025.M865916P104.hostname.mydomain.net,S=30411,W=30850:2,S' -rw------- 1 vmail vmail 39928 Sep 30 08:27 '1601447231.M21429P464.hostname.mydomain.net,S=39928,W=40487:2,S' -rw------- 1 vmail vmail 38528 Oct 15 08:24 '1602743049.M23930P5928.hostname.mydomain.net,S=38528,W=39224:2,S' -rw------- 1 vmail vmail 112458 Oct 16 13:05 '1602846323.M415371P7147.hostname.mydomain.net,S=112458,W=114768:2,S' -- Piotr "Malgond" Auksztulewicz firstname at lastname.net
On 18/10/2020 22:53, Piotr Auksztulewicz wrote:> On Sat, Oct 17, 2020 at 11:17:34PM +0200, John Fawcett wrote: >>> I have observed my mail storage directories on both instances and have >>> noticed that they diverge a little over time - the instance B not having >>> all the updates. However the moment I connect another client to instance >>> B and log in as a particular user, the state on instance B springs to the >>> correct state and I see the same content on both clients, and the storage >>> directories become identical as well. >> Are there any replication errors logged? Does doveadm replicator status or >> doveadm replicator status <mailbox> give any sign of errors? >> >> Can you give some example or evidence of the disalignment that is happening? > Thank you for response. I was asking if such difference in directories is > normal or not and do I have the approach right or is there some obvious error > in my configuration. > > I still have an example that prompted me to write this email. Instance B had > three mails missing in comparison to instance A, when I listed the storage > directories from the command line. They got replicated fine just after I > connected a client to instance B. No difference since then, but I need to > observe the servers for some more time, it seems like it happens over a few > weeks or so. No errors in dovecot logs, doveadm replicator status output does > not show any errors as well, but instances have been restarted, so it is not > a goof indication. I am going to enable debug to see what's going on. Maybe it > was some transient situation or human error on my part during setting it all up > or tweaking the configuration. I will come back if I observe such thing again. > > Here are the listings. Hostnames etc. replaced for privacy. > > --- Instance A > root at serverA:/var/vmail/user at example.net/mail/INBOX/.MailDir/cur# ll > total 14684 > drwx------ 2 vmail vmail 4096 Oct 16 13:15 . > drwx------ 5 vmail vmail 4096 Oct 16 13:15 .. > -rw------- 1 vmail vmail 303371 Oct 30 2019 '1572438988.M167729P10012.hostname.mydomain.net,S=303371,W=307546:2,S' > -rw------- 1 vmail vmail 1789790 Oct 30 2019 '1572438998.M633142P10012.hostname.mydomain.net,S=1789790,W=1814325:2,S' > -rw------- 1 vmail vmail 171191 Nov 17 2019 '1574012316.M188507P14607.hostname.mydomain.net,S=171191,W=173557:2,S' > -rw------- 1 vmail vmail 3430888 Nov 29 2019 '1574956300.M308957P25339.hostname.mydomain.net,S=3430888,W=3475534:2,S' > -rw------- 1 vmail vmail 7185 Mar 14 2020 '1584150674.M740366P21925.hostname.mydomain.net,S=7185,W=7294:2,S' > -rw------- 1 vmail vmail 5543 Mar 25 2020 '1585148832.M609262P877.hostname.mydomain.net,S=5543,W=5674:2,S' > -rw------- 1 vmail vmail 3052 Apr 19 22:23 '1587327803.M423484P19471.hostname.mydomain.net,S=3052,W=3122:2,S' > -rw------- 1 vmail vmail 13690 Apr 23 14:47 '1587646046.M939301P22621.hostname.mydomain.net,S=13690,W=13886:2,S' > -rw------- 1 vmail vmail 8917623 May 25 19:06 '1590426395.M585516P47979.hostname.mydomain.net,S=8917623,W=9033467:2,S' > -rw------- 1 vmail vmail 5473 Jun 20 20:14 '1592676840.M810974P68188.hostname.mydomain.net,S=5473,W=5576:2,S' > -rw------- 1 vmail vmail 8776 Jun 21 19:25 '1592760350.M765525P68795.hostname.mydomain.net,S=8776,W=8936:2,S' > -rw------- 1 vmail vmail 4624 Jul 23 17:25 '1595517903.M310464P9504.hostname.mydomain.net,S=4624,W=4715:2,S' > -rw------- 1 vmail vmail 4378 Aug 18 00:16 '1597702561.M109036P34930.hostname.mydomain.net,S=4378,W=4453:2,S' > -rw------- 1 vmail vmail 14522 Sep 17 11:52 '1600336372.M982371P3862.hostname.mydomain.net,S=14522,W=14813:2,S' > -rw------- 1 vmail vmail 30411 Sep 23 14:43 '1600865025.M865916P104.hostname.mydomain.net,S=30411,W=30850:2,S' > -rw------- 1 vmail vmail 39928 Sep 30 08:27 '1601447231.M21429P464.hostname.mydomain.net,S=39928,W=40487:2,S' > -rw------- 1 vmail vmail 8529 Oct 9 15:32 '1602250379.M823144P212.hostname.mydomain.net,S=8529,W=8701:2,S' > -rw------- 1 vmail vmail 17972 Oct 15 00:59 '1602716371.M291521P5700.hostname.mydomain.net,S=17972,W=18407:2,S' > -rw------- 1 vmail vmail 38528 Oct 15 08:24 '1602743049.M23930P5928.hostname.mydomain.net,S=38528,W=39224:2,S' > -rw------- 1 vmail vmail 52350 Oct 15 12:05 '1602756328.M324186P6101.hostname.mydomain.net,S=52350,W=53436:2,S' > -rw------- 1 vmail vmail 112458 Oct 16 13:05 '1602846323.M415371P7147.hostname.mydomain.net,S=112458,W=114768:2,S' > > > --- Instance B > root at serverB:/var/vmail/user at example.net/mail/INBOX/.MailDir/cur# ll > total 14608 > drwx------ 2 vmail vmail 12288 Oct 16 13:15 . > drwx------ 5 vmail vmail 4096 Oct 16 13:15 .. > -rw------- 1 vmail vmail 303371 Oct 30 2019 '1572438988.M167729P10012.hostname.mydomain.net,S=303371,W=307546:2,S' > -rw------- 1 vmail vmail 1789790 Oct 30 2019 '1572438998.M633142P10012.hostname.mydomain.net,S=1789790,W=1814325:2,S' > -rw------- 1 vmail vmail 171191 Nov 17 2019 '1574012316.M188507P14607.hostname.mydomain.net,S=171191,W=173557:2,S' > -rw------- 1 vmail vmail 3430888 Nov 29 2019 '1574956300.M308957P25339.hostname.mydomain.net,S=3430888,W=3475534:2,S' > -rw------- 1 vmail vmail 7185 Mar 14 2020 '1584150674.M740366P21925.hostname.mydomain.net,S=7185,W=7294:2,S' > -rw------- 1 vmail vmail 5543 Mar 25 2020 '1585148832.M609262P877.hostname.mydomain.net,S=5543,W=5674:2,S' > -rw------- 1 vmail vmail 3052 Apr 19 22:23 '1587327803.M423484P19471.hostname.mydomain.net,S=3052,W=3122:2,S' > -rw------- 1 vmail vmail 13690 Apr 23 14:47 '1587646046.M939301P22621.hostname.mydomain.net,S=13690,W=13886:2,S' > -rw------- 1 vmail vmail 8917623 May 25 19:06 '1590426395.M585516P47979.hostname.mydomain.net,S=8917623,W=9033467:2,S' > -rw------- 1 vmail vmail 5473 Jun 20 20:14 '1592676840.M810974P68188.hostname.mydomain.net,S=5473,W=5576:2,S' > -rw------- 1 vmail vmail 8776 Jun 21 19:25 '1592760350.M765525P68795.hostname.mydomain.net,S=8776,W=8936:2,S' > -rw------- 1 vmail vmail 4624 Jul 23 17:25 '1595517903.M310464P9504.hostname.mydomain.net,S=4624,W=4715:2,S' > -rw------- 1 vmail vmail 4378 Aug 18 00:16 '1597702561.M109036P34930.hostname.mydomain.net,S=4378,W=4453:2,S' > -rw------- 1 vmail vmail 14522 Sep 17 11:52 '1600336372.M982371P3862.hostname.mydomain.net,S=14522,W=14813:2,S' > -rw------- 1 vmail vmail 30411 Sep 23 14:43 '1600865025.M865916P104.hostname.mydomain.net,S=30411,W=30850:2,S' > -rw------- 1 vmail vmail 39928 Sep 30 08:27 '1601447231.M21429P464.hostname.mydomain.net,S=39928,W=40487:2,S' > -rw------- 1 vmail vmail 38528 Oct 15 08:24 '1602743049.M23930P5928.hostname.mydomain.net,S=38528,W=39224:2,S' > -rw------- 1 vmail vmail 112458 Oct 16 13:05 '1602846323.M415371P7147.hostname.mydomain.net,S=112458,W=114768:2,S' >Hi Piotr can you check if the missing emails are in the new directory on host B? My guess is that the missing emails appeared on host A in new and were replicated to host B in new. Then the imap client connected to host A moved them from new to cur on host A and that such movement is not subject to replica. So long as the emails have been replicated to either new or cur on host B, then it should not be an issue about losing emails if host A goes down, since connecting an imap client to host B will move the emails to cur (that is no replication involved). John
On Sun, Oct 18, 2020 at 11:14:26PM +0200, John Fawcett wrote:> can you check if the missing emails are in the new directory on host B?I cannot now. I did not think about that possibility when listing the contents of directories and have not checked the new directory.> My guess is that the missing emails appeared on host A in new and were > replicated to host B in new. Then the imap client connected to host A > moved them from new to cur on host A and that such movement is not > subject to replica. So long as the emails have been replicated to either > new or cur on host B, then it should not be an issue about losing emails > if host A goes down, since connecting an imap client to host B will move > the emails to cur (that is no replication involved).Now that I had connected a client to host B once, they must have got moved to cur. The next time I see this, I will check new directory as well. Maybe the whole issue is just misunderstanding. -- Piotr "Malgond" Auksztulewicz firstname at lastname.net