Hi, I recently performed a mail server migration for a client under fairly serious time constraints (ahem, yes one of those jobs). I would normally use imapsync to get all the mail copied to the new server, but under the circumstances, I had to write a script to rsync each of the mailboxes over ssh, from one server to another. This all worked fine, except for the fact that if a client was using a POP email client, when they connected to the new server, they re-downloaded all their mail, creating duplicates of everything in their Outlook client. Of course they weren't happy about this. IMAP email clients were OK. No duplicates, and everything was fine. So while I was migrating the mail, I did try for a while to understand the format of the UID files, but failed to do so in the available time, so the client just had to deal with duplicate emails. But now the smoke has cleared, I'd like to understand the problem a little better, and I was hoping someone on this forum could explain it to me, and the changes I'd need to make to the files so that the POP client DIDN'T download the duplicate emails. Here's what I understand. The list of dovecot files in an account looks like this: -rw-r----- 1 admin mail 0 Jul 11 2018 dovecot-acl-list -rw-r----- 1 admin mail 14K Sep 20 08:36 dovecot.index -rw-r----- 1 admin mail 216K Sep 20 17:24 dovecot.index.cache -rw-r----- 1 admin mail 29K Sep 20 17:24 dovecot.index.log -rw-r----- 1 admin mail 388 Dec 17 2017 dovecot.list.index -rw-r----- 1 admin mail 1.6K Jul 11 2018 dovecot.list.index.log -rw-r----- 1 admin mail 24 Jul 11 2018 dovecot.mailbox.log -rw-r----- 1 admin mail 53 Sep 12 09:42 dovecot-quota -rw-r----- 1 admin mail 52K Sep 20 17:24 dovecot-uidlist -rw-r----- 1 admin mail 8 Jul 11 2018 dovecot-uidvalidity -rw-r----- 1 admin mail 0 Jul 4 2016 dovecot-uidvalidity.5779bbeb It seems that the dovecot-uidlist is an index of all the mail files in the account. As far as I can gather, this should be the same on both servers after I've rsynced them over ssh: The files and the list of them in dovecot-uidlist should correspond. So what is it that the POP client looks at and decides to re-download all the emails? And how could I have stopped it from doing that, and get it to realize that the list of emails it holds internally is the same as the list on the server? What would I need to change in the dovecot files? Thanks for any insight into the process. As a secondary question -- and perhaps I should put this in a separate message -- I did notice that "doveadm sync" would apparently have helped me with this, but I wasn't able to get that to work either. I believe it was something to do with the fact that all the mailboxes were under the same linux user account on the target server, and I couldn't figure out all the paths and permissions in time. Would 'doveadm sync' have fixed all the UID and duplicate POP email issues? P.
On Sep 20, 2019, at 10:38 PM, Plutocrat <plutocrat at gmail.com> wrote:> I recently performed a mail server migration for a client under fairly serious time constraints (ahem, yes one of those jobs). I would normally use imapsync to get all the mail copied to the new server, but under the circumstances, I had to write a script to rsync each of the mailboxes over ssh, from one server to another. > > This all worked fine, except for the fact that if a client was using a POP email client, when they connected to the new server, they re-downloaded all their mail, creating duplicates of everything in their Outlook client. Of course they weren't happy about this. IMAP email clients were OK. No duplicates, and everything was fine.Dealing with weird POP3 things h is why I disabled it on my server more than a decade ago; users have to beg for POP access and promise they will ONLY use it to get their mail into gmail.> So while I was migrating the mail, I did try for a while to understand the format of the UID files, but failed to do so in the available time, so the client just had to deal with duplicate emails. But now the smoke has cleared, I'd like to understand the problem a little better, and I was hoping someone on this forum could explain it to me, and the changes I'd need to make to the files so that the POP client DIDN'T download the duplicate emails.Did you check <https://wiki.dovecot.org/Migration>? It has a lot of info on this. Seems dsync would hav been the best way to do this? <https://wiki.dovecot.org/Migration/Dsync>> As a secondary question -- and perhaps I should put this in a separate message -- I did notice that "doveadm sync" would apparently have helped me with this, but I wasn't able to get that to work either. I believe it was something to do with the fact that all the mailboxes were under the same linux user account on the target server, and I couldn't figure out all the paths and permissions in time. Would 'doveadm sync' have fixed all the UID and duplicate POP email issues?Don?t know, but dsync says it does this: "The pop3-migration plugin is used to preserve POP3 UIDLs. When dsync is handling IMAP INBOX and requests a POP3 UIDL, the plugin connects to the POP3 server and figures out which IMAP messages match which POP3 messages and then returns the appropriate POP3 UIDL.? Trouble is, if you are migrating POP and the server is not up, I am not sure what you can do with dsync? -- @mdhughes: One of the few regrets I have about lawn-less apartments: Shallow graves are so much harder to come by.
On 22/09/2019 4:08 AM, @lbutlr via dovecot wrote:>> So while I was migrating the mail, I did try for a while to understand the format of the UID files, but failed to do so in the available time, so the client just had to deal with duplicate emails. But now the smoke has cleared, I'd like to understand the problem a little better, and I was hoping someone on this forum could explain it to me, and the changes I'd need to make to the files so that the POP client DIDN'T download the duplicate emails. > > Did you check <https://wiki.dovecot.org/Migration>? It has a lot of info on this.I did see that page, but was unable to figure out which of the three UID related files I needed to keep, and which to edit in order to 'trick' the email client into not re-downloading the mails. The migration was from dovecot 2.x to 2.x, so I don't think there was any translation issue between servers. The POP clients were Outlook (my old nemesis). The only information I could find is that the dovecot UIDL format is pop3_uidl_format = %v.%u But that didn't really take me anywhere. So its all a bit of a mystery. I was hoping there would be a UID guru on this list who could walk me through it, or point me to some reference.> Don?t know, but dsync says it does this: > "The pop3-migration plugin is used to preserve POP3 UIDLs. When dsync is handling IMAP INBOX and requests a POP3 UIDL, the plugin connects to the POP3 server and figures out which IMAP messages match which POP3 messages and then returns the appropriate POP3 UIDL.? > Trouble is, if you are migrating POP and the server is not up, I am not sure what you can do with dsync?I did try to use dsync (aka doveadm sync) for a few hours, but couldn't get a test account to migrate across. It seems like this would have been the solution but I was unable to confirm. At times it seemed to login and complete without error, but I could never find where it had put the mail! Other times it refused to login or gave permission errors. I looked around for real life examples, but wasn't able to find any. Had I got it to work, I could maybe have figured out the UID problem. Maybe that will be a project for me in the future, when I have a spare few hours! Maybe if I gave a few details of the directory structure that would help. I was migrating from a Cpanel installation to a standalone mail server. In Cpanel, there was the admin account login, and under that ~/mail/domain1.com/mailbox1 ~/mail/domain1.com/mailbox2 ~/mail/domain2.com/mailbox1 ~/mail/domain2.com/mailbox2 On the target server the same structure existed. One login, admin, and under that account ~/mail/domain1.com/mailbox1 ~/mail/domain1.com/mailbox2 ~/mail/domain2.com/mailbox1 ~/mail/domain2.com/mailbox2 All files and directories were chown admin:mail The command I used was doveadm sync -u test at domain.com ssh -i id_rsa -o "StrictHostKeyChecking=no" admin at 100.110.120.130 I tried a few options to tell it where to put the mail, but dovecot on the target server didn't seem to know where each account was located. P.