Am 04.09.2018 um 17:18 schrieb Sami Ketola:> > >> On 4 Sep 2018, at 18.00, Robert Schetterer <rs at sys4.de> wrote: >> >> Am 04.09.2018 um 16:52 schrieb Sami Ketola: >>> >>> >>>> On 4 Sep 2018, at 17.47, Robert Schetterer <rs at sys4.de> wrote: >>>> >>>> Am 04.09.2018 um 09:41 schrieb Sami Ketola: >>>>> imapsync always loses data >>>> >>>> never saw this, be carefull by anounce such myths >>> >>> >>> It is a fact. Imapsync works over IMAP only and IMAP protocol does not even support transferring all data. At most at least UID numbering will be lost and end users need to invalidate their local caches. >> >> but using "looses data" might others think it also may fail with imap, >> so be more detailed next time > > > UID number is also data that is saved on the IMAP backend. If that is lost then it's "lost data". > > Sami >Sorry i migrated terrabytes of mail with imapsync and never had a problem, it works as designed, also with maildir rsync did a good job, what never worked as it should was dsync ,cause of bugs ,that may changed now so this is my answer to topic "Best way to move mail from one server to another" after all there is no "best way", there are a few ways choose the one which fits best to your needs goals and starting points ( kind of servers etc ) Best Regards MfG Robert Schetterer -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Schlei?heimer Stra?e 26/MG, 80333 M?nchen Sitz der Gesellschaft: M?nchen, Amtsgericht M?nchen: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
Quoting Robert Schetterer <rs at sys4.de>:> Am 04.09.2018 um 17:18 schrieb Sami Ketola: >>> On 4 Sep 2018, at 18.00, Robert Schetterer <rs at sys4.de> wrote: >>> >>> Am 04.09.2018 um 16:52 schrieb Sami Ketola: >>>>> On 4 Sep 2018, at 17.47, Robert Schetterer <rs at sys4.de> wrote: >>>>> >>>>> Am 04.09.2018 um 09:41 schrieb Sami Ketola: >>>>>> imapsync always loses data >>>>> >>>>> never saw this, be carefull by anounce such myths >>>> >>>> It is a fact. Imapsync works over IMAP only and IMAP protocol >>>> does not even support transferring all data. At most at least UID >>>> numbering will be lost and end users need to invalidate their >>>> local caches. >>> >>> but using "looses data" might others think it also may fail with imap, >>> so be more detailed next time >> >> UID number is also data that is saved on the IMAP backend. If that >> is lost then it's "lost data". >> >> Sami > > Sorry i migrated terrabytes of mail with imapsync and never had a > problem, it works as designed, also with maildir rsync did a good job, > what never worked as it should was dsync ,cause of bugs ,that may > changed now > > so this is my answer to topic > > "Best way to move mail from one server to another" > > after all there is no "best way", there are a few ways > choose the one which fits best to your needs goals and starting points ( > kind of servers etc )Seems like there's an easy solution to the UID problem -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20180904/47653697/attachment.html>
Quoting Robert Schetterer <rs at sys4.de>:> Am 04.09.2018 um 17:18 schrieb Sami Ketola: >>> On 4 Sep 2018, at 18.00, Robert Schetterer <rs at sys4.de> wrote: >>> >>> Am 04.09.2018 um 16:52 schrieb Sami Ketola: >>>>> On 4 Sep 2018, at 17.47, Robert Schetterer <rs at sys4.de> wrote: >>>>> >>>>> Am 04.09.2018 um 09:41 schrieb Sami Ketola: >>>>>> imapsync always loses data >>>>> >>>>> never saw this, be carefull by anounce such myths >>>> >>>> It is a fact. Imapsync works over IMAP only and IMAP protocol >>>> does not even support transferring all data. At most at least UID >>>> numbering will be lost and end users need to invalidate their >>>> local caches. >>> >>> but using "looses data" might others think it also may fail with imap, >>> so be more detailed next time >> >> UID number is also data that is saved on the IMAP backend. If that >> is lost then it's "lost data". >> >> Sami > > Sorry i migrated terrabytes of mail with imapsync and never had a > problem, it works as designed, also with maildir rsync did a good job, > what never worked as it should was dsync ,cause of bugs ,that may > changed now > > so this is my answer to topic > > "Best way to move mail from one server to another" > ?https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt Seems to use UIDs so that 'data' isn't lost. -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20180904/46e3faf9/attachment.html>
> On 4 Sep 2018, at 18.45, Rick Romero <rick at havokmon.com> wrote: > https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt > > Seems to use UIDs so that 'data' isn't lost. > >No it does not. This is different thing. This is about managing duplicates on multiple syncs. Imapsync seems to keep track of UID -> MAIL mapping for server A and UID -> MAIL mapping for server B so that on multiple runs it does not sync the same messages again. However it does not use the same UID -> MAIL for server B as is originally on server A. And that is the problem. Sami -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20180904/6e6d331d/attachment-0001.html>
Sami is correct. imapsync loses data. Source -> imapsync -> destination : the mailboxes are different to an IMAP client Source -> dsync -> destination (running Dovecot w/doveadm): the mailboxes are the same You may not care that your users w/100,000 message mailboxes that have been cached on their client now have to invalidate that entire cache (and most likely re-download all of those messages). So there is zero argument that, due to IMAP metadata loss, this is something that doesn't happen if dsync (or rsync) is used. michael> On September 4, 2018 at 9:45 AM Rick Romero <rick at havokmon.com> wrote: > > > Quoting Robert Schetterer <rs at sys4.de mailto:rs at sys4.de >: > > > > > > Am 04.09.2018 um 17:18 schrieb Sami Ketola: > > > > > > > > > > > > > > > > > > On 4 Sep 2018, at 18.00, Robert Schetterer <rs at sys4.de mailto:rs at sys4.de > wrote: > > > > > > > > Am 04.09.2018 um 16:52 schrieb Sami Ketola: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 4 Sep 2018, at 17.47, Robert Schetterer <rs at sys4.de mailto:rs at sys4.de > wrote: > > > > > > > > > > > > Am 04.09.2018 um 09:41 schrieb Sami Ketola: > > > > > > > > > > > > > > > > > > > > > > > > > > imapsync always loses data > > > > > > > > > > > > > > > > > > > > never saw this, be carefull by anounce such myths > > > > > > > > > > > > > > > > > It is a fact. Imapsync works over IMAP only and IMAP protocol does not even support transferring all data. At most at least UID numbering will be lost and end users need to invalidate their local caches. > > > > > > > > > > > > > > but using "looses data" might others think it also may fail with imap, > > > > so be more detailed next time > > > > > > > > > > > UID number is also data that is saved on the IMAP backend. If that is lost then it's "lost data". > > > > > > Sami > > > > > > > > Sorry i migrated terrabytes of mail with imapsync and never had a > > problem, it works as designed, also with maildir rsync did a good job, > > what never worked as it should was dsync ,cause of bugs ,that may > > changed now > > > > so this is my answer to topic > > > > "Best way to move mail from one server to another" > > > > > > > > https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt > > Seems to use UIDs so that 'data' isn't lost. > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20180904/e76c8ec5/attachment.html>
> On 4 Sep 2018, at 18.38, Robert Schetterer <rs at sys4.de> wrote: > > Sorry i migrated terrabytes of mail with imapsync and never had a > problem, it works as designed, also with maildir rsync did a good job, > what never worked as it should was dsync ,cause of bugs ,that may > changed now >I would like to very much hear about the problems you had with dsync as we have successfully migrated tens of petabytes of mails with it successfully while preserving the UID numbers. Imapsync would have been totally out of question as it is absolutely crucial in those migrations to keep the UID -> MAIL pairs matching legacy server. And usually also POP3 UIDLs. If not keeping the data the servers would have literally melted under load when switching over then of thousands of users forcing them to redownload headers of even mail bodies for millions of mails. Sami
On 9/4/18 12:27 PM, Sami Ketola wrote:> > >> On 4 Sep 2018, at 18.38, Robert Schetterer <rs at sys4.de> wrote: >> >> Sorry i migrated terrabytes of mail with imapsync and never had a >> problem, it works as designed, also with maildir rsync did a good job, >> what never worked as it should was dsync ,cause of bugs ,that may >> changed now >> > > I would like to very much hear about the problems you had with dsync as we have successfully migrated tens of petabytes of mails with it successfully while preserving the UID numbers. > Imapsync would have been totally out of question as it is absolutely crucial in those migrations to keep the UID -> MAIL pairs matching legacy server. And usually also POP3 UIDLs. If not keeping the data the servers would have literally melted under load when switching over then of thousands of users forcing them to redownload headers of even mail bodies for millions of mails. > > Sami > >why does not rsync work? Ruben -- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www.brooklyn-living.com Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
Am 04.09.2018 um 18:27 schrieb Sami Ketola:> > >> On 4 Sep 2018, at 18.38, Robert Schetterer <rs at sys4.de> wrote: >> >> Sorry i migrated terrabytes of mail with imapsync and never had a >> problem, it works as designed, also with maildir rsync did a good job, >> what never worked as it should was dsync ,cause of bugs ,that may >> changed now >> > > I would like to very much hear about the problems you had with dsync as we have successfully migrated tens of petabytes of mails with it successfully while preserving the UID numbers. > Imapsync would have been totally out of question as it is absolutely crucial in those migrations to keep the UID -> MAIL pairs matching legacy server. And usually also POP3 UIDLs. If not keeping the data the servers would have literally melted under load when switching over then of thousands of users forcing them to redownload headers of even mail bodies for millions of mails. > > Sami > >I think you should know the dove buglist, i used dovecot since the beginning and believe me dsync had a lot of problems, as i stated i hope this had changed now. I didnt used/tested it again the last years cause i had no big migration job to do, we ever tried to use it that days cause youre right we would have prefer not broke pop3 stuff at migration. But it never worked, so imapsync was the best way to go, also having courier, cyrus migrations with format/namespace change etc. Imapsync works as designed and migration might not be ideal at some points but it has no real stoppers if migration is well prepared and organized. Best Regards MfG Robert Schetterer -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Schlei?heimer Stra?e 26/MG, 80333 M?nchen Sitz der Gesellschaft: M?nchen, Amtsgericht M?nchen: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein