Greeting I've managed to backup my emails from external email server via dsync and imapc: protocol like this: doveadm -Dv -o imapc_user='user at gmail.com' -o imapc_password="pass" -o imapc_host=imap.gmail.com backup -R -u duke at ejoom.com imapc: taken from https://wiki.dovecot.org/Migration/Dsync I wish to do regular syncing of 5min interval to substitute 'getmail' program. When I repeat the command It doesn't download all the attachments(!good) but It seems that at retrieve all the the GUIDs and UIDs(!bad) again that take quite long considering 30K emails. The question is how to speed it up and retrieve only new emails? Does anyone have a solution for regular synchronization? What is the best way? Logs look like this: https://gist.githubusercontent.com/vnagara/87e0da1b21b51f68fd74270b3ec653b3/raw/70ebd6d98338aa80c33ecabf7f108f5e7ce10d49/dsync%2520logs.log It wrights that "Mailboxes are equal" but still verifies all the UIDs again. Thanks. -------------- next part -------------- A non-text attachment was scrubbed... Name: dsync.log Type: text/x-log Size: 34257 bytes Desc: not available URL: <https://dovecot.org/pipermail/dovecot/attachments/20200711/cf850180/attachment-0001.bin>
> On 11. Jul 2020, at 14.22, Vitalii <vnagara at yandex.com> wrote: > > Greeting > > I've managed to backup my emails from external email server via dsync > and imapc: protocol like this: > > doveadm -Dv -o imapc_user='user at gmail.com' -o imapc_password="pass" > -o imapc_host=imap.gmail.com backup -R -u duke at ejoom.com imapc: >It basicly does it to verify mail -> UID pairing. You can try to save imapc indexes locally by defining imapc location as: imapc:~/.imapc to see if it can be skipped. Sami
On Sat, 11 Jul 2020 20:36:44 +0300 Sami Ketola <sami.ketola at dovecot.fi> wrote:> > On 11. Jul 2020, at 14.22, Vitalii <vnagara at yandex.com> wrote: > > > > Greeting > > > > I've managed to backup my emails from external email server via > > dsync and imapc: protocol like this: > > > > doveadm -Dv -o imapc_user='user at gmail.com' -o imapc_password="pass" > > -o imapc_host=imap.gmail.com backup -R -u duke at ejoom.com imapc: > > > > It basicly does it to verify mail -> UID pairing. You can try to save > imapc indexes locally by defining imapc location as: imapc:~/.imapc > to see if it can be skipped. > > Sami >That's awesome Sami, thanks a lot. For gmail.com account with 43000 emails it went down from 6 mins to 15seconds (24 times faster). For yandex.com account with 15000 emails it went down from 32min to 28seconds (60 times faster). I started testing it more and found out that it works as a charm with new emails and move operations. However, it fails when I delete (expunge) email on gmail server. On backup I get this error: Panic: file imapc-sync.c: line 328 (imapc_initial_sync_check): assertion failed: (mail_index_is_expunged(view, lseq) || seq_range_exists(&ctx->mbox->delayed_expunged_uids, luid)) (Attached a bigger log) -------------- next part -------------- A non-text attachment was scrubbed... Name: dsync-error.log Type: text/x-log Size: 2387 bytes Desc: not available URL: <https://dovecot.org/pipermail/dovecot/attachments/20200712/02a51bfb/attachment.bin>
Am 11.07.2020 um 13:22 schrieb Vitalii:> Greeting > > I've managed to backup my emails from external email server via dsync > and imapc: protocol like this: > > doveadm -Dv -o imapc_user='user at gmail.com' -o imapc_password="pass" > -o imapc_host=imap.gmail.com backup -R -u duke at ejoom.com imapc: > > taken from https://wiki.dovecot.org/Migration/Dsync > > I wish to do regular syncing of 5min interval to substitute 'getmail' > program. > > When I repeat the command It doesn't download all the attachments(!good) > but It seems that at retrieve all the the GUIDs and UIDs(!bad) again > that take quite long considering 30K emails. > The question is how to speed it up and retrieve only new emails? Does > anyone have a solution for regular synchronization? What is the best > way? > > > Logs look like this: > https://gist.githubusercontent.com/vnagara/87e0da1b21b51f68fd74270b3ec653b3/raw/70ebd6d98338aa80c33ecabf7f108f5e7ce10d49/dsync%2520logs.log > > It wrights that "Mailboxes are equal" but still verifies all the UIDs > again. > > > Thanks. >perhaps this help getmail should use /usr/lib/dovecot/deliver https://blog.sys4.de/abholdienst-fur-mail-de.html and/or imapsync is your friend https://blog.sys4.de/mail-migration-mit-imapsync-de.html -- [*] sys4 AG https://sys4.de, +49 (89) 30 90 46 64 Schlei?heimer Stra?e 26/MG,80333 M?nchen Sitz der Gesellschaft: M?nchen, Amtsgericht M?nchen: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer, Wolfgang Stief Aufsichtsratsvorsitzender: Florian Kirstein
On Mon, 13 Jul 2020 10:50:09 +0200 Robert Schetterer <rs at sys4.de> wrote:> perhaps this help > > getmail should use /usr/lib/dovecot/deliverThat is what I used before and wanted to switch off. Here is the 2 main reasons: 1) It add extra header overhead similar to: Received by localost at time, etc....) 2) It put emails in local delivery queue so syncing becomes slow. 3) only backup, no syncing back.> > https://blog.sys4.de/abholdienst-fur-mail-de.html > > and/or imapsync is your friendDon't know about imapsync but I believe it has the same disadvantages as it isn't native dovecot.> > https://blog.sys4.de/mail-migration-mit-imapsync-de.html >