dovecot - Jan 2016 - How to skip errors on doveadm backup for migration ?

Hi there,

I have to migrate data out of a very old (debian 4) and buggy cyrus mailserver.
About 40 Users with 200000 Messages each.

Doveadm backup works quite well and gives me most of the mails. About every
100.000 Mail
cannot be exported by cyrus. I do not want to bother about these errors.
Maybe it is filesystem, maybe it is memory leak, or cyrus bug or whatever.

My problem is that doveadm backup exits with an error if it cannot read a mail.



Is there some way or option to make doveadm backup just skip the errors and
continue with the next message ?

I want to use doveadm sync because it preserves the uids of the mails.

Imapsync works great but does not prserve uids.

cyrus2dovecot is not an option because it does not work incremental and
conversion would take far too long.

Any ideas what I can do ?

Thanks
  Christoph

On 08 Jan 2016, at 12:50, Christoph Adomeit <Christoph.Adomeit at
gatworks.de> wrote:
> 
> Hi there,
> 
> I have to migrate data out of a very old (debian 4) and buggy cyrus
mailserver.
> About 40 Users with 200000 Messages each.
> 
> Doveadm backup works quite well and gives me most of the mails. About every
100.000 Mail
> cannot be exported by cyrus. I do not want to bother about these errors.
> Maybe it is filesystem, maybe it is memory leak, or cyrus bug or whatever.
> 
> My problem is that doveadm backup exits with an error if it cannot read a
mail.
What does Cyrus reply? Is it a NO reply to FETCH command?
> Is there some way or option to make doveadm backup just skip the errors and
continue with the next message ?
Not currently. You could try if the attached patch helps. It should write the
mail as empty.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: diff
Type: application/octet-stream
Size: 546 bytes
Desc: not available
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20160108/60e67c97/attachment.obj>
-------------- next part --------------

Hi Timo,

thanks for the patch.

I applied it, compiled doveadm and tried the new binary but it still baily out
with this message:


dsync(user at domain.de): Error: Exporting mailbox INBOX.Sent failed: Can't
lookup saved-date for UID=515: Internal error occurred. Refer to server log for
more information. [2016-01-10 17:02:08]

Do you have any idea how I can skip these errors ?
Thanks
  Christoph

On Fri, Jan 08, 2016 at 05:39:51PM +0200, Timo Sirainen
wrote:
> On 08 Jan 2016, at 12:50, Christoph Adomeit <Christoph.Adomeit at
gatworks.de> wrote:
> > 
> > Hi there,
> > 
> > I have to migrate data out of a very old (debian 4) and buggy cyrus
mailserver.
> > About 40 Users with 200000 Messages each.
> > 
> > Doveadm backup works quite well and gives me most of the mails. About
every 100.000 Mail
> > cannot be exported by cyrus. I do not want to bother about these
errors.
> > Maybe it is filesystem, maybe it is memory leak, or cyrus bug or
whatever.
> > 
> > My problem is that doveadm backup exits with an error if it cannot
read a mail.
> 
> What does Cyrus reply? Is it a NO reply to FETCH command?
> 
> > Is there some way or option to make doveadm backup just skip the
errors and continue with the next message ?
> 
> Not currently. You could try if the attached patch helps. It should write
the mail as empty.
> 
> 
>

If both servers are using IMAP, maybe imapsync 
(http://imapsync.lamiral.info) would be an easier way to go...

Farokh
----------------------------------------------------------------------------
MCF Software...simply dependably engineered
For all your computer and networking needs including hosting solutions
for every need.
Phone: 845-735-0210
Cell: 914-262-1594


> Christoph Adomeit <mailto:Christoph.Adomeit at gatworks.de>
> January 8, 2016 at 05:50
> Hi there,
>
> I have to migrate data out of a very old (debian 4) and buggy cyrus 
> mailserver.
> About 40 Users with 200000 Messages each.
>
> Doveadm backup works quite well and gives me most of the mails. About 
> every 100.000 Mail
> cannot be exported by cyrus. I do not want to bother about these errors.
> Maybe it is filesystem, maybe it is memory leak, or cyrus bug or whatever.
>
> My problem is that doveadm backup exits with an error if it cannot 
> read a mail.
>
>
>
> Is there some way or option to make doveadm backup just skip the 
> errors and continue with the next message ?
>
> I want to use doveadm sync because it preserves the uids of the mails.
>
> Imapsync works great but does not prserve uids.
>
> cyrus2dovecot is not an option because it does not work incremental 
> and conversion would take far too long.
>
> Any ideas what I can do ?
>
> Thanks
> Christoph

Hi Farokh,

yes, imapsync is great. I tested it myself and it fetched all the mails.

Only Problem: Imapsync will loose the Email-UIDs so all Mail will be downloaded
again by the clients after Server switch.

dsync is able to presere the Uids.

Thanks
  Christoph

On Sun, Jan 10, 2016 at 05:42:53PM -0500, Farokh Irani
wrote:
> If both servers are using IMAP, maybe imapsync
> (http://imapsync.lamiral.info) would be an easier way to go...
> 
> Farokh
>
----------------------------------------------------------------------------
> MCF Software...simply dependably engineered
> For all your computer and networking needs including hosting solutions
> for every need.
> Phone: 845-735-0210
> Cell: 914-262-1594
> 
> 
> 
> >Christoph Adomeit <mailto:Christoph.Adomeit at gatworks.de>
> >January 8, 2016 at 05:50
> >Hi there,
> >
> >I have to migrate data out of a very old (debian 4) and buggy cyrus
> >mailserver.
> >About 40 Users with 200000 Messages each.
> >
> >Doveadm backup works quite well and gives me most of the mails. About
> >every 100.000 Mail
> >cannot be exported by cyrus. I do not want to bother about these
errors.
> >Maybe it is filesystem, maybe it is memory leak, or cyrus bug or
whatever.
> >
> >My problem is that doveadm backup exits with an error if it cannot read
a
> >mail.
> >
> >
> >
> >Is there some way or option to make doveadm backup just skip the errors
> >and continue with the next message ?
> >
> >I want to use doveadm sync because it preserves the uids of the mails.
> >
> >Imapsync works great but does not prserve uids.
> >
> >cyrus2dovecot is not an option because it does not work incremental and
> >conversion would take far too long.
> >
> >Any ideas what I can do ?
> >
> >Thanks
> >Christoph
-- 
Christoph Adomeit
GATWORKS GmbH
Reststrauch 191
41199 Moenchengladbach
Sitz: Moenchengladbach
Amtsgericht Moenchengladbach, HRB 6303
Geschaeftsfuehrer:
Christoph Adomeit, Hans Wilhelm Terstappen

Christoph.Adomeit at gatworks.de     Internetloesungen vom Feinsten
Fon. +49 2166 9149-32                      Fax. +49 2166 9149-10