thr3ads.net - dovecot - [Dovecot] Mail migration to dovecot with doveadm backup [Jun 2012]

If this information is useful, please help other people find it:
Share via:

Reinhard Vicinus

2012-Jun-24 15:57 UTC

[Dovecot] Mail migration to dovecot with doveadm backup

Hi,

i try to migrate mails from a non dovecot imap server to a dovecot imap 
server with doveadm backup as described there:

http://wiki2.dovecot.org/Migration/Dsync

i first tried (local-mailbox port 18143 is the non dovecot imap server):

/usr/bin/doveadm -o imapc_user=user at example.org -o imapc_password=imappw 
-o imapc_host=local-mailbox -o imapc_features=rfc822.size -o 
imapc_port=18143 -o mailbox_list_index=no -v -D backup -R -f -u 
user at example.org -m Sent imapc:

and got the following error:

dsync(user at example.org): Fatal: dsync backup: Looks like you're trying 
to run backup in wrong direction. Source is empty and destination is not.

As the dovecot imap account is newly created and therefore empty it 
seams to try to backup from the dovecot imap server to the non dovecot 
imap server. So i tried instead:

/usr/bin/doveadm -o imapc_user=user at example.org -o imapc_password=imappw 
-o imapc_host=local-mailbox -o imapc_features=rfc822.size -o 
imapc_port=18143 -o mailbox_list_index=no -v -D backup -f -u 
user at example.org -m Sent imapc:

Sometimes (every other time?) i got the following segmentation fault:

bt
#0  0x00007f15e2c9ed74 in strcasecmp () from /lib/libc.so.6
#1  0x00007f15e327eaff in imapc_save_callback (reply=0x7fff56096a70, 
context=<value optimized out>) at imapc-save.c:168
#2  0x00007f15e32853fe in imapc_command_reply_free (conn=0x72f040) at 
imapc-connection.c:946
#3  imapc_connection_input_tagged (conn=0x72f040) at imapc-connection.c:1039
#4  0x00007f15e3285668 in imapc_connection_input_one (conn=0x72f040) at 
imapc-connection.c:1085
#5  imapc_connection_input_pending (conn=0x72f040) at 
imapc-connection.c:1407
#6  0x00007f15e3285922 in imapc_connection_input (conn=0x72f040) at 
imapc-connection.c:1100
#7  0x00007f15e2fe6176 in io_loop_call_io (io=0x792510) at ioloop.c:379
#8  0x00007f15e2fe71ff in io_loop_handler_run (ioloop=<value optimized 
out>) at ioloop-epoll.c:213
#9  0x00007f15e2fe6118 in io_loop_run (ioloop=0x7529a0) at ioloop.c:398
#10 0x00007f15e3281e49 in imapc_client_run_pre (client=0x7333e0) at 
imapc-client.c:142
#11 imapc_client_run (client=0x7333e0) at imapc-client.c:161
#12 0x00007f15e3280f24 in imapc_storage_run (storage=0x732bd0) at 
imapc-storage.c:118
#13 0x00007f15e327f003 in imapc_save_append (_ctx=0x74dcb0) at 
imapc-save.c:232
#14 imapc_save_finish (_ctx=0x74dcb0) at imapc-save.c:255
#15 0x00007f15e1bf06a1 in quota_save_finish (ctx=0x74dcb0) at 
quota-storage.c:227
#16 0x00007f15e3292487 in mailbox_save_finish (_ctx=0x7f15e2d4ca40) at 
mail-storage.c:1669
#17 0x000000000042b736 in local_worker_save_msg_continue 
(worker=0x73c770) at dsync-worker-local.c:1681
#18 0x000000000042b98c in local_worker_msg_save (_worker=0x73c770, 
msg=0x7f15e38e4298, data=0x7fff56096db0, callback=0x4269f0 
<msg_save_callback>, context=0x78a610) at dsync-worker-local.c:1739
#19 0x000000000042b0d9 in dsync_worker_msg_save (worker=0x73c770, 
msg=<value optimized out>, data=0x7fff56096db0, callback=0x4269f0 
<msg_save_callback>, context=0x78a610) at dsync-worker.c:234
#20 0x0000000000426ac5 in msg_get_callback (result=<value optimized 
out>, data=0x7fff56096db0, context=0x78a610) at dsync-brain-msgs-new.c:79
#21 0x000000000042dca9 in local_worker_msg_get_next (worker=0x726f30, 
get=0x7fff56096e00) at dsync-worker-local.c:1844
#22 0x000000000042def8 in local_worker_msg_get (_worker=0x0, 
mailbox=<value optimized out>, uid=3805596224, callback=0x2670, 
context=0x58) at dsync-worker-local.c:1865
#23 0x000000000042ace6 in dsync_worker_msg_get (worker=0x726f30, 
mailbox=0x7f15e38e40f1, uid=1, callback=0x426a40 <msg_get_callback>, 
context=<value optimized out>) at dsync-worker.c:261
#24 0x000000000042689e in dsync_brain_msg_sync_add_new_msg 
(iter=0x7f15e38e41d8) at dsync-brain-msgs-new.c:181
#25 dsync_brain_mailbox_add_new_msgs (iter=0x7f15e38e41d8) at 
dsync-brain-msgs-new.c:216
#26 dsync_brain_msg_sync_add_new_msgs (iter=0x7f15e38e41d8) at 
dsync-brain-msgs-new.c:315
#27 0x0000000000426164 in dsync_brain_msg_sync_more 
(sync=0x7f15e38e4050) at dsync-brain-msgs.c:436
#28 0x0000000000424979 in dsync_brain_sync_msgs (brain=0x7351c0) at 
dsync-brain.c:736
#29 dsync_brain_sync (brain=0x7351c0) at dsync-brain.c:857
#30 0x0000000000425849 in dsync_brain_subs_list_finished 
(context=0x750fa0) at dsync-brain.c:169
#31 dsync_worker_subs_input (context=0x750fa0) at dsync-brain.c:222
#32 0x0000000000424cbd in dsync_brain_sync (brain=0x7351c0) at 
dsync-brain.c:842
#33 0x00000000004256bc in dsync_brain_mailbox_list_finished 
(context=0x743a60) at dsync-brain.c:98
#34 dsync_worker_mailbox_input (context=0x743a60) at dsync-brain.c:125
#35 0x0000000000424afe in dsync_brain_sync (brain=0x7351c0) at 
dsync-brain.c:833
#36 0x0000000000425568 in dsync_brain_sync_all (brain=0x7351c0) at 
dsync-brain.c:897
#37 0x0000000000422ad6 in cmd_dsync_start (ctx=0x706560, worker1=<value 
optimized out>, worker2=<value optimized out>) at doveadm-dsync.c:342
#38 0x0000000000422dfa in cmd_dsync_run (_ctx=0x706560, user=0x7209c0) 
at doveadm-dsync.c:387
#39 0x000000000040f888 in doveadm_mail_next_user (ctx=0x706560, 
input=<value optimized out>, error_r=0x7fff560973d0) at doveadm-mail.c:311
#40 0x0000000000410071 in doveadm_mail_cmd (cmd=0x704880, argc=7, 
argv=0x6fe418) at doveadm-mail.c:518
#41 0x0000000000410501 in doveadm_mail_try_run (cmd_name=0x6fe4e4 
"backup", argc=1443460960, argv=0x4377c7) at doveadm-mail.c:577
#42 0x00000000004170d1 in main (argc=7, argv=0x6fe3e8) at doveadm.c:373


The other times it shows the following error message (It seams to 
connect both times to the non dovecot imap server):

doveadm(root): Debug: Loading modules from directory: 
/usr/lib/dovecot/modules
doveadm(root): Debug: Module loaded: 
/usr/lib/dovecot/modules/lib10_quota_plugin.so
doveadm(root): Debug: Loading modules from directory: 
/usr/lib/dovecot/modules/doveadm
doveadm(root): Debug: Skipping module doveadm_acl_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined 
symbol: acl_user_module (this is usually intentional, so just ignore 
this message)
doveadm(root): Debug: Skipping module doveadm_expire_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: 
undefined symbol: expire_set_lookup (this is usually intentional, so 
just ignore this message)
doveadm(root): Debug: Module loaded: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so
doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_zlib_plugin.so: undefined 
symbol: i_stream_create_deflate (this is usually intentional, so just 
ignore this message)
doveadm(root): Debug: Skipping module doveadm_fts_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined 
symbol: fts_list_backend (this is usually intentional, so just ignore 
this message)
doveadm(user at example.org): Debug: auth input: user at example.org 
home=/mail/dovecot/example.org/user uid=1000 gid=1000 
quota_rule=*:bytes=2000M:messages=0
doveadm(user at example.org): Debug: Added userdb setting: 
plugin/quota_rule=*:bytes=2000M:messages=0
doveadm(user at example.org): Debug: Effective uid=1000, gid=1000, 
home=/mail/dovecot/example.org/user
doveadm(user at example.org): Debug: Quota root: name=User quota 
backend=dict args=:proxy::quota
doveadm(user at example.org): Debug: Quota rule: root=User quota mailbox=* 
bytes=2097152000 messages=0
doveadm(user at example.org): Debug: Quota rule: root=User quota 
mailbox=Trash bytes=+104857600 messages=0
doveadm(user at example.org): Debug: Quota warning: bytes=1992294400 (95%) 
messages=0 reverse=no command=quota-warning 95 user at example.org
doveadm(user at example.org): Debug: Quota warning: bytes=1677721600 (80%) 
messages=0 reverse=no command=quota-warning 80 user at example.org
doveadm(user at example.org): Debug: dict quota: user=user at example.org, 
uri=proxy::quota, noenforcing=0
doveadm(user at example.org): Debug: fs: 
root=/mail/dovecot/example.org/user/mail, index=, control=, inbox=,
altdoveadm(user at example.org): Debug: Namespace : Using permissions from
/mail/dovecot/example.org/user/mail: mode=0700 gid=-1
dsync(user at example.org): Debug: Effective uid=1000, gid=1000, 
home=/mail/dovecot/example.org/user
dsync(user at example.org): Debug: Quota root: name=User quota backend=dict 
args=:proxy::quota
dsync(user at example.org): Debug: Quota rule: root=User quota mailbox=* 
bytes=2097152000 messages=0
dsync(user at example.org): Debug: Quota rule: root=User quota 
mailbox=Trash bytes=+104857600 messages=0
dsync(user at example.org): Debug: Quota warning: bytes=1992294400 (95%) 
messages=0 reverse=no command=quota-warning 95 user at example.org
dsync(user at example.org): Debug: Quota warning: bytes=1677721600 (80%) 
messages=0 reverse=no command=quota-warning 80 user at example.org
dsync(user at example.org): Debug: dict quota: user=user at example.org, 
uri=proxy::quota, noenforcing=0
dsync(user at example.org): Debug: imapc: root=, index=, control=, inbox=,
altdsync(user at example.org): Debug: imapc(local-mailbox:18143): Looking up
IP address
dsync(user at example.org): Debug: imapc(local-mailbox:18143): Connecting 
to 10.10.10.10:18143
dsync(user at example.org): Debug: imapc(local-mailbox:18143): Server 
capabilities: IMAP4 IMAP4rev1 AUTH=LOGIN ACL NAMESPACE CHILDREN SORT 
QUOTA THREAD=ORDEREDSUBJECT UNSELECT IDLE
dsync(user at example.org): Debug: imapc(local-mailbox:18143): 
Authenticating as user at example.org
dsync(user at example.org): Debug: imapc(local-mailbox:18143): 
Authenticated successfully
dsync(user at example.org): Debug: imapc(local-mailbox:18143): Looking up 
IP address
dsync(user at example.org): Debug: imapc(local-mailbox:18143): Connecting 
to 10.10.10.10:18143
dsync(user at example.org): Debug: imapc(local-mailbox:18143): Server 
capabilities: IMAP4 IMAP4rev1 AUTH=LOGIN ACL NAMESPACE CHILDREN SORT 
QUOTA THREAD=ORDEREDSUBJECT UNSELECT IDLE
dsync(user at example.org): Debug: imapc(local-mailbox:18143): 
Authenticating as user at example.org
dsync(user at example.org): Debug: imapc(local-mailbox:18143): 
Authenticated successfully
dsync(user at example.org): Warning: Destination mailbox Sent has been 
modified, need to recreate it before we can continue syncing
dsync(user at example.org): Warning: Mailbox changes caused a desync. You 
may want to run dsync again.
dsync(user at example.org): Debug: imapc(local-mailbox:18143): Disconnected
dsync(user at example.org): Debug: imapc(local-mailbox:18143): Disconnected

i think the problem could be that the account name on the remote server 
and the local server is absolute identical and doveadm backup has 
therefore problems discerning between the two locations. But that's only 
a stab in the dark and any help is appreciated.

Kind regards
Reinhard

Charles Marcus

2012-Jun-25 11:39 UTC

head link

[Dovecot] Mail migration to dovecot with doveadm backup

On 2012-06-24 11:57 AM, Reinhard Vicinus <r.vicinus at metaways.de>
wrote:> i try to migrate mails from a non dovecot imap server to a dovecot imap
> server with doveadm backup as described there:
What version of dovecot (doveconf -n output_?
> http://wiki2.dovecot.org/Migration/Dsync
>
> i first tried (local-mailbox port 18143 is the non dovecot imap server):
>
> /usr/bin/doveadm -o imapc_user=user at example.org -o imapc_password=imappw
> -o imapc_host=local-mailbox -o imapc_features=rfc822.size -o
> imapc_port=18143 -o mailbox_list_index=no -v -D backup -R -f -u
> user at example.org -m Sent imapc:
>
> and got the following error:
>
> dsync(user at example.org): Fatal: dsync backup: Looks like you're
trying
> to run backup in wrong direction. Source is empty and destination is not.

-- 

Best regards,

Charles Marcus
I.T. Director
Media Brokers International, Inc.
678.514.6200 x224 | 678.514.6299 fax

Reinhard Vicinus

2012-Jun-25 12:55 UTC

head link

[Dovecot] Mail migration to dovecot with doveadm backup

On 25/06/12 13:39, Charles Marcus wrote:> On 2012-06-24 11:57 AM, Reinhard Vicinus <r.vicinus at metaways.de>
wrote:
>> i try to migrate mails from a non dovecot imap server to a dovecot imap
>> server with doveadm backup as described there:
>
> What version of dovecot (doveconf -n output_? dovecot -n
# 2.1.7: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-40-server x86_64 Ubuntu 10.04.4 LTS
auth_cache_negative_ttl = 0
auth_cache_size = 10 M
auth_cache_ttl = 1 mins
auth_verbose = yes
auth_verbose_passwords = sha1
deliver_log_format = mailbox: deliver: msgid=%m from=%f: %$
dict {
   quota = mysql:/etc/dovecot/conf.d/dovecot-dict-sql.conf.ext
}
disable_plaintext_auth = no
doveadm_password = xxx
instance_name = dovecot-mailbox
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
login_greeting = Mailbox
login_log_format = mailbox: login: %$: %s
login_trusted_networks = 10.10.10.0/24
mail_debug = yes
mail_fsync = always
mail_gid = vmail
mail_home = /mail/dovecot/%d/%n
mail_location = mdbox:~/mail
mail_log_prefix = "mailbox: mail: %s(%u): "
mail_plugins = quota
mail_privileged_group = vmail
mail_uid = vmail
managesieve_implementation_string = Sieve
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date ihave
mdbox_rotate_interval = 1 weeks
mdbox_rotate_size = 50 M
mmap_disable = yes
passdb {
   args = /etc/dovecot/conf.d/dovecot-sql.conf.ext
   driver = sql
}
plugin {
   quota = dict:User quota::proxy::quota
   quota_rule = *:storage=10G
   quota_rule2 = Trash:storage=+100M
   quota_warning = storage=95%% quota-warning 95 %u
   quota_warning2 = storage=80%% quota-warning 80 %u
   sieve = ~/.dovecot.sieve
   sieve_dir = ~/sieve
}
protocols = imap pop3 lmtp sieve
service auth {
   unix_listener auth-userdb {
     group = dovecot
     mode = 0660
     user = dovecot
   }
}
service dict {
   unix_listener dict {
     group = vmail
     mode = 0660
   }
}
service doveadm {
   inet_listener doveadm-server {
     port = 19000
   }
}
service imap-login {
   inet_listener imap {
     port = 19143
   }
}
service imap-postlogin {
   executable = script-login /usr/local/bin/dovecot-postlogin
   user = $default_internal_user
}
service imap {
   executable = imap imap-postlogin
}
service lmtp {
   inet_listener lmtp {
     address = *
     port = 19024
   }
}
service managesieve-login {
   inet_listener sieve {
     port = 19200
   }
}
service pop3-login {
   inet_listener pop3 {
     port = 19110
   }
}
service pop3-postlogin {
   executable = script-login /usr/local/bin/dovecot-postlogin
   user = $default_internal_user
}
service pop3 {
   executable = pop3 pop3-postlogin
}
service quota-warning {
   executable = script /usr/local/bin/quota-warning
   extra_groups = dovecot
   unix_listener quota-warning {
     user = vmail
   }
   user = vmail
}
ssl = no
userdb {
   driver = prefetch
}
userdb {
   args = /etc/dovecot/conf.d/dovecot-sql.conf.ext
   driver = sql
}
verbose_proctitle = yes
protocol imap {
   imap_client_workarounds = delay-newmail tb-extra-mailbox-sep
   mail_plugins = quota imap_quota
}
protocol lmtp {
   mail_plugins = quota sieve
}

Timo Sirainen

2012-Jun-25 16:46 UTC

head link

[Dovecot] Mail migration to dovecot with doveadm backup

On 24.6.2012, at 18.57, Reinhard Vicinus wrote:
> i try to migrate mails from a non dovecot imap server to a dovecot imap
server with doveadm backup as described there:
> 
> http://wiki2.dovecot.org/Migration/Dsync
> 
> i first tried (local-mailbox port 18143 is the non dovecot imap server):
> 
> /usr/bin/doveadm -o imapc_user=user at example.org -o imapc_password=imappw
-o imapc_host=local-mailbox -o imapc_features=rfc822.size -o imapc_port=18143 -o
mailbox_list_index=no -v -D backup -R -f -u user at example.org -m Sent imapc:
> 
> and got the following error:
> 
> dsync(user at example.org): Fatal: dsync backup: Looks like you're
trying to run backup in wrong direction. Source is empty and destination is not.
Strange. -R is supposed to make it copy from imapc to mdbox.. Have you tried if
Dovecot can see mails at all from the remote server? Try doveadm -o mail=imapc:
-o ... fetch instead of doveadm backup command.
> As the dovecot imap account is newly created and therefore empty it seams
to try to backup from the dovecot imap server to the non dovecot imap server. So
i tried instead:
> 
> /usr/bin/doveadm -o imapc_user=user at example.org -o imapc_password=imappw
-o imapc_host=local-mailbox -o imapc_features=rfc822.size -o imapc_port=18143 -o
mailbox_list_index=no -v -D backup -f -u user at example.org -m Sent imapc:
Now this is copying from mdbox to imapc, which is also why you're getting
the crash:
> Sometimes (every other time?) i got the following segmentation fault:
> 
> bt
> #0  0x00007f15e2c9ed74 in strcasecmp () from /lib/libc.so.6
> #1  0x00007f15e327eaff in imapc_save_callback (reply=0x7fff56096a70,
context=<value optimized out>) at imapc-save.c:168
Note how it's saving a mail to imapc. But still, that's a bug, fixed:
http://hg.dovecot.org/dovecot-2.1/rev/20703dbd1168
> dsync(user at example.org): Warning: Destination mailbox Sent has been
modified, need to recreate it before we can continue syncing
I think this is also because it's going to wrong direction.
> i think the problem could be that the account name on the remote server and
the local server is absolute identical and doveadm backup has therefore problems
discerning between the two locations. But that's only a stab in the dark and
any help is appreciated.
Shouldn't be a problem.

Maybe Matching Threads

Search for more possibly parallel threads

dovecot - Jun 2012 - Mail migration to dovecot with doveadm backup

[Dovecot] Mail migration to dovecot with doveadm backup

[Dovecot] Mail migration to dovecot with doveadm backup

[Dovecot] Mail migration to dovecot with doveadm backup

[Dovecot] Mail migration to dovecot with doveadm backup

Maybe Matching Threads