Hi,
I'm running dsync migrations using imapc and the source IMAP server is
just too slow. It has taken 8 hours to migrate a mailbox with 47,000
messages. It seems most of the mailboxes are never accessed with IMAP,
so it would be just as good to migrate them via POP3 which should work
faster in my case.
Is this actually meant to work?
doveadm -D -o pop3c_user=xxxxxx at example.com -o
'pop3c_password=*******'
-o pop3c_host=120.121.122.123 backup -u yyyyy at example.com -R pop3c:
gives:
doveadm(root): Debug: Loading modules from directory:
/usr/lib/dovecot/modules
doveadm(root): Debug: Module loaded:
/usr/lib/dovecot/modules/lib20_listescape_plugin.so
doveadm(root): Debug: Loading modules from directory:
/usr/lib/dovecot/modules/doveadm
doveadm(root): Debug: Skipping module doveadm_acl_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined
symbol: acl_user_module (this is usually intentional, so just ignore
this message)
doveadm(root): Debug: Skipping module doveadm_expire_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so:
undefined symbol: expire_set_init (this is usually intentional, so just
ignore this message)
doveadm(root): Debug: Skipping module doveadm_quota_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so:
undefined symbol: quota_user_module (this is usually intentional, so
just ignore this message)
doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_zlib_plugin.so: undefined
symbol: i_stream_create_deflate (this is usually intentional, so just
ignore this message)
doveadm(root): Debug: Skipping module doveadm_fts_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined
symbol: fts_list_backend (this is usually intentional, so just ignore
this message)
doveadm(yyyyy at example.com): Debug: Added userdb setting:
mail=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir
doveadm(yyyyy at example.com): Debug: Added userdb setting:
plugin/quota_rule=*:storage=5120M
doveadm(yyyyy at example.com): Debug: Effective uid=1001, gid=1001,
home=/stor/real/domains/example.com/yy/yyyyy
doveadm(yyyyy at example.com): Debug: Namespace : type=private,
prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no,
subscriptions=yes location=pop3c:
doveadm(yyyyy at example.com): Debug: fs: root=, index=, control=, inbox=,
altdoveadm(yyyyy at example.com): Debug: Namespace inbox: type=private,
prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes
location=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir
doveadm(yyyyy at example.com): Debug: maildir++:
root=/stor/real/domains/example.com/yy/yyyyy/Maildir, index=, control=,
inbox=/stor/real/domains/example.com/yy/yyyyy/Maildir, altdsync(yyyyy at
example.com): Debug: Effective uid=1001, gid=1001,
home=/stor/real/domains/example.com/yy/yyyyy
dsync(yyyyy at example.com): Debug: Namespace : type=private,
prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no,
subscriptions=yes location=pop3c:
dsync(yyyyy at example.com): Debug: fs: root=, index=, control=, inbox=,
altdsync(yyyyy at example.com): Debug: Namespace inbox: type=private, prefix=,
sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=pop3c:
dsync(yyyyy at example.com): Debug: fs: root=, index=, control=, inbox=,
altSegmentation fault
remembering something about that... :
# mkdir /tmp/pop3c
# chown vmail:vmail /tmp/pop3c/
# doveadm -D -o pop3c_user=xxxxxx at example.com -o
'pop3c_password=*******' -o pop3c_host=120.121.122.123 backup -u
yyyyy at example.com -R pop3c:/tmp/pop3c
doveadm(root): Debug: Loading modules from directory:
/usr/lib/dovecot/modules
doveadm(root): Debug: Module loaded:
/usr/lib/dovecot/modules/lib20_listescape_plugin.so
doveadm(root): Debug: Loading modules from directory:
/usr/lib/dovecot/modules/doveadm
doveadm(root): Debug: Skipping module doveadm_acl_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined
symbol: acl_user_module (this is usually intentional, so just ignore
this message)
doveadm(root): Debug: Skipping module doveadm_expire_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so:
undefined symbol: expire_set_init (this is usually intentional, so just
ignore this message)
doveadm(root): Debug: Skipping module doveadm_quota_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so:
undefined symbol: quota_user_module (this is usually intentional, so
just ignore this message)
doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_zlib_plugin.so: undefined
symbol: i_stream_create_deflate (this is usually intentional, so just
ignore this message)
doveadm(root): Debug: Skipping module doveadm_fts_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined
symbol: fts_list_backend (this is usually intentional, so just ignore
this message)
doveadm(yyyyy at example.com): Debug: Added userdb setting:
mail=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir
doveadm(yyyyy at example.com): Debug: Added userdb setting:
plugin/quota_rule=*:storage=5120M
doveadm(yyyyy at example.com): Debug: Effective uid=1001, gid=1001,
home=/stor/real/domains/example.com/yy/yyyyy
doveadm(yyyyy at example.com): Debug: Namespace : type=private,
prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no,
subscriptions=yes location=pop3c:
doveadm(yyyyy at example.com): Debug: fs: root=, index=, control=, inbox=,
altdoveadm(yyyyy at example.com): Debug: Namespace inbox: type=private,
prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes
location=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir
doveadm(yyyyy at example.com): Debug: maildir++:
root=/stor/real/domains/example.com/yy/yyyyy/Maildir, index=, control=,
inbox=/stor/real/domains/example.com/yy/yyyyy/Maildir, altdsync(yyyyy at
example.com): Debug: Effective uid=1001, gid=1001,
home=/stor/real/domains/example.com/yy/yyyyy
dsync(yyyyy at example.com): Debug: Namespace : type=private,
prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no,
subscriptions=yes location=pop3c:
dsync(yyyyy at example.com): Debug: fs: root=, index=, control=, inbox=,
altdsync(yyyyy at example.com): Debug: Namespace inbox: type=private, prefix=,
sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes
location=pop3c:/tmp/pop3c
dsync(yyyyy at example.com): Debug: fs: root=/tmp/pop3c/.,
index=/tmp/pop3c, control=, inbox=, altdsync(yyyyy at example.com): Debug:
Namespace : Using permissions from
/tmp/pop3c/.: mode=0755 gid=-1
dsync(yyyyy at example.com): Debug: Namespace : Using permissions from
/stor/real/domains/example.com/yy/yyyyy/Maildir: mode=0700 gid=-1
dsync(yyyyy at example.com): Debug: Namespace : /tmp/pop3c/./INBOX doesn't
exist yet, using default permissions
dsync(yyyyy at example.com): Debug: Namespace : Using permissions from
/tmp/pop3c/.: mode=0755 gid=-1
dsync(yyyyy at example.com): Debug: pop3c(120.121.122.123): Looking up IP
address
dsync(yyyyy at example.com): Debug: pop3c(120.121.122.123): Connecting to
120.121.122.123:110
dsync(yyyyy at example.com): Debug: pop3c(120.121.122.123): Authenticating
as xxxxxx at example.com
dsync(yyyyy at example.com): Panic: file mail-storage.c: line 1311
(mailbox_get_metadata): assertion failed: ((items &
MAILBOX_METADATA_GUID) == 0 || !guid_128_is_empty(metadata_r->guid))
dsync(yyyyy at example.com): Error: Raw backtrace:
/usr/lib/dovecot/libdovecot.so.0(+0x45caa) [0xb761ccaa] ->
/usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x47)
[0xb761cdc7] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0xb761d647]
-> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_get_metadata+0x120)
[0xb76f3040] -> doveadm() [0x80740bc] ->
doveadm(dsync_worker_mailbox_iter_next+0x21) [0x8071c11] -> doveadm()
[0x806c1f7] -> doveadm(dsync_brain_sync+0x6c0) [0x806b890] ->
doveadm(dsync_brain_sync_all+0x18) [0x806c4f8] -> doveadm() [0x806961b]
-> doveadm() [0x8069bbf] -> doveadm() [0x80541ec] ->
doveadm(doveadm_mail_try_run+0x257) [0x8054ee7] -> doveadm(main+0x391)
[0x805d0c1] ->
/lib/i386-linux-gnu/i686/cmov/libc.so.6(__libc_start_main+0xe6)
[0xb748ae46] -> doveadm() [0x8054001]
Aborted
Now I'm lost...
Am I in the right direction?
Thanks,
Gedalya
# dovecot -n
# 2.1.17: /etc/dovecot/dovecot.conf
# OS: Linux 3.2.0-4-686-pae i686 Debian 7.1
auth_default_realm = example.com
auth_mechanisms = plain login cram-md5
auth_verbose = yes
auth_verbose_passwords = plain
default_client_limit = 8096
dict {
expire = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext
quota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext
}
disable_plaintext_auth = no
imapc_host = mail01.example.com
mail_gid = vmail
mail_plugins = listescape
mail_prefetch_count = 50
mail_uid = vmail
namespace {
hidden = yes
list = no
location = pop3c:
prefix = POP3-MIGRATION-NS/
}
namespace inbox {
inbox = yes
location prefix separator = /
type = private
}
passdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
expire = Trash
expire2 = Trash.*
expire3 = Junk
expire_dict = proxy::expire
pop3_migration_mailbox = POP3-MIGRATION-NS/INBOX
sieve = ~/.dovecot.sieve
sieve_before = /var/local/lib/dovecot/sieve/global/fileinto-spam.sieve
sieve_dir = ~/sieve
}
pop3c_host = 120.121.122.123
postmaster_address = postmaster at example.com
service auth-worker {
user = $default_internal_user
}
service auth {
client_limit = 20480
}
service dict {
unix_listener dict {
mode = 0600
user = vmail
}
}
service imap-login {
process_min_avail = 4
service_count = 0
vsz_limit = 128 M
}
service imap {
process_limit = 10240
}
service lmtp {
inet_listener lmtp {
address = 0.0.0.0
port = 7025
}
}
service pop3-login {
process_min_avail = 4
service_count = 0
vsz_limit = 128 M
}
service pop3 {
process_limit = 4096
}
ssl_cert = </etc/ssl/certs/dovecot.pem
ssl_key = </etc/ssl/private/dovecot.pem
userdb {
driver = prefetch
}
userdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
verbose_proctitle = yes
protocol doveadm {
mail_plugins = listescape pop3_migration
}
protocol lmtp {
mail_plugins = listescape sieve
}
protocol imap {
mail_max_userip_connections = 25
mail_plugins = listescape imap_quota
ssl_cert = </etc/dovecot/imap.example.com.crt
ssl_key = </etc/dovecot/imap.example.com.key
}
protocol pop3 {
mail_max_userip_connections = 10
mail_plugins = listescape
pop3_fast_size_lookups = yes
pop3_no_flag_updates = yes
pop3_reuse_xuidl = no
pop3_save_uidl = yes
pop3_uidl_format = %08Xu%08Xv
ssl_cert = </etc/dovecot/pop.example.com.crt
ssl_key = </etc/dovecot/pop.example.com.key
}
/etc/dovecot/dovecot-sql.conf.ext :
driver = mysql
connect = .....
default_pass_scheme = PLAIN
password_query = SELECT password, \
'/stor/real/domains/%d/%2n/%n' AS userdb_home, \
'maildir:/stor/real/domains/%d/%2n/%n/Maildir' as userdb_mail, \
concat('*:storage=', quota_mb, 'M') as userdb_quota_rule, \
'vmail' AS userdb_uid, 'vmail' AS userdb_gid \
FROM email WHERE userid = '%n' AND domain = '%d' and
deleted=0
user_query = SELECT '/stor/real/domains/%d/%2n/%n' AS home, \
'maildir:/stor/real/domains/%d/%2n/%n/Maildir' as mail, \
concat('*:storage=', quota_mb, 'M') as quota_rule, \
'vmail' AS uid, 'vmail' AS gid \
FROM email WHERE userid = '%n' AND domain = '%d' and
deleted=0
iterate_query = SELECT userid as user, domain FROM email where deleted=0
OK so now I tried a doveadm backup -R with pop3c using dovecot 2.2.4
(50117c22151e) (auto build) and it works fine, and indeed very fast.
Funny, I ran into a segmentation fault by mistakenly passing
pop3c_password instead of imapc_password, thought it would be worth posting.
I'm using the cutting-edge dovecot over NFS storing to the production
server running 2.1.7. I'll be seeing "Corrupted index cache file ..
Invalid magic in hole header" when every account logs in for the first
time post-migration. Is this OK? I'm using Maildir.
Segfault:
Starting program: /usr/bin/doveadm -D -o imapc_user=xxxx at example.com -o
pop3c_password=******* -o imapc_host=mail01.example.com import -s -u
xxxx at example.com imapc: '' NOT mailbox INBOX
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
doveadm(root): Debug: Loading modules from directory:
/usr/lib/dovecot/modules
doveadm(root): Debug: Module loaded:
/usr/lib/dovecot/modules/lib20_listescape_plugin.so
doveadm(root): Debug: Loading modules from directory:
/usr/lib/dovecot/modules/doveadm
doveadm(root): Debug: Skipping module doveadm_acl_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined
symbol: acl_user_module (this is usually intentional, so just ignore
this message)
doveadm(root): Debug: Skipping module doveadm_expire_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so:
undefined symbol: expire_set_deinit (this is usually intentional, so
just ignore this message)
doveadm(root): Debug: Skipping module doveadm_quota_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so:
undefined symbol: quota_user_module (this is usually intentional, so
just ignore this message)
doveadm(root): Debug: Module loaded:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so
doveadm(root): Debug: Skipping module doveadm_fts_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined
symbol: fts_backend_rescan (this is usually intentional, so just ignore
this message)
doveadm(root): Debug: Effective uid=0, gid=0, home=/root
Program received signal SIGSEGV, Segmentation fault.
imapc_storage_client_unref (_client=_client at entry=0x80c3130) at
imapc-storage.c:256
256 imapc-storage.c: No such file or directory.
(gdb) bt full
#0 imapc_storage_client_unref (_client=_client at entry=0x80c3130) at
imapc-storage.c:256
client = 0x0
cb = <optimized out>
__FUNCTION__ = "imapc_storage_client_unref"
#1 0xb7ed4660 in imapc_list_deinit (_list=0x80c3028) at imapc-list.c:106
list = 0x80c3028
#2 0xb7ef5c44 in mailbox_list_create (driver=driver at entry=0xb7f57ad2
"imapc", ns=ns at entry=0x80c2f78, set=set at entry=0xbffff854,
flags=MAILBOX_LIST_FLAG_NO_MAIL_FILES,
list_r=list_r at entry=0xbffff844, error_r=error_r at entry=0xbffff928)
at mailbox-list.c:184
list = 0x80c3028
idx = 6
__FUNCTION__ = "mailbox_list_create"
#3 0xb7eedadb in mail_storage_create_full (ns=ns at entry=0x80c2f78,
driver=<optimized out>, driver at entry=0x0, data=<optimized out>,
flags=flags at entry=0, storage_r=storage_r at entry=0xbffff8cc,
error_r=error_r at entry=0xbffff928) at mail-storage.c:356
storage_class = 0xb7f8d060
storage = 0x0
list = 0xbffff880
list_set = {layout = 0xb7f57ad2 "imapc", root_dir = 0x0,
index_dir = 0x0, index_pvt_dir = 0x0, control_dir = 0x0, alt_dir = 0x0,
inbox_path = 0x0, subscription_fname = 0x0,
maildir_name = 0xb7f52fbc "", mailbox_dir_name = 0xb7f52fbc
"", escape_char = 37 '%', broken_char = 0 '\000', utf8
= false,
alt_dir_nocheck = false}
list_flags = <optimized out>
p = <optimized out>
__FUNCTION__ = "mail_storage_create_full"
#4 0xb7eede35 in mail_storage_create (ns=ns at entry=0x80c2f78,
driver=driver at entry=0x0, flags=flags at entry=0,
error_r=error_r at entry=0xbffff928) at mail-storage.c:407
storage = 0xb7eeddf0
#5 0xb7ee788f in mail_namespaces_init_location (user=0x80bef90,
location=location at entry=0x80b12ab "imapc:",
error_r=error_r at entry=0xbffff978) at mail-namespace.c:460
inbox_set = 0x80bfd50
unexpanded_inbox_set = 0x80bfd80
ns = 0x80c2f78
mail_set = 0x80bf638
error = <optimized out>
driver = <optimized out>
location_source = <optimized out>
default_location = <optimized out>
__FUNCTION__ = "mail_namespaces_init_location"
#6 0x0805b072 in cmd_import_init (_ctx=0x80b4ed8, args=0x80b1200) at
doveadm-mail-import.c:179
ctx = 0x80b4ed8
input = {module = 0x808e8e5 "mail", service = 0x0, username =
0x8092f03 "doveadm", session_id = 0x0, local_ip = {family = 0, u =
{ip6
= {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
__u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 =
{0, 0, 0, 0}}}, ip4 = {s_addr = 0}}}, remote_ip = {family = 0, u = {ip6
= {__in6_u = {
__u6_addr8 = '\000' <repeats 15 times>,
__u6_addr16 =
{0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, ip4 = {s_addr =
0}}}, local_port = 0, remote_port = 0,
userdb_fields = 0x0, flags_override_add = 2056,
flags_override_remove = MAIL_STORAGE_SERVICE_FLAG_USERDB_LOOKUP,
no_userdb_lookup = 0}
service_user = 0x80b8e90
user = 0x80bef90
---Type <return> to continue, or q <return> to quit---
src_location = 0x80b12ab "imapc:"
error = 0x4 <Address 0x4 out of bounds>
#7 0x080575f8 in doveadm_mail_single_user (ctx=0x80b4ed8,
input=0xbffffa6c, error_r=0xbffffa64) at doveadm-mail.c:328
__FUNCTION__ = "doveadm_mail_single_user"
#8 0x08057c37 in doveadm_mail_cmd (argv=0x80b1200, argc=<optimized
out>, cmd=0x80b4388) at doveadm-mail.c:516
input = {module = 0x0, service = 0x8092f03 "doveadm",
username
= 0x80b1299 "xxxx at example.com", session_id = 0x0, local_ip =
{family =
0, u = {ip6 = {__in6_u = {
__u6_addr8 = '\000' <repeats 15 times>,
__u6_addr16 =
{0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, ip4 = {s_addr =
0}}}, remote_ip = {family = 0, u = {ip6 = {
__in6_u = {__u6_addr8 = '\000' <repeats 15
times>,
__u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}},
ip4 = {s_addr = 0}}}, local_port = 0, remote_port = 0,
userdb_fields = 0x0, flags_override_add = 0,
flags_override_remove = 0, no_userdb_lookup = 0}
ctx = 0x80b4ed8
getopt_args = <optimized out>
ret = <optimized out>
c = <optimized out>
wildcard_user = 0x0
error = <optimized out>
#9 doveadm_mail_try_run (cmd_name=cmd_name at entry=0x80b128c
"import",
argc=argc at entry=9, argv=argv at entry=0x80b11f0) at doveadm-mail.c:608
cmd = 0x80b4388
#10 0x08056b26 in main (argc=9, argv=0x80b11f0) at doveadm.c:398
cmd_name = <optimized out>
i = <optimized out>
quick_init = false
c = <optimized out>