On Tue, 2020-03-31 at 09:06 +0300, Aki Tuomi wrote:> > > > On 30/03/2020 22:11 Ben Mulvihill <ben.mulvihill at gmail.com> wrote: > > > > ? > > I am trying to backup a gmail account (not the one I am writing > > from) > > to dovecot, using doveadm-backup and imapc, but am having ssl > > connection problems. > > > > ted at expectation:~# doveadm backup -D -R -u ted imapc: > > dsync(ted): Info: imapc(imap.gmail.com:993): Connected to > > 74.125.71.108:993 (local 10.7.1.179:53852) > > dsync(ted): Warning: imapc(imap.gmail.com:993): Server disconnected > > unexpectedly: SSL_connect() failed: error:14094410:SSL > > routines:ssl3_read_bytes:sslv3 alert handshake failure: SSL alert > > number 40 - reconnecting (delay 0 ms) > > dsync(ted): Info: imapc(imap.gmail.com:993): Connected to > > 74.125.71.109:993 (local 10.7.1.179:59052) > > dsync(ted): Error: imapc(imap.gmail.com:993): Server disconnected > > unexpectedly: SSL_connect() failed: error:14094410:SSL > > routines:ssl3_read_bytes:sslv3 alert handshake failure: SSL alert > > number 40 - disconnecting > > dsync(ted): Error: User initialization failed: imapc: Login to > > imap.gmail.com failed: Disconnected from serv > > > > > > I am using dovecot version 2.2.33.2 on ubuntu, with the > > configuration below. > > I have also enabled "allow access from unsecure apps" in my > > gmail settings. > > > > My first thought looking at the error messages was has that > > perhaps doveadm-backup was trying to connect with ssl3, which > > is no longer supported by gmail or anyone else nowadays. > > But apparently the ssl3_read_bytes function in openssl also > > handles tls, so the reference to ssl3 in the message is > > misleading, and the real problem must be elsewhere. > > > > If anyone can help me debug this I'd be grateful. > > > > Many thanks,? > > Ben > > > > root at expectation:/etc/dovecot# doveconf -N > > # 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf > Hi! > > This is very old version of dovecot so this could be a bug that has > been fixed in more recent version. > > Can you verify that you have the required CA certs with > > openssl s_client -connect imap.gmail.com:993 -servername > imap.gmail.com -CApath /etc/ssl/certs > > and make sure the cert gets validated by openssl. > > If it does, then you should probably consider upgrading to some more > recent version. We provide packages at https://repo.dovecot.org if > you are able to upgrade. > > AkiHi Aki! I checked the CA certificate and it was validated OK, so I followed your suggestion and installed dovecot version 2.3.10 from the dovecot repository (the version I was using previously was the latest available in my distribution). The ssl handshake now completes successfully. I have hit? further problems later in the process, but I'll investigate myself first and then maybe post again with the details if I am still stuck. Thank you for your help! Ben
Hello again,
I am still stuck I'm afraid.
I now have doveadm backup working perfectly from
a small gmail mailbox (a few hundred messages), but
when I try the same configuration (apart from usernames
and passwords obviously) with a large gmail mailbox
(around 60,000 messages), doveadm connects successfully,
replicates the gmail folder tree, then produces no further
output until about 30 minutes later when the server times
out and doveadm crashes. No actual messages are downloaded.?
Does doveadm backup try by default to download all
messages in a single transaction? And if so, is there
a way to split it up?
Command output and configuration are at the end of this
message. Dovecot version is 2.3.10.
Thanks in advance for any help.
Ben
--------------------
ben at expectation:~$ doveadm backup -D -R -u ben imapc:
dsync(ben): Info: imapc(imap.gmail.com:993): Connected to
74.125.133.109:993 (local MY_IP_ADDRESS:53244)
dsync(ben): Warning: imapc(imap.gmail.com:993): Server disconnected
with message: System Error - reconnecting (delay 0 ms)
dsync(ben): Info: imapc(imap.gmail.com:993): Connected to
74.125.133.108:993 (local?MY_IP_ADDRESS:34076)
dsync(ben): Panic: file imapc-client.c: line 173
(imapc_client_run_pre): assertion failed: (client->ioloop == NULL)
dsync(ben): Error: Raw backtrace:
/usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x42)
[0x7f94271610f2] ->
/usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7f942716120e]
-> /usr/lib/dovecot/libdovecot.so.0(+0xe6aee) [0x7f942716baee] ->
/usr/lib/dovecot/libdovecot.so.0(+0xe6b31) [0x7f942716bb31] ->
/usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f94270c61bb] ->
/usr/lib/dovecot/libdovecot-storage.so.0(imapc_client_run+0x158)
[0x7f94274c27e8] -> /usr/lib/dovecot/libdovecot-
storage.so.0(imapc_simple_run+0x3c) [0x7f94274bf8ac] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0x94a23) [0x7f94274b7a23] ->
/usr/lib/dovecot/libdovecot-
storage.so.0(imapc_list_get_mailbox_flags+0x68) [0x7f94274b8288] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0x98e69) [0x7f94274bbe69] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0x98fed) [0x7f94274bbfed] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0x9bcc9) [0x7f94274becc9] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0xa27cb) [0x7f94274c57cb] ->
/usr/lib/dovecot/libdovecot-
storage.so.0(imapc_connection_input_pending+0x227) [0x7f94274c7dc7] ->
/usr/lib/dovecot/libdovecot-storage.so.0(+0xa4e1a) [0x7f94274c7e1a] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69) [0x7f9427184999]
-> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x134)
[0x7f9427186274] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x4c)
[0x7f9427184a9c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38)
[0x7f9427184cc8] -> /usr/lib/dovecot/libdovecot-
storage.so.0(imapc_client_run+0xbb) [0x7f94274c274b] ->
/usr/lib/dovecot/libdovecot-
storage.so.0(imapc_mailbox_run_nofetch+0x20) [0x7f94274bf7a0] ->
/usr/lib/dovecot/libdovecot-storage.so.0(imapc_mail_fetch+0x199)
[0x7f94274b9419] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x9772f)
[0x7f94274ba72f] -> /usr/lib/dovecot/libdovecot-
storage.so.0(mail_get_stream_because+0x64) [0x7f9427461fa4] ->
/usr/lib/dovecot/libdovecot-
storage.so.0(index_mail_update_access_parts_post+0x7a) [0x7f94274e6c2a]
-> /usr/lib/dovecot/libdovecot-storage.so.0(+0xcaf3f) [0x7f94274edf3f]
-> /usr/lib/dovecot/libdovecot-
storage.so.0(index_storage_search_next_nonblock+0x10d) [0x7f94274ee1cd]
-> /usr/lib/dovecot/libdovecot-
storage.so.0(mailbox_search_next_nonblock+0x22) [0x7f94274706e2] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next+0x3d)
[0x7f942747074d] ->
/usr/bin/doveadm(dsync_mailbox_export_next_mail+0x57) [0x562a19bf0597]
Aborted (core dumped)
------------------------
# 2.3.10 (0da0eff44): /etc/dovecot/dovecot.conf
# OS: Linux 4.15.0-91-generic x86_64 Ubuntu 18.04.4 LTS
# Hostname: expectation
imapc_features = rfc822.size gmail-migration fetch-headers
imapc_host = imap.gmail.com
imapc_password = # hidden, use -P to show it
imapc_port = 993
imapc_ssl = imaps
imapc_ssl_verify = no
imapc_user = MY_GMAIL_ADDRESS
mail_location = maildir:~/Maildir
mail_prefetch_count = 20
namespace inbox {
? inbox = yes
? location ? mailbox Drafts {
????special_use = \Drafts
? }
? mailbox Junk {
????special_use = \Junk
? }
? mailbox Sent {
????special_use = \Sent
? }
? mailbox "Sent Messages" {
????special_use = \Sent
? }
? mailbox Trash {
????special_use = \Trash
? }
? prefix ? separator = .
}
passdb {
? driver = pam
}
protocols = " imap"
service stats {
? unix_listener stats-reader {
????group = mail
????mode = 0660
????user = dovecot
? }
? unix_listener stats-writer {
????group = mail
????mode = 0660
????user = dovecot
? }
}
ssl = no
ssl_client_ca_dir = /etc/ssl/certs
userdb {
? driver = passwd
}
On Wed, 1 Apr 2020 at 10:32, Ben Mulvihill <ben.mulvihill at gmail.com> wrote:> > Hello again, > > I am still stuck I'm afraid. > > I now have doveadm backup working perfectly from > a small gmail mailbox (a few hundred messages), but > when I try the same configuration (apart from usernames > and passwords obviously) with a large gmail mailbox > (around 60,000 messages), doveadm connects successfully, > replicates the gmail folder tree, then produces no further > output until about 30 minutes later when the server times > out and doveadm crashes. No actual messages are downloaded. > > Does doveadm backup try by default to download all > messages in a single transaction? And if so, is there > a way to split it up? > > Command output and configuration are at the end of this > message. Dovecot version is 2.3.10. > > Thanks in advance for any help. > > Ben > > -------------------- > > ben at expectation:~$ doveadm backup -D -R -u ben imapc: > dsync(ben): Info: imapc(imap.gmail.com:993): Connected to > 74.125.133.109:993 (local MY_IP_ADDRESS:53244) > dsync(ben): Warning: imapc(imap.gmail.com:993): Server disconnected > with message: System Error - reconnecting (delay 0 ms) > dsync(ben): Info: imapc(imap.gmail.com:993): Connected to > 74.125.133.108:993 (local MY_IP_ADDRESS:34076) > dsync(ben): Panic: file imapc-client.c: line 173 > (imapc_client_run_pre): assertion failed: (client->ioloop == NULL) > dsync(ben): Error: Raw backtrace: > /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x42) > [0x7f94271610f2] -> > /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7f942716120e] > -> /usr/lib/dovecot/libdovecot.so.0(+0xe6aee) [0x7f942716baee] -> > /usr/lib/dovecot/libdovecot.so.0(+0xe6b31) [0x7f942716bb31] -> > /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f94270c61bb] -> > /usr/lib/dovecot/libdovecot-storage.so.0(imapc_client_run+0x158) > [0x7f94274c27e8] -> /usr/lib/dovecot/libdovecot- > storage.so.0(imapc_simple_run+0x3c) [0x7f94274bf8ac] -> > /usr/lib/dovecot/libdovecot-storage.so.0(+0x94a23) [0x7f94274b7a23] -> > /usr/lib/dovecot/libdovecot- > storage.so.0(imapc_list_get_mailbox_flags+0x68) [0x7f94274b8288] -> > /usr/lib/dovecot/libdovecot-storage.so.0(+0x98e69) [0x7f94274bbe69] -> > /usr/lib/dovecot/libdovecot-storage.so.0(+0x98fed) [0x7f94274bbfed] -> > /usr/lib/dovecot/libdovecot-storage.so.0(+0x9bcc9) [0x7f94274becc9] -> > /usr/lib/dovecot/libdovecot-storage.so.0(+0xa27cb) [0x7f94274c57cb] -> > /usr/lib/dovecot/libdovecot- > storage.so.0(imapc_connection_input_pending+0x227) [0x7f94274c7dc7] -> > /usr/lib/dovecot/libdovecot-storage.so.0(+0xa4e1a) [0x7f94274c7e1a] -> > /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69) [0x7f9427184999] > -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x134) > [0x7f9427186274] -> > /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x4c) > [0x7f9427184a9c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) > [0x7f9427184cc8] -> /usr/lib/dovecot/libdovecot- > storage.so.0(imapc_client_run+0xbb) [0x7f94274c274b] -> > /usr/lib/dovecot/libdovecot- > storage.so.0(imapc_mailbox_run_nofetch+0x20) [0x7f94274bf7a0] -> > /usr/lib/dovecot/libdovecot-storage.so.0(imapc_mail_fetch+0x199) > [0x7f94274b9419] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x9772f) > [0x7f94274ba72f] -> /usr/lib/dovecot/libdovecot- > storage.so.0(mail_get_stream_because+0x64) [0x7f9427461fa4] -> > /usr/lib/dovecot/libdovecot- > storage.so.0(index_mail_update_access_parts_post+0x7a) [0x7f94274e6c2a] > -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xcaf3f) [0x7f94274edf3f] > -> /usr/lib/dovecot/libdovecot- > storage.so.0(index_storage_search_next_nonblock+0x10d) [0x7f94274ee1cd] > -> /usr/lib/dovecot/libdovecot- > storage.so.0(mailbox_search_next_nonblock+0x22) [0x7f94274706e2] -> > /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next+0x3d) > [0x7f942747074d] -> > /usr/bin/doveadm(dsync_mailbox_export_next_mail+0x57) [0x562a19bf0597] > Aborted (core dumped) > > ------------------------ > > # 2.3.10 (0da0eff44): /etc/dovecot/dovecot.conf > # OS: Linux 4.15.0-91-generic x86_64 Ubuntu 18.04.4 LTS > # Hostname: expectation > imapc_features = rfc822.size gmail-migration fetch-headers > imapc_host = imap.gmail.com > imapc_password = # hidden, use -P to show it > imapc_port = 993 > imapc_ssl = imaps > imapc_ssl_verify = no > imapc_user = MY_GMAIL_ADDRESS > mail_location = maildir:~/Maildir > mail_prefetch_count = 20 > namespace inbox { > inbox = yes > location > mailbox Drafts { > special_use = \Drafts > } > mailbox Junk { > special_use = \Junk > } > mailbox Sent { > special_use = \Sent > } > mailbox "Sent Messages" { > special_use = \Sent > } > mailbox Trash { > special_use = \Trash > } > prefix > separator = . > } > passdb { > driver = pam > } > protocols = " imap" > service stats { > unix_listener stats-reader { > group = mail > mode = 0660 > user = dovecot > } > unix_listener stats-writer { > group = mail > mode = 0660 > user = dovecot > } > } > ssl = no > ssl_client_ca_dir = /etc/ssl/certs > userdb { > driver = passwd > } >Hi, Wouldn't a software like imapsync be a better tool here? Rgds/DP
> On 31 Mar 2020, at 23.18, Ben Mulvihill <ben.mulvihill at gmail.com> wrote: > > Hello again, > > I am still stuck I'm afraid. > > I now have doveadm backup working perfectly from > a small gmail mailbox (a few hundred messages), but > when I try the same configuration (apart from usernames > and passwords obviously) with a large gmail mailbox > (around 60,000 messages), doveadm connects successfully, > replicates the gmail folder tree, then produces no further > output until about 30 minutes later when the server times > out and doveadm crashes. No actual messages are downloaded. >It's a while since we last performed migrations from gmail. Maybe they have changed something in the meanwhile. Can you record imapc rawlogs from the session to see where it gets stuck? It might be gmail throttling the speed. Sami