On Thu, 2020-04-09 at 13:48 +0300, Sami Ketola wrote:>
> >
> > On 31 Mar 2020, at 23.18, Ben Mulvihill <ben.mulvihill at
gmail.com>
> > wrote:
> >
> > Hello again,
> >
> > I am still stuck I'm afraid.
> >
> > I now have doveadm backup working perfectly from
> > a small gmail mailbox (a few hundred messages), but
> > when I try the same configuration (apart from usernames
> > and passwords obviously) with a large gmail mailbox
> > (around 60,000 messages), doveadm connects successfully,
> > replicates the gmail folder tree, then produces no further
> > output until about 30 minutes later when the server times
> > out and doveadm crashes. No actual messages are downloaded.?
> >
> It's a while since we last performed migrations from gmail. Maybe
> they have changed something in the meanwhile.
> Can you record imapc rawlogs from the session to see where it gets
> stuck? It might be gmail throttling the speed.
>
> Sami
>
Thanks for the reply. Here are my logs, edited to remove personal
information and lots of repeated lines. Dovecot configuration is?
as above, with the addition of the two options
? imapc_rawlog_dir
? imapc_features = throttle:50:16000:500
Doveadm seems to be fetching lots of?
message IDs but no actual messages, and then it core dumps.
Ben
============================================================
edited-imapc-log.in
1586512882.754448 * OK Gimap ready for requests from XXX.XXX.XXX.XXX
t129mb20216081wmt
1586512882.790345 * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA
ID XLIST CHILDREN X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH2 AUTH=PLAIN
AUTH=PLAIN-CLIENTTOKEN AUTH=OAUTHBEARER AUTH=XOAUTH
1586512882.790345 1 OK Thats all she wrote! t129mb20216081wmt
1586512883.870337 * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA
ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE
CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL-
SPECIAL-USE APPENDLIMIT=35651584
1586512883.870337 2 OK MY_GMAIL_ADDRESS authenticated (Success)
1586512884.014073 * LIST (\Noselect) "/" "/"
1586512884.014073 3 OK Success
1586512884.160944 * LIST (\HasNoChildren) "/" "INBOX"
1586512884.160944 * LIST (\HasChildren \Noselect) "/" "[Google
Mail]"
1586512884.160944 * LIST (\All \HasNoChildren) "/" "[Google
Mail]/All
Mail"
1586512884.160944 * LIST (\HasNoChildren \Trash) "/" "[Google
Mail]/Bin"
1586512884.160944 * LIST (\Drafts \HasNoChildren) "/" "[Google
Mail]/Drafts"
1586512884.160944 * LIST (\HasNoChildren \Important) "/" "[Google
Mail]/Important"
1586512884.160944 * LIST (\HasNoChildren) "/" "[Google
Mail]/MY_FOLDER_NAME"
1586512884.160944 * LIST (\HasNoChildren \Sent) "/" "[Google
Mail]/Sent
Mail"
1586512884.160944 * LIST (\HasNoChildren \Junk) "/" "[Google
Mail]/Spam"
1586512884.160944 * LIST (\Flagged \HasNoChildren) "/" "[Google
Mail]/Starred"
1586512884.161789 * LIST (\HasNoChildren) "/" "[Google
Mail]/mark_as_spam"
1586512884.161789 * LIST (\HasNoChildren) "/" "[Google
Mail]/mining"
1586512884.161789 4 OK Success
1586512884.317802 * STATUS "INBOX" (UIDNEXT 110434 UIDVALIDITY
602828375)
1586512884.317802 5 OK Success
1586512884.462612 * STATUS "[Google Mail]/All Mail" (UIDNEXT 231431
UIDVALIDITY 602828395)
1586512884.462612 6 OK Success
1586512884.606828 * STATUS "[Google Mail]/Bin" (UIDNEXT 21532
UIDVALIDITY 602828376)
1586512884.606828 7 OK Success
1586512884.749952 * STATUS "[Google Mail]/Drafts" (UIDNEXT 17485
UIDVALIDITY 602828378)
1586512884.749952 8 OK Success
1586512884.894350 * STATUS "[Google Mail]/Important" (UIDNEXT 29288
UIDVALIDITY 602828400)
1586512884.894350 9 OK Success
1586512885.037461 * STATUS "[Google Mail]/MY_FOLDER_NAME" (UIDNEXT
3648
UIDVALIDITY 602828399)
1586512885.037461 10 OK Success
1586512885.181415 * STATUS "[Google Mail]/Sent Mail" (UIDNEXT 14061
UIDVALIDITY 602828374)
1586512885.181415 11 OK Success
1586512885.325056 * STATUS "[Google Mail]/Spam" (UIDNEXT 98883
UIDVALIDITY 602828379)
1586512885.325056 12 OK Success
1586512885.468928 * STATUS "[Google Mail]/Starred" (UIDNEXT 21
UIDVALIDITY 602828380)
1586512885.468928 13 OK Success
1586512885.612611 * STATUS "[Google Mail]/mark_as_spam" (UIDNEXT 76
UIDVALIDITY 602828397)
1586512885.612611 14 OK Success
1586512885.755909 * STATUS "[Google Mail]/mining" (UIDNEXT 23
UIDVALIDITY 602828396)
1586512885.755909 15 OK Success
1586512885.899075 * LSUB (\HasNoChildren) "/" "INBOX"
1586512885.899075 * LSUB (\HasChildren \Noselect) "/" "[Google
Mail]"
1586512885.899075 * LSUB (\All \HasNoChildren) "/" "[Google
Mail]/All
Mail"
1586512885.900047 * LSUB (\HasNoChildren \Trash) "/" "[Google
Mail]/Bin"
1586512885.900047 * LSUB (\Drafts \HasNoChildren) "/" "[Google
Mail]/Drafts"
1586512885.900047 * LSUB (\HasNoChildren \Important) "/" "[Google
Mail]/Important"
1586512885.900047 * LSUB (\HasNoChildren) "/" "[Google
Mail]/MY_FOLDER_NAME"
1586512885.900047 * LSUB (\HasNoChildren \Sent) "/" "[Google
Mail]/Sent
Mail"
1586512885.901074 * LSUB (\HasNoChildren \Junk) "/" "[Google
Mail]/Spam"
1586512885.901074 * LSUB (\Flagged \HasNoChildren) "/" "[Google
Mail]/Starred"
1586512885.901074 * LSUB (\HasNoChildren) "/" "[Google
Mail]/mark_as_spam"
1586512885.901074 * LSUB (\HasNoChildren) "/" "[Google
Mail]/mining"
1586512885.901074 16 OK Success
1586512886.990868 * LIST (\HasNoChildren) "/" "INBOX"
1586512886.990868 * LIST (\HasChildren \Noselect) "/" "[Google
Mail]"
1586512886.990868 * LIST (\All \HasNoChildren) "/" "[Google
Mail]/All
Mail"
1586512886.990868 * LIST (\HasNoChildren \Trash) "/" "[Google
Mail]/Bin"
1586512886.990868 * LIST (\Drafts \HasNoChildren) "/" "[Google
Mail]/Drafts"
1586512886.990868 * LIST (\HasNoChildren \Important) "/" "[Google
Mail]/Important"
1586512886.990868 * LIST (\HasNoChildren) "/" "[Google
Mail]/MY_FOLDER_NAME"
1586512886.990868 * LIST (\HasNoChildren \Sent) "/" "[Google
Mail]/Sent
Mail"
1586512886.990868 * LIST (\HasNoChildren \Junk) "/" "[Google
Mail]/Spam"
1586512886.990868 * LIST (\Flagged \HasNoChildren) "/" "[Google
Mail]/Starred"
1586512886.991783 * LIST (\HasNoChildren) "/" "[Google
Mail]/mark_as_spam"
1586512886.991783 * LIST (\HasNoChildren) "/" "[Google
Mail]/mining"
1586512886.991783 17 OK Success
1586512887.436407 * FLAGS (\Answered \Flagged \Draft \Deleted \Seen
$NotPhishing $Phishing $has_cal JUNK receipt-handled)
1586512887.436407 * OK [PERMANENTFLAGS ()] Flags permitted.
1586512887.436407 * OK [UIDVALIDITY 602828375] UIDs valid.
1586512887.436407 * 48540 EXISTS
1586512887.436407 * 0 RECENT
1586512887.436407 * OK [UIDNEXT 110434] Predicted next UID.
1586512887.436407 * OK [HIGHESTMODSEQ 8740446]
1586512887.436407 18 OK [READ-ONLY] INBOX selected. (Success)
1586512891.905633 * 1 FETCH (UID 62 FLAGS (\Seen))
1586512891.905633 * 2 FETCH (UID 65 FLAGS (\Seen))
1586512891.905633 * 3 FETCH (UID 69 FLAGS (\Answered \Seen))
1586512891.905633 * 4 FETCH (UID 70 FLAGS (\Answered \Seen))
1586512891.905633 * 5 FETCH (UID 71 FLAGS (\Answered \Seen))
1586512891.905633 * 6 FETCH (UID 72 FLAGS (\Answered \Seen))
1586512891.905633 * 7 FETCH (UID 73 FLAGS (\Seen))
1586512891.905633 * 8 FETCH (UID 76 FLAGS (\Seen))
1586512891.905633 * 9 FETCH (UID 78 FLAGS (\Answered \Seen))
1586512891.905633 * 10 FETCH (UID 79 FLAGS (\Seen))
1586512891.905633 * 11 FETCH (UID 80 FLAGS (\Answered \Seen))
1586512891.905633 * 12 FETCH (UID 81 FLAGS (\Seen))
1586512891.905633 * 13 FETCH (UID 83 FLAGS (\Answered \Seen))
1586512891.907319 * 14 FETCH (UID 84 FLAGS ())
1586512891.907319 * 15 FETCH (UID 85 FLAGS (\Seen))
1586512891.907319 * 16 FETCH (UID 86 FLAGS (\Seen))
1586512891.907319 * 17 FETCH (UID 87 FLAGS (\Answered \Seen))
1586512891.907319 * 18 FETCH (UID 88 FLAGS ())
1586512891.907319 * 19 FETCH (UID 89 FLAGS (\Seen))
1586512891.907319 * 20 FETCH (UID 91 FLAGS (\Answered \Seen))
1586512891.907319 * 21 FETCH (UID 92 FLAGS (\Seen))
....... LOTS OF SIMILAR LINES .......
1586512893.541503 * 48522 FETCH (UID 110415 FLAGS ())
1586512893.541503 * 48523 FETCH (UID 110416 FLAGS ())
1586512893.541503 * 48524 FETCH (UID 110417 FLAGS ())
1586512893.541503 * 48525 FETCH (UID 110418 FLAGS ())
1586512893.541503 * 48526 FETCH (UID 110419 FLAGS ())
1586512893.541503 * 48527 FETCH (UID 110420 FLAGS ())
1586512893.541503 * 48528 FETCH (UID 110421 FLAGS ())
1586512893.541503 * 48529 FETCH (UID 110422 FLAGS ())
1586512893.541503 * 48530 FETCH (UID 110423 FLAGS ())
1586512893.541503 * 48531 FETCH (UID 110424 FLAGS ())
1586512893.541503 * 48532 FETCH (UID 110425 FLAGS ())
1586512893.541503 * 48533 FETCH (UID 110426 FLAGS ())
1586512893.541503 * 48534 FETCH (UID 110427 FLAGS ())
1586512893.541503 * 48535 FETCH (UID 110428 FLAGS ())
1586512893.541503 * 48536 FETCH (UID 110429 FLAGS ())
1586512893.541503 * 48537 FETCH (UID 110430 FLAGS ())
1586512893.541503 * 48538 FETCH (UID 110431 FLAGS ())
1586512893.541503 * 48539 FETCH (UID 110432 FLAGS ())
1586512893.541503 * 48540 FETCH (UID 110433 FLAGS ())
1586512893.541503 19 OK Success
1586512893.704625 * 1 FETCH (X-GM-MSGID 1265813037243496020 UID 62)
1586512893.704625 * 2 FETCH (X-GM-MSGID 1266142060943449934 UID 65)
1586512893.704625 * 3 FETCH (X-GM-MSGID 1266240767076216106 UID 69)
1586512893.704625 * 4 FETCH (X-GM-MSGID 1266317645039054507 UID 70)
1586512893.704625 * 5 FETCH (X-GM-MSGID 1266319104567433759 UID 71)
1586512893.704625 * 6 FETCH (X-GM-MSGID 1266338339735148225 UID 72)
1586512893.704625 * 7 FETCH (X-GM-MSGID 1266350811947568709 UID 73)
1586512893.704625 * 8 FETCH (X-GM-MSGID 1266427483536331721 UID 76)
1586512893.704625 * 9 FETCH (X-GM-MSGID 1266482766386716417 UID 78)
1586512893.704625 * 10 FETCH (X-GM-MSGID 1266512748587747731 UID 79)
1586512893.704625 * 11 FETCH (X-GM-MSGID 1266523171112424759 UID 80)
1586512893.704625 * 12 FETCH (X-GM-MSGID 1266584914541703703 UID 81)
1586512893.704625 * 13 FETCH (X-GM-MSGID 1266626775786784521 UID 83)
1586512893.704625 * 14 FETCH (X-GM-MSGID 1266663913156455749 UID 84)
1586512893.704625 * 15 FETCH (X-GM-MSGID 1266684556515340123 UID 85)
1586512893.704625 * 16 FETCH (X-GM-MSGID 1266684780091647650 UID 86)
1586512893.704625 * 17 FETCH (X-GM-MSGID 1266690080881874411 UID 87)
1586512893.704625 * 18 FETCH (X-GM-MSGID 1266697947934426922 UID 88)
1586512893.704625 * 19 FETCH (X-GM-MSGID 1266841785830673314 UID 89)
1586512893.704625 * 20 FETCH (X-GM-MSGID 1266890981732444471 UID 91)
1586512893.704625 * 21 FETCH (X-GM-MSGID 1266891244711722087 UID 92)
1586512893.704625 20 OK Success
1586512893.861241 * 22 FETCH (X-GM-MSGID 1266959738134105978 UID 94)
1586512893.861241 * 23 FETCH (X-GM-MSGID 1267022593809794009 UID 95)
1586512893.861241 * 24 FETCH (X-GM-MSGID 1267023552222412653 UID 96)
1586512893.861241 * 25 FETCH (X-GM-MSGID 1267028400626761068 UID 97)
1586512893.861241 * 26 FETCH (X-GM-MSGID 1267058789660419022 UID 98)
1586512893.861241 * 27 FETCH (X-GM-MSGID 1267060956576095193 UID 99)
1586512893.861241 * 28 FETCH (X-GM-MSGID 1267078074477602342 UID 100)
1586512893.861241 * 29 FETCH (X-GM-MSGID 1267144516612296284 UID 102)
1586512893.861241 * 30 FETCH (X-GM-MSGID 1267169233934891212 UID 103)
1586512893.861241 * 31 FETCH (X-GM-MSGID 1267181621874937138 UID 104)
1586512893.861241 * 32 FETCH (X-GM-MSGID 1267322905811934828 UID 106)
1586512893.861241 * 33 FETCH (X-GM-MSGID 1267362788296163573 UID 107)
1586512893.861241 * 34 FETCH (X-GM-MSGID 1267409013248654879 UID 110)
1586512893.861241 * 35 FETCH (X-GM-MSGID 1267429601732986948 UID 111)
1586512893.861241 * 36 FETCH (X-GM-MSGID 1267433450902833783 UID 112)
1586512893.861241 * 37 FETCH (X-GM-MSGID 1267433996158684970 UID 113)
1586512893.861241 * 38 FETCH (X-GM-MSGID 1267479985482540073 UID 114)
1586512893.861241 * 39 FETCH (X-GM-MSGID 1267512244483485892 UID 117)
1586512893.861241 * 40 FETCH (X-GM-MSGID 1267512434336402096 UID 118)
1586512893.861241 * 41 FETCH (X-GM-MSGID 1267514033433119045 UID 119)
1586512893.861241 * 42 FETCH (X-GM-MSGID 1267586382946441314 UID 120)
1586512893.861241 21 OK Success
1586512894.009036 * 43 FETCH (X-GM-MSGID 1267586414720205278 UID 121)
1586512894.009036 * 44 FETCH (X-GM-MSGID 1267594567233703154 UID 123)
1586512894.009036 * 45 FETCH (X-GM-MSGID 1267861214156448451 UID 127)
....... LOTS OF SIMILAR LINES .......
1586513176.097992 * 40414 FETCH (X-GM-MSGID 1642763475047250557 UID
99636)
1586513176.097992 * 40415 FETCH (X-GM-MSGID 1642764349836495198 UID
99637)
1586513176.097992 * 40416 FETCH (X-GM-MSGID 1642767015801735363 UID
99639)
1586513176.097992 * 40417 FETCH (X-GM-MSGID 1642768317624659015 UID
99640)
1586513176.097992 * 40418 FETCH (X-GM-MSGID 1642769644449001118 UID
99641)
1586513176.097992 * 40419 FETCH (X-GM-MSGID 1642769732523676610 UID
99642)
1586513176.097992 * 40420 FETCH (X-GM-MSGID 1642770568707862020 UID
99643)
1586513176.097992 * 40421 FETCH (X-GM-MSGID 1642771085461609988 UID
99644)
1586513176.097992 * 40422 FETCH (X-GM-MSGID 1642773510009904699 UID
99645)
1586513176.097992 * 40423 FETCH (X-GM-MSGID 1642773705798532019 UID
99646)
1586513176.097992 * 40424 FETCH (X-GM-MSGID 1642774528944320238 UID
99647)
1586513176.097992 * 40425 FETCH (X-GM-MSGID 1642801024168852473 UID
99648)
1586513176.126970 * 48541 EXISTS
1586513176.126970 1944 OK Success
============================================================
edited-imapc-log.out
1586512882.754618 1 CAPABILITY
1586512882.790509 2 LOGIN "MY_GMAIL_ADDRESS"
"MY_GMAIL_PASSWORD"
1586512883.870605 3 LIST "" ""
1586512884.014301 4 LIST "" "*"
1586512884.162105 5 STATUS "INBOX" (UIDNEXT UIDVALIDITY)
1586512884.318081 6 STATUS "[Google Mail]/All Mail" (UIDNEXT
UIDVALIDITY)
1586512884.462890 7 STATUS "[Google Mail]/Bin" (UIDNEXT UIDVALIDITY)
1586512884.607101 8 STATUS "[Google Mail]/Drafts" (UIDNEXT
UIDVALIDITY)
1586512884.750230 9 STATUS "[Google Mail]/Important" (UIDNEXT
UIDVALIDITY)
1586512884.894624 10 STATUS "[Google Mail]/MY_FOLDER_NAME" (UIDNEXT
UIDVALIDITY)
1586512885.037737 11 STATUS "[Google Mail]/Sent Mail" (UIDNEXT
UIDVALIDITY)
1586512885.181692 12 STATUS "[Google Mail]/Spam" (UIDNEXT UIDVALIDITY)
1586512885.325389 13 STATUS "[Google Mail]/Starred" (UIDNEXT
UIDVALIDITY)
1586512885.469204 14 STATUS "[Google Mail]/mark_as_spam" (UIDNEXT
UIDVALIDITY)
1586512885.612928 15 STATUS "[Google Mail]/mining" (UIDNEXT
UIDVALIDITY)
1586512885.756217 16 LSUB "" "*"
1586512886.839842 17 LIST "" "*"
1586512886.992192 18 EXAMINE "INBOX"
1586512887.436620 19 UID FETCH 1:* (FLAGS)
1586512893.560147 20 UID FETCH
62,65,69,70,71,72,73,76,78,79,80,81,83,84,85,86,87,88,89,91,92 (X-GM-
MSGID)
1586512893.717661 21 UID FETCH
94,95,96,97,98,99,100,102,103,104,106,107,110,111,112,113,114,117,118,1
19,120 (X-GM-MSGID)
1586512893.861887 22 UID FETCH
121,123,127,128,129,131,132,134,136,137,139,140,141,142,143,144,145,147
,148,149,150 (X-GM-MSGID)
1586512894.009702 23 UID FETCH
151,153,156,160,161,162,346,347,348,349,352,354,355,357,359,361,362,363
,364,365,366 (X-GM-MSGID)
1586512894.159770 24 UID FETCH
375,376,377,379,380,382,383,384,385,392,393,394,397,400,401,402,403,404
,406,407,411 (X-GM-MSGID)
1586512894.305890 25 UID FETCH
412,413,415,417,418,419,425,426,427,429,430,433,434,435,436,437,438,439
,442,443,444 (X-GM-MSGID)
1586512894.450197 26 UID FETCH
447,448,449,454,455,457,459,460,462,465,468,469,470,471,472,473,479,480
,481,482,483 (X-GM-MSGID)
1586512894.595853 27 UID FETCH
485,486,488,489,490,491,492,494,498,499,501,502,505,507,508,510,511,512
,513,514,515 (X-GM-MSGID)
1586512894.741263 28 UID FETCH
517,518,519,520,522,523,524,525,526,532,533,534,535,537,538,540,542,543
,544,545,548 (X-GM-MSGID)
?....... LOTS OF SIMILAR LINES ......
1586513175.222346 1939 UID FETCH
99497,99498,99500,99501,99502,99503,99506,99508,99510,99511,99512,99515
,99516,99517,99518,99519,99520,99521,99522,99523,99525 (X-GM-MSGID)
1586513175.368035 1940 UID FETCH
99526,99527,99528,99529,99530,99531,99532,99533,99534,99535,99536,99537
,99538,99539,99540,99541,99542,99543,99544,99546,99547 (X-GM-MSGID)
1586513175.513463 1941 UID FETCH
99548,99550,99551,99553,99554,99555,99556,99557,99558,99559,99560,99561
,99562,99563,99564,99565,99566,99567,99568,99569,99570 (X-GM-MSGID)
1586513175.658881 1942 UID FETCH
99571,99572,99573,99575,99576,99577,99578,99579,99580,99581,99582,99583
,99584,99585,99586,99587,99588,99589,99590,99591,99592 (X-GM-MSGID)
1586513175.804616 1943 UID FETCH
99596,99598,99599,99601,99602,99603,99605,99606,99607,99608,99609,99611
,99612,99613,99615,99616,99618,99619,99620,99621,99623 (X-GM-MSGID)
1586513175.951965 1944 UID FETCH
99624,99627,99628,99629,99631,99632,99633,99634,99635,99636,99637,99639
,99640,99641,99642,99643,99644,99645,99646,99647,99648 (X-GM-MSGID)
1586513176.127109 1945 LIST "" "*"