Hi, I'm trying to use dsync to convert sdbox + sis mailboxes to mbox (mbox is chosen here to "re-attach" the attachments to their original place) # dsync -Dv -u $LOGIN -o "mail_location=sdbox:/zfspool/clone_srv_mail/$LOGIN" -o "mail_attachment_dir=/zfspool/clone_srv_attachments" backup "mbox:/zfspool/restore/$LOGIN/mbox:DIRNAME=mBoX-MeSsAgEs:INDEX=/zfspool/restore/$LOGIN/indexes:CONTROL=/zfspool/restore/$LOGIN/control" For 5 users out of a sample of 24, here is what's happening: dsync(user1): Error: read(/zfspool/clone_srv_attachments/cb/0a/cb0aad465a4ff95bf6fa6ece0fba94b43e8892cf-19dc51309fc2f3527e31000044c1c7e7-b55eb9176ca1b350e565000094b229f9-30810[base64:19 b/l]) failed: Stream is larger than expected (163244 > 163243, eof=1) dsync(user1): Error: copy: i_stream_read() failed: Invalid argument dsync(user1): Panic: file mail-index-transaction-update.c: line 19 (mail_index_transaction_lookup): assertion failed: (seq >t->first_new_seq && seq <= t->last_new_seq) dsync(user1): Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x6889a) [0x7f58a95a189a] -> /usr/local/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x32) [0x7f58a95a19a2] -> /usr/local/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f58a955b1cf] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0xc0287) [0x7f58a98ca287] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0xc3145) [0x7f58a98cd145] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mail_cache_decision_state_update+0xb6) [0x7f58a98bcb06] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mail_cache_lookup_headers+0x91) [0x7f58a98be5e1] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0xa0ac3) [0x7f58a98aaac3] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(index_mail_get_first_header+0x4a) [0x7f58a98ab04a] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0x9c021) [0x7f58a98a6021] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0x9c151) [0x7f58a98a6151] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(index_mail_close+0xf5) [0x7f58a98a6295] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mailbox_save_cancel+0x48) [0x7f58a98867c8] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mail_storage_copy+0x92) [0x7f58a9880e32] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mailbox_copy+0x5f) [0x7f58a9886c2f] -> dsync() [0x42f750] -> dsync(dsync_brain_sync_mails+0x459) [0x42e9c9] -> dsync(dsync_brain_run+0x2a1) [0x42ac51] -> dsync() [0x42876f] -> dsync() [0x411c97] -> dsync(doveadm_mail_try_run+0x238) [0x4120a8] -> dsync(main+0x3d1) [0x41aaa1] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f58a91c3d1d] -> dsync() [0x411419] dsync(user2): Error: read(/zfspool/clone_srv_attachments/ad/0c/ad0cef35cc6f0b2dae2197c4ff2b61a2bd58070d-9e8345192ccbf352c210000044c1c7e7-6efa5f2e522db350ed3d000094b229f9-15470[base64:18 b/l]) failed: Stream is larger than expected (194476 > 194475, eof=1) dsync(user2): Error: copy: i_stream_read() failed: Invalid argument dsync(user2): Panic: file mail-index-transaction-update.c: line 19 (mail_index_transaction_lookup): assertion failed: (seq >t->first_new_seq && seq <= t->last_new_seq) dsync(user2): Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x6889a) [0x7f2e2248d89a] -> /usr/local/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x32) [0x7f2e2248d9a2] -> /usr/local/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f2e224471cf] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0xc0287) [0x7f2e227b6287] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0xc3145) [0x7f2e227b9145] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mail_cache_decision_state_update+0xb6) [0x7f2e227a8b06] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mail_cache_lookup_headers+0x91) [0x7f2e227aa5e1] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0xa0ac3) [0x7f2e22796ac3] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(index_mail_get_first_header+0x4a) [0x7f2e2279704a] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0x9c021) [0x7f2e22792021] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0x9c151) [0x7f2e22792151] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(index_mail_close+0xf5) [0x7f2e22792295] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mailbox_save_cancel+0x48) [0x7f2e227727c8] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mail_storage_copy+0x92) [0x7f2e2276ce32] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mailbox_copy+0x5f) [0x7f2e22772c2f] -> dsync() [0x42f750] -> dsync(dsync_brain_sync_mails+0x459) [0x42e9c9] -> dsync(dsync_brain_run+0x2a1) [0x42ac51] -> dsync() [0x42876f] -> dsync() [0x411c97] -> dsync(doveadm_mail_try_run+0x238) [0x4120a8] -> dsync(main+0x3d1) [0x41aaa1] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f2e220afd1d] -> dsync() [0x411419] dsync(user3): Error: read(/zfspool/clone_srv_attachments/23/8a/238a781b53bb4d1b1bee989a5ff38636b616d5c5-41ba47152912f4522c6f000044c1c7e7-f3b06c2f5aa1b350d565000094b229f9-38650[base64:19 b/l]) failed: Stream is larger than expected (33191 > 33190, eof=1) dsync(user3): Error: copy: i_stream_read() failed: Invalid argument dsync(user3): Panic: file mail-index-transaction-update.c: line 19 (mail_index_transaction_lookup): assertion failed: (seq >t->first_new_seq && seq <= t->last_new_seq) dsync(user3): Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x6889a) [0x7f88d4bde89a] -> /usr/local/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x32) [0x7f88d4bde9a2] -> /usr/local/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f88d4b981cf] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0xc0287) [0x7f88d4f07287] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0xc3145) [0x7f88d4f0a145] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mail_cache_decision_state_update+0xb6) [0x7f88d4ef9b06] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mail_cache_lookup_headers+0x91) [0x7f88d4efb5e1] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0xa0ac3) [0x7f88d4ee7ac3] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(index_mail_get_first_header+0x4a) [0x7f88d4ee804a] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0x9c021) [0x7f88d4ee3021] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0x9c151) [0x7f88d4ee3151] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(index_mail_close+0xf5) [0x7f88d4ee3295] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mailbox_save_cancel+0x48) [0x7f88d4ec37c8] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mail_storage_copy+0x92) [0x7f88d4ebde32] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mailbox_copy+0x5f) [0x7f88d4ec3c2f] -> dsync() [0x42f750] -> dsync(dsync_brain_sync_mails+0x459) [0x42e9c9] -> dsync(dsync_brain_run+0x2a1) [0x42ac51] -> dsync() [0x42876f] -> dsync() [0x411c97] -> dsync(doveadm_mail_try_run+0x238) [0x4120a8] -> dsync(main+0x3d1) [0x41aaa1] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f88d4800d1d] -> dsync() [0x411419] dsync(user4): Error: read(/zfspool/clone_srv_attachments/11/fa/11fa090f48879e8514a07040a837b5544364e091-d5dd321af54ff452790f000044c1c7e7-98da7910e38db350b75a000094b229f9-50167[base64:19 b/l]) failed: Stream is larger than expected (56522 > 56521, eof=1) dsync(user4): Error: copy: i_stream_read() failed: Invalid argument dsync(user4): Panic: file mail-index-transaction-update.c: line 19 (mail_index_transaction_lookup): assertion failed: (seq >t->first_new_seq && seq <= t->last_new_seq) dsync(user4): Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x6889a) [0x7f416564a89a] -> /usr/local/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x32) [0x7f416564a9a2] -> /usr/local/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f41656041cf] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0xc0287) [0x7f4165973287] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0xc3145) [0x7f4165976145] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mail_cache_decision_state_update+0xb6) [0x7f4165965b06] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mail_cache_lookup_headers+0x91) [0x7f41659675e1] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0xa0ac3) [0x7f4165953ac3] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(index_mail_get_first_header+0x4a) [0x7f416595404a] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0x9c021) [0x7f416594f021] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0x9c151) [0x7f416594f151] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(index_mail_close+0xf5) [0x7f416594f295] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mailbox_save_cancel+0x48) [0x7f416592f7c8] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mail_storage_copy+0x92) [0x7f4165929e32] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mailbox_copy+0x5f) [0x7f416592fc2f] -> dsync() [0x42f750] -> dsync(dsync_brain_sync_mails+0x459) [0x42e9c9] -> dsync(dsync_brain_run+0x2a1) [0x42ac51] -> dsync() [0x42876f] -> dsync() [0x411c97] -> dsync(doveadm_mail_try_run+0x238) [0x4120a8] -> dsync(main+0x3d1) [0x41aaa1] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f416526cd1d] -> dsync() [0x411419] dsync(user5): Error: read(/zfspool/clone_srv_attachments/77/aa/77aa90d73107cc35fd0305c22a84eb00e793c935-a0d2ef302a20f552de5a000044c1c7e7-2accb7218264b350a423000094b229f9-11713[base64:19 b/l]) failed: Stream is larger than expected (43663 > 43662, eof=1) dsync(user5): Error: copy: i_stream_read() failed: Invalid argument dsync(user5): Panic: file mail-index-transaction-update.c: line 19 (mail_index_transaction_lookup): assertion failed: (seq >t->first_new_seq && seq <= t->last_new_seq) dsync(user5): Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x6889a) [0x7f2ab375489a] -> /usr/local/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x32) [0x7f2ab37549a2] -> /usr/local/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f2ab370e1cf] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0xc0287) [0x7f2ab3a7d287] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0xc3145) [0x7f2ab3a80145] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mail_cache_decision_state_update+0xb6) [0x7f2ab3a6fb06] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mail_cache_lookup_headers+0x91) [0x7f2ab3a715e1] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0xa0ac3) [0x7f2ab3a5dac3] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(index_mail_get_first_header+0x4a) [0x7f2ab3a5e04a] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0x9c021) [0x7f2ab3a59021] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(+0x9c151) [0x7f2ab3a59151] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(index_mail_close+0xf5) [0x7f2ab3a59295] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mailbox_save_cancel+0x48) [0x7f2ab3a397c8] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mail_storage_copy+0x92) [0x7f2ab3a33e32] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mailbox_copy+0x5f) [0x7f2ab3a39c2f] -> dsync() [0x42f750] -> dsync(dsync_brain_sync_mails+0x459) [0x42e9c9] -> dsync(dsync_brain_run+0x2a1) [0x42ac51] -> dsync() [0x42876f] -> dsync() [0x411c97] -> dsync(doveadm_mail_try_run+0x238) [0x4120a8] -> dsync(main+0x3d1) [0x41aaa1] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f2ab3376d1d] -> dsync() [0x411419] Regards, Gilles.
Hi Timo, I've made some further research on this issue (Dovecot was upgraded to the latest release in the meantime but, unsurprisingly, to no avail) and here's what I've found so far. On 09/02/2014 10:42, Gilles Chauvin wrote:> dsync(user2): Error: > read(/zfspool/clone_srv_attachments/ad/0c/ad0cef35cc6f0b2dae2197c4ff2b61a2bd58070d-9e8345192ccbf352c210000044c1c7e7-6efa5f2e522db350ed3d000094b229f9-15470[base64:18 > b/l]) failed: Stream is larger than expected (194476 > 194475, eof=1) > dsync(user2): Error: copy: i_stream_read() failed: Invalid argument > dsync(user2): Panic: file mail-index-transaction-update.c: line 19 > (mail_index_transaction_lookup): assertion failed: (seq >> t->first_new_seq && seq <= t->last_new_seq)The original mail got an attachment which is base64 encoded on 72 cols. The last 3 lines are: MAAxADMAIAAyADAAOgAwADEAOgA1ADQADQAKAGwAJwB1AHQAaQBsAGkAcwBhAHQAZQB1AHIA IABkAGUAIABsAG8AZwBpAG4AOgAgAGsAZQBsAGUAbQBhAHIAaQAgAGEAIADpAHQA6QAgAGMA cgDpAOkAIABsAGUAIAAyADEALwAwADMALwAyADAAMQAzACAAMgAwADoAMAAyADoAMAA0AA0ACgA For no good reason, the last line lacks a CR before the final "CgA=" part. I guess this is where Dovecot yells about the "stream larger than expected" because when it reencodes the attachment, it does it correctly by adding a proper CR before "CgA=" hence the one byte difference (tested using the "base64" command line tool). During my tests, each time dsync failed with this particular error, the same pattern applied (malformed base64 last line). Looks like a pretty hard problem to solve but, for now, it prevents us from restoring a mailbox. Regards, Gilles