Ralf Becker
2021-Aug-07 17:47 UTC
Replication stalled by failed attemts to read attachments (mail_attachment_dir)
A separate mail-attachment store and replication seems to a very bad idea! If anything goes wrong eg. full filesystem or an accidentality deleted attachment-file, you wont get your replication up again :( Only way I found so far is greping the log for the error and use that to expunge the whole mails, something like: docker logs -f dovecot 2>&1 | grep 'Error: dsync(61c8ab10dbe7): read(attachments-connector' | cut -d/ -f5,6,8,10 | cut -d ')' -f1 | sort | uniq | sed -e 's#/\(u.\)\?# #g' | while read domain user mailbox uid; do ??? doveadm expunge -u $user@$domain mailbox $mailbox uid $uid done (The above delete the mail from both nodes!) While having the above running on the node with the problem, you need to trigger a full sync on the *other* node with: ??? doveadm replicator replicate -f $user@$domain until there are no more errors and ??? watch doveadm replicator status $user@$domain shows the mailbox in sync again. I'm happy to hear better suggestions, specially some more automatic and - possibly - only removing the missing attachment and not the whole mail. Ralf Am 05.08.21 um 18:03 schrieb Ralf Becker:> I'm migrating an older Dovecot 2.2.19 installation with dbox and > mail_attachment_dir to 2.3.15 with replication to a second server. > > The storage from the old server was rsync'ed to a new server running > Dovecot 2.3.15 in a container using similar configuration with dbox > and mail_attachments. That servers alone seems to run fine. > > As a next step I tried to establish replication to the second empty > server, which is configured to use mdbox and no mail_attachment_dir, > as the rest of my Dovecot servers. Looking at "doveadm replicator > status '*'" some of the mailboxes replicate correct, but roughly half > of them fail :( > > Looking a the Dovecot log with mail_debug=yes shows the replication > fails on the source (server with dbox and mail_attachment_dir) > typically with one of two errors: > > a) attachment has different size: > > Aug 05 15:41:58 doveadm: Debug: Mailbox INBOX: UID 30: Looked up field > date.received from mail cache > Aug 05 15:41:58 doveadm: Error: dsync(cdaf50f7580e): > read(attachments-connector(/var/dovecot/imap/<domain>/<user>/mailboxes/INBOX/dbox-Mails/u.30)) > failed: > read(/var/dovecot/imap/attachments/64/4e/644ecfbc09223b8ccafc6f36aa86603cde91e76716b9b5ed0cf875bc0e183b6b4581d671f8321647be99f26d2d22088e999714529e189b25a6e7d5b428fbf5d5-1a6b701e68c9c050b10a0000838cbfe1-f86a701e68c9c050b10a0000838cbfe1-30[base64:18 > b/l]) failed: Stream is larger than expected (262658 > 262657, eof=1) > (last sent=mail, last recv=mail_request (EOL)) > > showing on the other log as: > > Aug 05 15:49:18 doveadm: Debug: Mailbox INBOX: UID 29: Looked up field > guid from mail cache > Aug 05 15:49:19 doveadm: Error: dsync(142878d8d6bf): read() failed: > read(10.44.99.1) failed: dot-input stream ends without '.' line (last > sent=mail_request (EOL), last recv=mail) > > b) attachment is not found at all: > > Aug 05 15:43:58 doveadm(<user>@<domain> 1441): Debug: fs-sis: fs-sis: > fs-posix: > open(/var/dovecot/imap/attachments/e2/0a/e20ab165f3a2e7164520341cddb06cd053dc1a87fdd5c6360b45c20a2c9d33576c8ea2daa42062f9bb0fd94f9c5f9919ad86fb11e29effe3bdf4e853fa7c0e1d-20a0b63613e6c050620b0000838cbfe1-c8d5221c39a40b61250900007dc04144-3) > failed: No such file or directory > Aug 05 15:43:58 doveadm(<user>@<domain> 1441): Error: > dsync(cdaf50f7580e): > read(attachments-connector(/var/dovecot/imap/<domain>/<user>/mailboxes/INBOX/dbox-Mails/u.3)) > failed: > read(/var/dovecot/imap/attachments/e2/0a/e20ab165f3a2e7164520341cddb06cd053dc1a87fdd5c6360b45c20a2c9d33576c8ea2daa42062f9bb0fd94f9c5f9919ad86fb11e29effe3bdf4e853fa7c0e1d-20a0b63613e6c050620b0000838cbfe1-c8d5221c39a40b61250900007dc04144-3[base64:19 > b/l]) failed: > open(/var/dovecot/imap/attachments/e2/0a/e20ab165f3a2e7164520341cddb06cd053dc1a87fdd5c6360b45c20a2c9d33576c8ea2daa42062f9bb0fd94f9c5f9919ad86fb11e29effe3bdf4e853fa7c0e1d-20a0b63613e6c050620b0000838cbfe1-c8d5221c39a40b61250900007dc04144-3) > failed: No such file or directory (last sent=mail, last > recv=mail_request (EOL)) > > That caused the replication to fail and not newer mails to be > replicated for the given user and mailbox, which can be verified eg. > with "doveadm mailbox status -u <user>@<domain> all inbox". > > The concerned mails are pretty old / the attachment itself can not be > recovered, and nothing can be done about them. But how can I repair > the mailbox, so all unaffected mails / attachments get replicated? > > I already tried multiple "doveadm force-resync -u <user>@<domain> > INBOX" in case a) with no visible effect / still same error. > > Both Dovecot configurations are attached, in case they matter ... > > Ralf >-- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Gesch?ftsf?hrer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0
Ralf Becker
2021-Aug-09 08:41 UTC
Replication stalled by failed attemts to read attachments (mail_attachment_dir)
Made some progress, the attachments are not lost, the new Dovecot server (it was a migration from 2.2.19 to 2.3.15 on a different host), searches them under a partially different filename: Aug 09 08:26:19 doveadm: Error: dsync(61c8ab10dbe7): read(attachments-connector(/var/dovecot/imap/$domain/$user/mailboxes/INBOX/dbox-Mails/u.23306)) failed: read(/var/dovecot/imap/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-e8584c2cf5a10b61640000007dc04144-23306[base64:19 b/l]) failed: open(/var/dovecot/imap/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-e8584c2cf5a10b61640000007dc04144-23306) failed: No such file or directory (last sent=mail, last recv=mail_request (EOL)) Looking at the filesystem: root at ka-nfs-mail:~# ls -l /var/dovecot/imap/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-e8584c2cf5a10b61640000007dc04144-23306 ls: cannot access '/poolN/nordenham/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-e8584c2cf5a10b61640000007dc04144-23306': No such file or directory root at ka-nfs-mail:~# ls -l /var/dovecot/imap/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-*-23306 -rw------- 1 90 systemd-journal 605950 Jul 19 09:18 /var/dovecot/imap/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-578fed299419c150550c0000838cbfe1-23306 So the attachment still exists (verified by it content), but the 3rd dash-separated filename-part has changed: 1. configured (mail_attachment_hash) sha256 hash of the attachment-content 2. guid_128 (https://github.com/dovecot/core/blob/a5209c83c3a82386c94d466eec5fea394973e88f/src/lib-storage/index/index-attachment.c#L114) 3. no idea, but changed 4. uid of the mail So the questions are: - what is that 3rd part - how could it have changed by the migration - is there a way to force it back to the existing file-names Unfortunately I found no documentation but the source-code on Github about the inner working of that attachment store :( Any help or ideas would be really appreciated :) Ralf Am 07.08.21 um 19:47 schrieb Ralf Becker:> A separate mail-attachment store and replication seems to a very bad > idea! > > If anything goes wrong eg. full filesystem or an accidentality deleted > attachment-file, you wont get your replication up again :( > > Only way I found so far is greping the log for the error and use that > to expunge the whole mails, something like: > > docker logs -f dovecot 2>&1 | > grep 'Error: dsync(61c8ab10dbe7): read(attachments-connector' | > cut -d/ -f5,6,8,10 | cut -d ')' -f1 | sort | uniq | sed -e > 's#/\(u.\)\?# #g' | > while read domain user mailbox uid; do > ??? doveadm expunge -u $user@$domain mailbox $mailbox uid $uid > done > > (The above delete the mail from both nodes!) > > While having the above running on the node with the problem, you need > to trigger a full sync on the *other* node with: > > ??? doveadm replicator replicate -f $user@$domain > > until there are no more errors and > > ??? watch doveadm replicator status $user@$domain > > shows the mailbox in sync again. > > I'm happy to hear better suggestions, specially some more automatic > and - possibly - only removing the missing attachment and not the > whole mail. > > Ralf > > > Am 05.08.21 um 18:03 schrieb Ralf Becker: >> I'm migrating an older Dovecot 2.2.19 installation with dbox and >> mail_attachment_dir to 2.3.15 with replication to a second server. >> >> The storage from the old server was rsync'ed to a new server running >> Dovecot 2.3.15 in a container using similar configuration with dbox >> and mail_attachments. That servers alone seems to run fine. >> >> As a next step I tried to establish replication to the second empty >> server, which is configured to use mdbox and no mail_attachment_dir, >> as the rest of my Dovecot servers. Looking at "doveadm replicator >> status '*'" some of the mailboxes replicate correct, but roughly half >> of them fail :( >> >> Looking a the Dovecot log with mail_debug=yes shows the replication >> fails on the source (server with dbox and mail_attachment_dir) >> typically with one of two errors: >> >> a) attachment has different size: >> >> Aug 05 15:41:58 doveadm: Debug: Mailbox INBOX: UID 30: Looked up >> field date.received from mail cache >> Aug 05 15:41:58 doveadm: Error: dsync(cdaf50f7580e): >> read(attachments-connector(/var/dovecot/imap/<domain>/<user>/mailboxes/INBOX/dbox-Mails/u.30)) >> failed: >> read(/var/dovecot/imap/attachments/64/4e/644ecfbc09223b8ccafc6f36aa86603cde91e76716b9b5ed0cf875bc0e183b6b4581d671f8321647be99f26d2d22088e999714529e189b25a6e7d5b428fbf5d5-1a6b701e68c9c050b10a0000838cbfe1-f86a701e68c9c050b10a0000838cbfe1-30[base64:18 >> b/l]) failed: Stream is larger than expected (262658 > 262657, eof=1) >> (last sent=mail, last recv=mail_request (EOL)) >> >> showing on the other log as: >> >> Aug 05 15:49:18 doveadm: Debug: Mailbox INBOX: UID 29: Looked up >> field guid from mail cache >> Aug 05 15:49:19 doveadm: Error: dsync(142878d8d6bf): read() failed: >> read(10.44.99.1) failed: dot-input stream ends without '.' line (last >> sent=mail_request (EOL), last recv=mail) >> >> b) attachment is not found at all: >> >> Aug 05 15:43:58 doveadm(<user>@<domain> 1441): Debug: fs-sis: fs-sis: >> fs-posix: >> open(/var/dovecot/imap/attachments/e2/0a/e20ab165f3a2e7164520341cddb06cd053dc1a87fdd5c6360b45c20a2c9d33576c8ea2daa42062f9bb0fd94f9c5f9919ad86fb11e29effe3bdf4e853fa7c0e1d-20a0b63613e6c050620b0000838cbfe1-c8d5221c39a40b61250900007dc04144-3) >> failed: No such file or directory >> Aug 05 15:43:58 doveadm(<user>@<domain> 1441): Error: >> dsync(cdaf50f7580e): >> read(attachments-connector(/var/dovecot/imap/<domain>/<user>/mailboxes/INBOX/dbox-Mails/u.3)) >> failed: >> read(/var/dovecot/imap/attachments/e2/0a/e20ab165f3a2e7164520341cddb06cd053dc1a87fdd5c6360b45c20a2c9d33576c8ea2daa42062f9bb0fd94f9c5f9919ad86fb11e29effe3bdf4e853fa7c0e1d-20a0b63613e6c050620b0000838cbfe1-c8d5221c39a40b61250900007dc04144-3[base64:19 >> b/l]) failed: >> open(/var/dovecot/imap/attachments/e2/0a/e20ab165f3a2e7164520341cddb06cd053dc1a87fdd5c6360b45c20a2c9d33576c8ea2daa42062f9bb0fd94f9c5f9919ad86fb11e29effe3bdf4e853fa7c0e1d-20a0b63613e6c050620b0000838cbfe1-c8d5221c39a40b61250900007dc04144-3) >> failed: No such file or directory (last sent=mail, last >> recv=mail_request (EOL)) >> >> That caused the replication to fail and not newer mails to be >> replicated for the given user and mailbox, which can be verified eg. >> with "doveadm mailbox status -u <user>@<domain> all inbox". >> >> The concerned mails are pretty old / the attachment itself can not be >> recovered, and nothing can be done about them. But how can I repair >> the mailbox, so all unaffected mails / attachments get replicated? >> >> I already tried multiple "doveadm force-resync -u <user>@<domain> >> INBOX" in case a) with no visible effect / still same error. >> >> Both Dovecot configurations are attached, in case they matter ... >> >> Ralf >> >-- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Gesch?ftsf?hrer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0