Ralf Becker
2021-Aug-09 08:41 UTC
Replication stalled by failed attemts to read attachments (mail_attachment_dir)
Made some progress, the attachments are not lost, the new Dovecot server (it was a migration from 2.2.19 to 2.3.15 on a different host), searches them under a partially different filename: Aug 09 08:26:19 doveadm: Error: dsync(61c8ab10dbe7): read(attachments-connector(/var/dovecot/imap/$domain/$user/mailboxes/INBOX/dbox-Mails/u.23306)) failed: read(/var/dovecot/imap/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-e8584c2cf5a10b61640000007dc04144-23306[base64:19 b/l]) failed: open(/var/dovecot/imap/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-e8584c2cf5a10b61640000007dc04144-23306) failed: No such file or directory (last sent=mail, last recv=mail_request (EOL)) Looking at the filesystem: root at ka-nfs-mail:~# ls -l /var/dovecot/imap/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-e8584c2cf5a10b61640000007dc04144-23306 ls: cannot access '/poolN/nordenham/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-e8584c2cf5a10b61640000007dc04144-23306': No such file or directory root at ka-nfs-mail:~# ls -l /var/dovecot/imap/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-*-23306 -rw------- 1 90 systemd-journal 605950 Jul 19 09:18 /var/dovecot/imap/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-578fed299419c150550c0000838cbfe1-23306 So the attachment still exists (verified by it content), but the 3rd dash-separated filename-part has changed: 1. configured (mail_attachment_hash) sha256 hash of the attachment-content 2. guid_128 (https://github.com/dovecot/core/blob/a5209c83c3a82386c94d466eec5fea394973e88f/src/lib-storage/index/index-attachment.c#L114) 3. no idea, but changed 4. uid of the mail So the questions are: - what is that 3rd part - how could it have changed by the migration - is there a way to force it back to the existing file-names Unfortunately I found no documentation but the source-code on Github about the inner working of that attachment store :( Any help or ideas would be really appreciated :) Ralf Am 07.08.21 um 19:47 schrieb Ralf Becker:> A separate mail-attachment store and replication seems to a very bad > idea! > > If anything goes wrong eg. full filesystem or an accidentality deleted > attachment-file, you wont get your replication up again :( > > Only way I found so far is greping the log for the error and use that > to expunge the whole mails, something like: > > docker logs -f dovecot 2>&1 | > grep 'Error: dsync(61c8ab10dbe7): read(attachments-connector' | > cut -d/ -f5,6,8,10 | cut -d ')' -f1 | sort | uniq | sed -e > 's#/\(u.\)\?# #g' | > while read domain user mailbox uid; do > ??? doveadm expunge -u $user@$domain mailbox $mailbox uid $uid > done > > (The above delete the mail from both nodes!) > > While having the above running on the node with the problem, you need > to trigger a full sync on the *other* node with: > > ??? doveadm replicator replicate -f $user@$domain > > until there are no more errors and > > ??? watch doveadm replicator status $user@$domain > > shows the mailbox in sync again. > > I'm happy to hear better suggestions, specially some more automatic > and - possibly - only removing the missing attachment and not the > whole mail. > > Ralf > > > Am 05.08.21 um 18:03 schrieb Ralf Becker: >> I'm migrating an older Dovecot 2.2.19 installation with dbox and >> mail_attachment_dir to 2.3.15 with replication to a second server. >> >> The storage from the old server was rsync'ed to a new server running >> Dovecot 2.3.15 in a container using similar configuration with dbox >> and mail_attachments. That servers alone seems to run fine. >> >> As a next step I tried to establish replication to the second empty >> server, which is configured to use mdbox and no mail_attachment_dir, >> as the rest of my Dovecot servers. Looking at "doveadm replicator >> status '*'" some of the mailboxes replicate correct, but roughly half >> of them fail :( >> >> Looking a the Dovecot log with mail_debug=yes shows the replication >> fails on the source (server with dbox and mail_attachment_dir) >> typically with one of two errors: >> >> a) attachment has different size: >> >> Aug 05 15:41:58 doveadm: Debug: Mailbox INBOX: UID 30: Looked up >> field date.received from mail cache >> Aug 05 15:41:58 doveadm: Error: dsync(cdaf50f7580e): >> read(attachments-connector(/var/dovecot/imap/<domain>/<user>/mailboxes/INBOX/dbox-Mails/u.30)) >> failed: >> read(/var/dovecot/imap/attachments/64/4e/644ecfbc09223b8ccafc6f36aa86603cde91e76716b9b5ed0cf875bc0e183b6b4581d671f8321647be99f26d2d22088e999714529e189b25a6e7d5b428fbf5d5-1a6b701e68c9c050b10a0000838cbfe1-f86a701e68c9c050b10a0000838cbfe1-30[base64:18 >> b/l]) failed: Stream is larger than expected (262658 > 262657, eof=1) >> (last sent=mail, last recv=mail_request (EOL)) >> >> showing on the other log as: >> >> Aug 05 15:49:18 doveadm: Debug: Mailbox INBOX: UID 29: Looked up >> field guid from mail cache >> Aug 05 15:49:19 doveadm: Error: dsync(142878d8d6bf): read() failed: >> read(10.44.99.1) failed: dot-input stream ends without '.' line (last >> sent=mail_request (EOL), last recv=mail) >> >> b) attachment is not found at all: >> >> Aug 05 15:43:58 doveadm(<user>@<domain> 1441): Debug: fs-sis: fs-sis: >> fs-posix: >> open(/var/dovecot/imap/attachments/e2/0a/e20ab165f3a2e7164520341cddb06cd053dc1a87fdd5c6360b45c20a2c9d33576c8ea2daa42062f9bb0fd94f9c5f9919ad86fb11e29effe3bdf4e853fa7c0e1d-20a0b63613e6c050620b0000838cbfe1-c8d5221c39a40b61250900007dc04144-3) >> failed: No such file or directory >> Aug 05 15:43:58 doveadm(<user>@<domain> 1441): Error: >> dsync(cdaf50f7580e): >> read(attachments-connector(/var/dovecot/imap/<domain>/<user>/mailboxes/INBOX/dbox-Mails/u.3)) >> failed: >> read(/var/dovecot/imap/attachments/e2/0a/e20ab165f3a2e7164520341cddb06cd053dc1a87fdd5c6360b45c20a2c9d33576c8ea2daa42062f9bb0fd94f9c5f9919ad86fb11e29effe3bdf4e853fa7c0e1d-20a0b63613e6c050620b0000838cbfe1-c8d5221c39a40b61250900007dc04144-3[base64:19 >> b/l]) failed: >> open(/var/dovecot/imap/attachments/e2/0a/e20ab165f3a2e7164520341cddb06cd053dc1a87fdd5c6360b45c20a2c9d33576c8ea2daa42062f9bb0fd94f9c5f9919ad86fb11e29effe3bdf4e853fa7c0e1d-20a0b63613e6c050620b0000838cbfe1-c8d5221c39a40b61250900007dc04144-3) >> failed: No such file or directory (last sent=mail, last >> recv=mail_request (EOL)) >> >> That caused the replication to fail and not newer mails to be >> replicated for the given user and mailbox, which can be verified eg. >> with "doveadm mailbox status -u <user>@<domain> all inbox". >> >> The concerned mails are pretty old / the attachment itself can not be >> recovered, and nothing can be done about them. But how can I repair >> the mailbox, so all unaffected mails / attachments get replicated? >> >> I already tried multiple "doveadm force-resync -u <user>@<domain> >> INBOX" in case a) with no visible effect / still same error. >> >> Both Dovecot configurations are attached, in case they matter ... >> >> Ralf >> >-- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Gesch?ftsf?hrer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0
Timo Sirainen
2021-Aug-09 09:08 UTC
Replication stalled by failed attemts to read attachments (mail_attachment_dir)
On 9. Aug 2021, at 10.41, Ralf Becker <rb at egroupware.org> wrote:> > Made some progress, the attachments are not lost, the new Dovecot server (it was a migration from 2.2.19 to 2.3.15 on a different host), searches them under a partially different filename: > > Aug 09 08:26:19 doveadm: Error: dsync(61c8ab10dbe7): read(attachments-connector(/var/dovecot/imap/$domain/$user/mailboxes/INBOX/dbox-Mails/u.23306)) failed: read(/var/dovecot/imap/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-e8584c2cf5a10b61640000007dc04144-23306[base64:19 b/l]) failed: open(/var/dovecot/imap/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-e8584c2cf5a10b61640000007dc04144-23306) failed: No such file or directory (last sent=mail, last recv=mail_request (EOL))..> So the questions are: > - what is that 3rd partThat's the mailbox GUID. dsync and replication is supposed to preserved them. You could check them with: doveadm mailbox status -u user guid '*'> - how could it have changed by the migration > - is there a way to force it back to the existing file-namesLooks like they have changed. You could change the GUIDs afterwards also: doveadm mailbox update -u user --guid <newguid> mailboxname BTW. The GUID can also be used to see its creation timestamp: 578fed299419c150550c0000838cbfe1 = Fri 7 Dec 00:17:56 EET 2012 e8584c2cf5a10b61640000007dc04144 = Thu 5 Aug 11:31:49 EEST 2021 Using guid2date.sh: #!/bin/bash guid=$1 hex=`printf $guid|cut -c 9-16|sed 's/\(..\)\(..\)\(..\)\(..\)/\4\3\2\1/'` dec=`printf "%d" 0x$hex` time=`date -d "1970-01-01 UTC $dec seconds"` printf "$guid\nhex: $hex\ndec: $dec\ntime: $time\n"