dovecot - Dec 2018 - Errors with missing links to files when using Single Instance Storage and zlib (Dovecot 2.3.4)

Having had a week with the system I'm now inclined to believe that the 
issues I've seen are not necessarily related to SIS and zlib but may be 
due to replication instead.

After experiencing these issues and looking at historical mailing list posts I
wrote a basic script (as was suggested as a possible diagnostic approach for SIS
before SIS was even implemented!) which pulled all the mail from a user

doveadm fetch -u username body all > /dev/null 2>error.log

This generated errors in error.log for every e-mail with a problem.  From that
and some use of sed,uniq,cut and ln I was able to (re)create the missing links. 
Unfortunately the error generated was only the first missing link for all the
links expected for an e-mail.  So my script had to be run multiple times per
user until no more errors were reported.  This wasn't all that onerous given
I was basically running on a test box before mass migrating my users.  A better
approach would have been to scan the mdbox files for all links expected and
check the existence of them all.

Having (re)created all missing links I've not had a single instance of a
link being missing after a week of delivery of mail (where the errors were
regular before this procedure).

I have also since looked at how replication handles links.  What I hadn't
expected was that the links created on the server pair are not the same. 
Although I had two servers which started in the same state (one a clone of the
other) after a month or so of replication they had diverged to the state where
if I sent a test message to the one server I could see the file name of the link
created to the has was not the same on both servers (although obviously the hash
itself was the same name and content wise).  This leads me to believe the errors
I saw where most likely down to my expecting the two servers in the pair to have
identical SIS attachment directories/hardlinks to hashes and the way I handled
my migration.  Coming from a database mail server were replication master/slave
means the backend should be absolutely identical if all is working I may simply
have gotten the wrong idea of dovecot replication.

With my 2.3.1 and 2.3.4 replication pair there were a lot of the tcp errors
which the changelog for Dovecot stated had been fixed post 2.3.1; mainly the
connection timing out after 10 minutes.

Looking at the code for replication (which I'm not going to claim I've
fully go to grips with) it looks like the name of a hardlink is made up of
essentially [hash]-[identifier including mailbox GUID].  It must be the case
then that the [identifier including mailbox GUID] can diverge between server
pairs, at least with 2.3.1 as a version involved.







  


  Errors with missing links to files when using Single Instance Storage
  and zlib (Dovecot 2.3.4)

*Daniel Sch?tze*daniel at 
cwa.uk.com<mailto:dovecot%40dovecot.org?Subject=Re:%20Re%3A%20Errors%20with%20missing%20links%20to%20files%20when%20using%20Single%20Instance%20Storage%0A%20and%20zlib%20%28Dovecot%202.3.4%29&In-Reply-To=%3Ccc3b8bf3-055f-6908-95a0-d337544a8998%40cwa.uk.com%3E>
/Tue Dec 18 15:39:41 EET 2018/

  * Previous message:Change default mode for attachment files
    <https://www.dovecot.org/pipermail/dovecot/2018-December/113952.html>
  * Next message:High Load average on NFS Spool - v.2.1.15 & 2.2.13
    <https://www.dovecot.org/pipermail/dovecot/2018-December/113957.html>
  * *Messages sorted by:*[ date ]
   
<https://www.dovecot.org/pipermail/dovecot/2018-December/date.html#113954>[
    thread ]
   
<https://www.dovecot.org/pipermail/dovecot/2018-December/thread.html#113954>[
    subject ]
   
<https://www.dovecot.org/pipermail/dovecot/2018-December/subject.html#113954>[
    author ]
   
<https://www.dovecot.org/pipermail/dovecot/2018-December/author.html#113954>

------------------------------------------------------------------------

I'm running Dovecot 2.3.4 with Single Instance Storage (SIS) and zlib
and I am frequently seeing errors where files are missing in the
attachments directory even though the zipped file in the hash directory
is actually there.? This appears not to have been an issue in Dovecot
2.3.1 Requested output of the server below.

The error looks like this in the log

Dec 18 08:58:58 dovecot01 dovecot:
imap(username)<3692><RKnPGEh9fC3AqAAB>: Error: Mailbox INBOX/FOLDER:
UID=41:
read(attachments-connector(zlib(/usr/home/vmail/mail/username//mdbox/storage/m.411)))
failed:
read(/usr/home/vmail/attachments/91/08/91085773a0774909207852b8f9c98dbe22fe5a31-e88ba0276aae135cd2050000d09efc50[base64:19
b/l]) failed:
open(/usr/home/vmail/attachments/91/08/91085773a0774909207852b8f9c98dbe22fe5a31-e88ba0276aae135cd2050000d09efc50)
failed: No such file or directory (FETCH BODY[])

I'm also seeing errors when replication fails

dovecot: dsync-local(username)<bzg3MqjhGFxmFwAAzRvpBw>: Error:
dsync(servername):
read(attachments-connector(zlib(/usr/home/vmail/mail/username//mdbox/storage/m.413)))
failed:
read(/usr/home/vmail/attachments/45/42/4542dcf385f6b4a4dc8752b96db1d7ad8a8023d5-ac735007f6bd135c93440100cd1be907[base64:19
b/l]) failed:
open(/usr/home/vmail/attachments/45/42/4542dcf385f6b4a4dc8752b96db1d7ad8a8023d5-ac735007f6bd135c93440100cd1be907)
failed: No such file or directory (last sent=mail, last recv=mail (EOL))

It is not the case that every replication fail has a corresponding imap
access fail.

I am running Dovecot 2.3.4 on FreeBSD11.2 and I did not see this error
on Dovecot 2.3.1 FreeBSD 10.4.? Both installs had the same configuration
for Dovecot.? Both installs were made from source code and not packages
(as the need for the non standard fts and mysql options).

During my testing I had the 2.3.1 install receive mail and replicate to
the 2.3.4 and that was working fine (except for tcp connection issues
which I saw was fixed in the changelog).

Unfortunately having gone into testing with real account and mail being
sent to the 2.3.4 install I'm seeing the above errors.

My clients are using Thunderbird on desktops and Roundcube for webmail.

My users report they can see the e-mails fine in their webmail
(Roundcube) and only Thunderbird is giving issues.? I admit I don't know
how this can be the case as both systems use the same account details.

In everycase so far there is a file in the hashes directory I have hand
crafted the missing file (i.e. link) using ln and this restores access
in Thunderbird.

Clearly this is an big issue for my install and I'd appreciate any help!

At the moment it looks like my course will be to try and script creation
(recreation?) of the links from the error logs (I've seen people talking
about this online) but that isn't really what I'm after.

DOVECOT CONFIG


# 2.3.4 (0ecbaf23d): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.4 (60b0f48d)
# OS: FreeBSD 11.2-RELEASE-p5 amd64
# Hostname: dovecot01.cwa.uk.com
dict {
  ? acl = mysql:/usr/local/etc/dovecot/dovecot-dict-sql.conf.ext
}
disable_plaintext_auth = no
doveadm_password = # hidden, use -P to show it
first_valid_uid = 145
last_valid_uid = 145
mail_access_groups = mail
mail_attachment_dir = /usr/home/vmail/attachments
mail_attachment_min_size = 64 k
mail_gid = vmail
mail_location = mdbox:~/mdbox:INDEX=/indexdisk/indexes/%n
mail_plugins = " fts fts_solr acl zlib notify replication"
mail_privileged_group = mail
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart extracttext
namespace {
  ? list = yes
  ? location = mdbox:%%h/mdbox:INDEXPVT=~/mdbox/shared/%%n
  ? prefix = shared/%%n/
  ? separator = /
  ? subscriptions = yes
  ? type = shared
}
namespace inbox {
  ? inbox = yes
  ? location   ? mailbox Drafts {
  ??? auto = subscribe
  ??? special_use = \Drafts
  ? }
  ? mailbox Junk {
  ??? auto = subscribe
  ??? special_use = \Junk
  ? }
  ? mailbox Sent {
  ??? auto = subscribe
  ??? special_use = \Sent
  ? }
  ? mailbox "Sent Messages" {
  ??? special_use = \Sent
  ? }
  ? mailbox Trash {
  ??? auto = subscribe
  ??? special_use = \Trash
  ? }
  ? prefix   ? type = private
}
passdb {
  ? args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
  ? driver = sql
}
plugin {
  ? acl = vfile
  ? acl_shared_dict = proxy::acl
  ? fts = solr
  ? fts_autoindex = yes
  ? fts_solr = url=http://localhost:8983/solr/dovecot/
  ? mail_replica = tcp:192.168.0.138:12345
  ? replication_dsync_parameters = -d -l 30 -U
  ? sieve = ~/.dovecot.sieve
  ? sieve_dir = ~/sieve
  ? sieve_duplicate_default_period = 1h
  ? sieve_duplicate_max_period = 1d
  ? sieve_global_dir = /usr/home/vmail/sieve/global/
  ? sieve_global_path = /usr/home/vmail/default.sieve
  ? sieve_vacation_default_period = 1d
  ? zlib_save = gz
  ? zlib_save_level = 6
}
protocols = imap sieve
service aggregator {
  ? fifo_listener replication-notify-fifo {
  ??? user = vmail
  ? }
  ? unix_listener replication-notify {
  ??? user = vmail
  ? }
}
service auth {
  ? unix_listener auth-userdb {
  ??? mode = 0777
  ? }
}
service dict {
  ? unix_listener dict {
  ??? mode = 0600
  ??? user = vmail
  ? }
}
service doveadm {
  ? inet_listener {
  ??? port = 12345
  ? }
}
service replicator {
  ? process_min_avail = 1
  ? unix_listener replicator-doveadm {
  ??? mode = 0600
  ??? user = vmail
  ? }
}
ssl = no
userdb {
  ? driver = prefetch
}
userdb {
  ? args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
  ? driver = sql
}
protocol lda {
  ? mail_plugins = " fts fts_solr acl zlib notify replication sieve"
  ? submission_host = 192.168.0.1:25
}
protocol imap {
  ? mail_plugins = " fts fts_solr acl zlib notify replication imap_acl
imap_zlib"
}


------------------------------------------------------------------------

  * Previous message:Change default mode for attachment files
    <https://www.dovecot.org/pipermail/dovecot/2018-December/113952.html>
  * Next message:High Load average on NFS Spool - v.2.1.15 & 2.2.13
    <https://www.dovecot.org/pipermail/dovecot/2018-December/113957.html>
  * *Messages sorted by:*[ date ]
   
<https://www.dovecot.org/pipermail/dovecot/2018-December/date.html#113954>[
    thread ]
   
<https://www.dovecot.org/pipermail/dovecot/2018-December/thread.html#113954>[
    subject ]
   
<https://www.dovecot.org/pipermail/dovecot/2018-December/subject.html#113954>[
    author ]
   
<https://www.dovecot.org/pipermail/dovecot/2018-December/author.html#113954>

------------------------------------------------------------------------
More information about the dovecot mailing list 
<https://dovecot.org/mailman/listinfo/dovecot>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20181227/d114b930/attachment-0001.html>