I have Solr FTS enabled in dovecot.? It is behaving differently than I
would expect.
When I delete messages in Thunderbird normally, using IMAP to talk to
dovecot, the deletion hits Solr right away.? If I then ask Thunderbird
to empty the trash, that delete also hits Solr immediately.
But if I use shift-delete in Thunderbird, which deletes the message
immediately without going through Trash, things are different.? One
delete that I did (6 messages) took at least a minute before Solr saw
the request.? And on other tests that I ran, deleting one message, I
never did see the delete request hit Solr.? I waited for several minutes.
I use the shift-delete mechanism a lot more than the method that sends
to trash.? That way I do not need to remember to empty the trash.
Dovecot version is 2:2.3.16-2+ubuntu20.04 installed from the dovecot
repo, not from distro packages.? I don't think the Solr version matters,
but just in case I am wrong, that is 8.10.1.? I don't recall exactly
what I changed in the solrconfig.xml and managed-schema files, but it
should not be anything that affects how Solr FTS works.? The problem
seems to be that Solr FTS isn't sending deletes, not that Solr is
behaving incorrectly.
I just discovered that trying to get the dovecot config failed:
root at bilbo:~# doveconf -n
# 2.3.16 (7e2e900c1a): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.16 (09c29328)
doveconf: Fatal: Error in configuration file
/etc/dovecot/conf.d/10-master.conf line 64: Invalid size: $default_vsz_limit
After fixing that, this is what I get for the config:
root at bilbo:/etc/dovecot# doveconf -n
# 2.3.16 (7e2e900c1a): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.16 (09c29328)
# OS: Linux 5.11.0-1020-aws x86_64 Ubuntu 20.04.3 LTS ext4
# Hostname: REDACTED
default_vsz_limit = 1 G
dict {
? quota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext
}
first_valid_uid = 150
last_valid_uid = 150
listen = *
mail_gid = mail
mail_location = maildir:/var/vmail/%d/%u
mail_plugins = " quota fts fts_solr"
mail_privileged_group = mail
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart extracttext
namespace inbox {
? inbox = yes
? location ? mailbox Drafts {
??? special_use = \Drafts
? }
? mailbox Junk {
??? special_use = \Junk
? }
? mailbox Sent {
??? special_use = \Sent
? }
? mailbox "Sent Messages" {
??? special_use = \Sent
? }
? mailbox Trash {
??? special_use = \Trash
? }
? prefix }
passdb {
? args = /etc/dovecot/dovecot-sql.conf.ext
? driver = sql
}
plugin {
? fts = solr
? fts_autoindex = yes
? fts_solr = url=http://localhost:8983/solr/dovecot/
? quota = dict:User quota::proxy::quota
? quota_grace = 10%%
? quota_max_mail_size = 100M
? quota_status_nouser = DUNNO
? quota_status_overquota = 552 5.2.2 Mailbox is full
? quota_status_success = DUNNO
? sieve =
file:/var/vmail/sieve/%d/%u/sieve;active=/var/vmail/sieve/%d/%u/.dovecot.sieve
? sieve_default = file:/var/vmail/global.sieve
}
postmaster_address = postmaster at REDACTED
protocols = " imap lmtp sieve pop3"
service auth {
? unix_listener /var/spool/postfix/private/auth {
??? group = postfix
??? mode = 0666
??? user = postfix
? }
? unix_listener auth-userdb {
??? group = mail
??? mode = 0666
??? user = vmail
? }
}
service dict {
? unix_listener dict {
??? group = postfix
??? mode = 0660
??? user = vmail
? }
}
service imap {
? vsz_limit = 1 G
}
service stats {
? unix_listener stats-reader {
??? group = vmail
??? mode = 0660
??? user = vmail
? }
? unix_listener stats-writer {
??? group = vmail
??? mode = 0660
??? user = vmail
? }
}
ssl_cert = </etc/ssl/certs/local/mainwildcards.pem
ssl_client_ca_dir = /etc/ssl/certs
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
userdb {
? args = /etc/dovecot/dovecot-sql.conf.ext
? driver = sql
}
protocol lmtp {
? mail_plugins = " quota fts fts_solr sieve"
}
protocol lda {
? mail_plugins = " quota fts fts_solr sieve"
}
protocol imap {
? mail_max_userip_connections = 20
? mail_plugins = " quota fts fts_solr imap_quota"
}
I repeated the delete test after fixing the config and there was no
change -- the delete request never made it to Solr.
Reindexing completely for my dovecot install takes about 8 minutes.? I
have less than 150K messages.? Reindexing for some could take hours or
days.? This is what I use to initiate a full reindex:
root at bilbo:~# cat /usr/local/sbin/reindexsolr
#!/bin/sh
DOVECOT_SOLR_URL_BASE="http://localhost:8983/solr/dovecot"
DEL_ALL_QUERY_XML="<delete><query>*:*</query></delete>"
MAIL_ROOT=/var/vmail
service dovecot stop
curl \
? "${DOVECOT_SOLR_URL_BASE}/update?commit=true&optimize=true" \
? -H "Content-Type: text/xml" \
? --data-binary "${DEL_ALL_QUERY_XML}"
find ${MAIL_ROOT} -name "dovecot.index*" -print0 | xargs -0 rm -f
service dovecot start
doveadm index -A -q '*'