> On 2. Dec 2021, at 10.57, Einar Bjarni Halld?rsson <einar at isnic.is> wrote: > root at ht-mailstore01:/data/mail/hostmasterlog/mdbox/mailboxes/INBOX/dbox-Mails # doveadm dump . | grep -c hdr.MESSAGE-ID > 4464736 > > In dovecot config I have `fts_enforced = yes` and after the search for Message-ID finishes and returns UID, I can do a FETCH .. ENVELOPE on it and it responds immediately. > I was under the impression that fts_enforced forces all searches, headers and body, to go to solr. Then all dovecot would have to do was to return the UID returned by solr. > Unless solr doesn't return UID and dovecot has to take the result from solr and lookup the UID, and with a full 1 GB cache file it always has to scan the whole index? > > Nothing seems to be completely broken, I always receive a result, it's just that it takes 30 seconds when I really want it to be ~5 seconds at most. > > I guess if we can't find a solution and 30 seconds becomes a real problem, we'll split the mailbox up by years. It should help with the size of the cache. It makes the searching code a little more complicated since it has to figure out the sent date before it can search, but it's doable.If you have fts_enforced = yes then that SEARCH probably goes to solr even if message-id is found from cache. So the delay is probably on solr side. Can you try to remove fts_enforced for that one particular user? Sami
> If you have fts_enforced = yes then that SEARCH probably goes to solr even if message-id is found from cache. So the delay is probably on solr side. > > Can you try to remove fts_enforced for that one particular user? >I only added fts_enforced yesterday. When I sent the original post I wasn't running with fts_enforced. I did finally manage to query solr with curl with the query dovecot issues and I get this response: <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> ? <int name="status">0</int> ? <int name="QTime">1</int> ? <lst name="params"> ??? <str name="json">{??? "params": {??????? "q": "{!lucene q.op=AND} hdr:<64jf4CStuxzUvATjBbRVP3IbJnMbEguVXzOa1OSmjIg at min.isnic.is>", "fl": "uid,score",??????? "sort": "uid asc",??????? "fq": "+box:c92f64f79f0d1ed01e6d5b314f04886c +user:hostmasterlog", "rows": 5327110,??????? "wt": xml???? }}</str> ? </lst> </lst> <result name="response" numFound="1" start="0" maxScore="131.64821" numFoundExact="true"> ? <doc> ??? <long name="uid">5166713</long> ??? <float name="score">131.64821</float></doc> </result> </response> -- If I search for the UID I get -- 3 fetch 5166713 ENVELOPE 3 BAD Error in IMAP command FETCH: Invalid messageset (0.001 + 0.000 secs). -- If I run the search in IMAP I get -- 4 SEARCH HEADER Messgage-ID <64jf4CStuxzUvATjBbRVP3IbJnMbEguVXzOa1OSmjIg at min.isnic.is> * SEARCH * 4465180 EXISTS * 88 RECENT 4 OK Search completed (29.674 + 0.000 + 29.523 secs). -- Isn't it likely that the dovecot index and solr index are out of sync? Solr seems to point to a UID that doesn't exist in dovecot. If so, how can I fix it? .einar