Timo Sirainen
2018-May-28 11:04 UTC
Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
On 28 May 2018, at 13.28, kfx <kadafax at gmail.com> wrote:> > >> Especially what is in the "fts" header vs. "next uid" header? Does the UID in "fts" header keep changing every time you save a new mail? I suppose it will. > > Diff between 2 emails: > next uid = 30104 | next uid = 30105 > last_indexed_uid = 30103 | last_indexed_uid = 30104So Dovecot thinks it has indexed everything.>> You could also monitor (e.g. tcpdump/wireshark) the network traffic between Dovecot <-> Solr what happens when a new mail arrives. I suspect Dovecot sends it to Solr, which for whatever reason just ignores the update. > > ################### TCPDUMP #################################### > POST /solr/dovecot/update HTTP/1.1 > Host: localhost:8983 > Date: Mon, 28 May 2018 10:18:05 GMT > Transfer-Encoding: chunked > Connection: Keep-Alive > Content-Type: text/xml > > <add><doc><field name="uid">37581</field><field name="box">e0c58a3093235153110800003ea484a8</field><field name="user">username</field><field name="id">37581/e0c58a3093235153110800003ea484a8/username</field><field name="body">Search Pattern: Kai8oovi..> </field></doc></add>And Dovecot sends the mail.> ######### SOLR'S RESPONSE ############### > <?xml version="1.0" encoding="UTF-8"?> > <response> > > <lst name="responseHeader"> > <int name="status">0</int> > <int name="QTime">0</int> > </lst> > </response>And Solr receives it. Your tcpdump doesn't show <commit softCommit="true" waitSearcher="true"/> being sent though. Do you see it being sent anywhere? Does it make the mails visible if you run it yourself? Or if you run hard commit? : curl http://<host>:8983/solr/update?commit=true -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20180528/ae857324/attachment-0001.html>
kfx
2018-May-28 11:23 UTC
Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
On 28/05/2018 13:04, Timo Sirainen wrote:> On 28 May 2018, at 13.28, kfx <kadafax at gmail.com > <mailto:kadafax at gmail.com>> wrote: >> >> >>> Especially what is in the "fts" header vs. "next uid" header? Does >>> the UID in "fts" header keep changing every time you save a new mail? >>> I suppose it will. >> >> Diff between 2 emails: >> next uid = 30104 ???| ???next uid = 30105 >> last_indexed_uid = 30103 ???| ???last_indexed_uid = 30104 > > So Dovecot thinks it has indexed everything. > >>> You could also monitor (e.g. tcpdump/wireshark) the network traffic >>> between Dovecot <-> Solr what happens when a new mail arrives. I >>> suspect Dovecot sends it to Solr, which for whatever reason just >>> ignores the update. >> >> ################### TCPDUMP #################################### >> POST /solr/dovecot/update HTTP/1.1 >> Host: localhost:8983 >> Date: Mon, 28 May 2018 10:18:05 GMT >> Transfer-Encoding: chunked >> Connection: Keep-Alive >> Content-Type: text/xml >> >> <add><doc><field name="uid">37581</field><field >> name="box">e0c58a3093235153110800003ea484a8</field><field >> name="user">username</field><field >> name="id">37581/e0c58a3093235153110800003ea484a8/username</field><field name="body">Search >> Pattern: Kai8oovi > .. >> </field></doc></add> > > And Dovecot sends the mail. > >> ######### SOLR'S RESPONSE ############### >> <?xml version="1.0" encoding="UTF-8"?> >> <response> >> >> <lst name="responseHeader"> >> ?<int name="status">0</int> >> ?<int name="QTime">0</int> >> </lst> >> </response> > > And Solr receives it. Your tcpdump doesn't show <commit > softCommit="true" waitSearcher="true"/> being sent though. Do you see it > being sent anywhere?Yes: ################### TCPDUMP ################################### POST /solr/dovecot/update HTTP/1.1 Host: localhost:8983 Date: Mon, 28 May 2018 10:18:05 GMT Expect: 100-continue Content-Length: 47 Connection: Keep-Alive Content-Type: text/xml <commit softCommit="true" waitSearcher="true"/> ################### /TCPDUMP ###################################> Does it make the mails visible if you run it > yourself? Or if you run hard commit? : > > curl http://<host>:8983/solr/update?commit=true ># curl http://127.0.0.1:8983/solr/update?commit=true <html> [0/0] <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Not Found</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/update. Reason: <pre> Not Found</pre></p> </body> </html> # curl http://127.0.0.1:8983/solr/dovecot/update?commit=true <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">0</int> </lst> </response> # doveadm search -u username mailbox INBOX body Kai8oovi ==> No result ('Kai8oovi' is the search pattern, it should returns 4 results)
kfx
2018-May-28 11:30 UTC
Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
On 28/05/2018 13:23, kfx wrote:> On 28/05/2018 13:04, Timo Sirainen wrote: >> On 28 May 2018, at 13.28, kfx <kadafax at gmail.com >> <mailto:kadafax at gmail.com>> wrote: >>> >>> >>>> Especially what is in the "fts" header vs. "next uid" header? Does >>>> the UID in "fts" header keep changing every time you save a new >>>> mail? I suppose it will. >>> >>> Diff between 2 emails: >>> next uid = 30104 ???| ???next uid = 30105 >>> last_indexed_uid = 30103 ???| ???last_indexed_uid = 30104 >> >> So Dovecot thinks it has indexed everything. >> >>>> You could also monitor (e.g. tcpdump/wireshark) the network traffic >>>> between Dovecot <-> Solr what happens when a new mail arrives. I >>>> suspect Dovecot sends it to Solr, which for whatever reason just >>>> ignores the update. >>> >>> ################### TCPDUMP #################################### >>> POST /solr/dovecot/update HTTP/1.1 >>> Host: localhost:8983 >>> Date: Mon, 28 May 2018 10:18:05 GMT >>> Transfer-Encoding: chunked >>> Connection: Keep-Alive >>> Content-Type: text/xml >>> >>> <add><doc><field name="uid">37581</field><field >>> name="box">e0c58a3093235153110800003ea484a8</field><field >>> name="user">username</field><field >>> name="id">37581/e0c58a3093235153110800003ea484a8/username</field><field >>> name="body">Search Pattern: Kai8oovi >> .. >>> </field></doc></add> >> >> And Dovecot sends the mail. >> >>> ######### SOLR'S RESPONSE ############### >>> <?xml version="1.0" encoding="UTF-8"?> >>> <response> >>> >>> <lst name="responseHeader"> >>> ?<int name="status">0</int> >>> ?<int name="QTime">0</int> >>> </lst> >>> </response> >> >> And Solr receives it. Your tcpdump doesn't show <commit >> softCommit="true" waitSearcher="true"/> being sent though. Do you see >> it being sent anywhere? > > Yes: > > ################### TCPDUMP ################################### > POST /solr/dovecot/update HTTP/1.1 > Host: localhost:8983 > Date: Mon, 28 May 2018 10:18:05 GMT > Expect: 100-continue > Content-Length: 47 > Connection: Keep-Alive > Content-Type: text/xml > > <commit softCommit="true" waitSearcher="true"/> > ################### /TCPDUMP ################################### > > > >> Does it make the mails visible if you run it yourself? Or if you run >> hard commit? : >> >> curl http://<host>:8983/solr/update?commit=true >> > > # curl http://127.0.0.1:8983/solr/update?commit=true > <html> > > > ??????????????????????? [0/0] > <head> > <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> > <title>Error 404 Not Found</title> > </head> > <body><h2>HTTP ERROR 404</h2> > <p>Problem accessing /solr/update. Reason: > <pre>??? Not Found</pre></p> > </body> > </html> > > # curl http://127.0.0.1:8983/solr/dovecot/update?commit=true > <?xml version="1.0" encoding="UTF-8"?> > <response> > > <lst name="responseHeader"> > ? <int name="status">0</int> > ? <int name="QTime">0</int> > </lst> > </response> > > # doveadm search -u username mailbox INBOX body Kai8oovi > ==> No result ('Kai8oovi' is the search pattern, it should returns 4 > results)In the web interface of solr at http://127.0.0.1:8983/solr/#/~cores/dovecot I can see:> lastModified: less than a minute ago > version:1428772 > numDocs:6353615 > maxDoc:6356213 > deletedDocs:2598So it IS indexing :( Just below I see the "optimized:" parameter followed by an icon which seems saying that is NOT "optimized". Don't know if it's relevant. This is driving me crazy :(
Maybe Matching Threads
- Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
- Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
- Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
- Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
- Solr connection timeout hardwired to 60s