search for: softcommit

...not really </div> <div> >> concerned about qucik visibility but just getting things into the index </div> <div> >> efficiently, a better approach would be for dovecot plugin not to send </div> <div> >> any commit or softCommit (or waitSearcher either) and that should speed </div> <div> >> things up. You'd need to configure solr with a long autoSoftCommit </div> <div> >> maxTime and a reasonable autoCommit maxTime, which you could then </div> <div&...

On 4/4/19 6:47 PM, dovecot-request at dovecot.org wrote: > For a typical Solr index, 60 seconds is an eternity. Most people aim > for query times of 100 milliseconds or less, and they often achieve > that goal. I'm pretty sure I get these while indexing, not querying. Apr 04 16:44:50 host dovecot[114690]: indexer-worker(me at example.com): Error: fts_solr: Indexing failed: Request

...r whether this is the case by default. I think it is. > If there is an issue on initial indexing, where you are not really > concerned about qucik visibility but just getting things into the index > efficiently, a better approach would be for dovecot plugin not to send > any commit or softCommit (or waitSearcher either) and that should speed > things up. You'd need to configure solr with a long autoSoftCommit > maxTime and a reasonable autoCommit maxTime, which you could then > reconfigure when the load was done. Solr ships with autoCommit set to 15 seconds and openSearcher s...

...> Error: fts_solr: Indexing failed: Request timed out (Request queued > 66.015 secs ago, 1 attempts in 66.005 secs, 63.146 in http ioloop, 0.000 > in other ioloops, connected 94.903 secs ago) Doing a TCP dump on indexing operations which consistently fail, I see that there's a lot of softCommits which never get an HTTP answer: ============================================== POST /solr/dovebody/update HTTP/1.1 Host: localhost:8983 Date: Wed, 10 Apr 2019 14:22:29 GMT Expect: 100-continue Content-Length: 47 Connection: Keep-Alive Content-Type: text/xml HTTP/1.1 100 Continue <commit soft...

...e case by default.? I think it is. > >> If there is an issue on initial indexing, where you are not really >> concerned about qucik visibility but just getting things into the index >> efficiently, a better approach would be for dovecot plugin not to send >> any commit or softCommit (or waitSearcher either) and that should speed >> things up. You'd need to configure solr with a long autoSoftCommit >> maxTime and a reasonable autoCommit maxTime, which you could then >> reconfigure when the load was done. > > Solr ships with autoCommit set to 15 secon...

...m the TCP windows running full and Dovecot backing of ... until it times out and close the connection. >From my understanding of how Dovecot operates towards Solr it will flush ~1000 documents towards Solr in /update request until it has traversed the mailbox (let's say 20.000 mails), doing softCommits after each. But is it really reasonable for Dovecot to expect that no request will take more than 60s to process by Solr? It doesn't seem like my Solr can handle that, although it does process documents and it does reasonably fast clear pending documents after Dovecot closes the connection....

...t distracted half way and forgot to put a meaningfull subject so the archive could figure out the thread. - resending. On 4/14/19 4:04 PM, dovecot-request at dovecot.org wrote: >> Solr ships with autoCommit set to 15 seconds and openSearcher set to >> false on the autoCommit.? The autoSoftCommit setting is not enabled by >> default, but depending on how the index was created, Solr might try to >> set autoSoftCommit to 3 seconds ... which is WAY too short. I just run with the default. 15s autoCommit and no autoSoftCommit >> This thread says that dovecot is sending explic...

...ot;?> > <response> > > <lst name="responseHeader"> > <int name="status">0</int> > <int name="QTime">0</int> > </lst> > </response> And Solr receives it. Your tcpdump doesn't show <commit softCommit="true" waitSearcher="true"/> being sent though. Do you see it being sent anywhere? Does it make the mails visible if you run it yourself? Or if you run hard commit? : curl http://<host>:8983/solr/update?commit=true -------------- next part -------------- An HTML attac...

...forgot to put a meaningfull > subject so the archive could figure out the thread. - resending. > > On 4/14/19 4:04 PM, dovecot-request at dovecot.org wrote: > >>> Solr ships with autoCommit set to 15 seconds and openSearcher set to >>> false on the autoCommit.? The autoSoftCommit setting is not enabled by >>> default, but depending on how the index was created, Solr might try to >>> set autoSoftCommit to 3 seconds ... which is WAY too short. > I just run with the default. 15s autoCommit and no autoSoftCommit > >>> This thread says that dovec...

...t;responseHeader"> >>> ?<int name="status">0</int> >>> ?<int name="QTime">0</int> >>> </lst> >>> </response> >> >> And Solr receives it. Your tcpdump doesn't show <commit >> softCommit="true" waitSearcher="true"/> being sent though. Do you see >> it being sent anywhere? > > Yes: > > ################### TCPDUMP ################################### > POST /solr/dovecot/update HTTP/1.1 > Host: localhost:8983 > Date: Mon, 28 May 20...

I've built a new dovecot + fts-solr instance; I've now picked up & am running the recently released solr 8.7.0. In a test account, I've one message -- in the 'Drafts' folder. On exec of fts 'index' doveadm index -u testuser at example.com -q '*' I get a 401 unauthorized indexing failure, only for the one dir+message in 'Drafts', 2020-11-13

...nning full and Dovecot backing of ... until it times out > and close the connection. > > From my understanding of how Dovecot operates towards Solr it will flush > ~1000 documents towards Solr in /update request until it has traversed > the mailbox (let's say 20.000 mails), doing softCommits after each. > > But is it really reasonable for Dovecot to expect that no request will > take more than 60s to process by Solr? > It doesn't seem like my Solr can handle that, although it does process > documents and it does reasonably fast clear pending documents after > Dov...

...>> <lst name="responseHeader"> >> ?<int name="status">0</int> >> ?<int name="QTime">0</int> >> </lst> >> </response> > > And Solr receives it. Your tcpdump doesn't show <commit > softCommit="true" waitSearcher="true"/> being sent though. Do you see it > being sent anywhere? Yes: ################### TCPDUMP ################################### POST /solr/dovecot/update HTTP/1.1 Host: localhost:8983 Date: Mon, 28 May 2018 10:18:05 GMT Expect: 100-continue Co...

On 4/14/19 4:04 PM, dovecot-request at dovecot.org wrote: >> Solr ships with autoCommit set to 15 seconds and openSearcher set to >> false on the autoCommit.? The autoSoftCommit setting is not enabled by >> default, but depending on how the index was created, Solr might try to >> set autoSoftCommit to 3 seconds ... which is WAY too short. I just run with the default. 15s autoCommit and no autoSoftCommit >> This thread says that dovecot is sending explic...

...rty update.autoCreateFields \ -value false are fine. it's the indexing that's not ... > You may also want to test your solr installation from the command line. For example this does a manual soft commit: > > curl https://user:password at solr.example.com:443/solr/dovecot/update?softCommit=true <https://user:password at solr.example.com:443/solr/dovecot/update?softCommit=true> > If your solr server is correctly configured you should get back something like this: > > <?xml version="1.0" encoding="UTF-8"?> > <response> > > <...

...ious thing to check is the url= part of your fts_solr setting including the credentials and the solr core name. You may also want to test your solr installation from the command line. For example this does a manual soft commit: curl https://user:password at solr.example.com:443/solr/dovecot/update?softCommit=true <https://user:password at solr.example.com:443/solr/dovecot/update?softCommit=true> If your solr server is correctly configured you should get back something like this: <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader&...

On 21 May 2018, at 14.11, kadafax at gmail.com wrote: > > Le 21/05/2018 ? 12:38, Aki Tuomi a ?crit : >> can you try turning on pluign { fts_enforced = yes } and repeat your test? > > Same (wrong) result: > 1. Send an email with "too6Ouka" in the body > > 2. Search against "too6Ouka": > # doveadm search -u username mailbox INBOX body too6Ouka