Looking further at tcpdumps of the Dovecot->Solr traffic and Solr metrics it doesn't seem like there's anything suspicious apart from the TCP windows running full and Dovecot backing of ... until it times out and close the connection.>From my understanding of how Dovecot operates towards Solr it will flush~1000 documents towards Solr in /update request until it has traversed the mailbox (let's say 20.000 mails), doing softCommits after each. But is it really reasonable for Dovecot to expect that no request will take more than 60s to process by Solr? It doesn't seem like my Solr can handle that, although it does process documents and it does reasonably fast clear pending documents after Dovecot closes the connection. On the surface it looks like Dovecot is too impatient. /Peter On 4/10/19 6:25 PM, Peter Mogensen wrote:> > > On 4/4/19 6:57 PM, Peter Mogensen wrote: >> >> >> On 4/4/19 6:47 PM, dovecot-request at dovecot.org wrote: >>> For a typical Solr index, 60 seconds is an eternity. Most people aim >>> for query times of 100 milliseconds or less, and they often achieve >>> that goal. >> >> I'm pretty sure I get these while indexing, not querying. >> >> Apr 04 16:44:50 host dovecot[114690]: indexer-worker(me at example.com): >> Error: fts_solr: Indexing failed: Request timed out (Request queued >> 66.015 secs ago, 1 attempts in 66.005 secs, 63.146 in http ioloop, 0.000 >> in other ioloops, connected 94.903 secs ago) > > Doing a TCP dump on indexing operations which consistently fail, I see > that there's a lot of softCommits which never get an HTTP answer: > > =============================================> POST /solr/dovebody/update HTTP/1.1 > Host: localhost:8983 > Date: Wed, 10 Apr 2019 14:22:29 GMT > Expect: 100-continue > Content-Length: 47 > Connection: Keep-Alive > Content-Type: text/xml > > HTTP/1.1 100 Continue > > <commit softCommit="true" waitSearcher="true"/> > ===============================================....
On 12/04/2019 12:09, Peter Mogensen via dovecot wrote:> Looking further at tcpdumps of the Dovecot->Solr traffic and Solr > metrics it doesn't seem like there's anything suspicious apart from the > TCP windows running full and Dovecot backing of ... until it times out > and close the connection. > > From my understanding of how Dovecot operates towards Solr it will flush > ~1000 documents towards Solr in /update request until it has traversed > the mailbox (let's say 20.000 mails), doing softCommits after each. > > But is it really reasonable for Dovecot to expect that no request will > take more than 60s to process by Solr? > It doesn't seem like my Solr can handle that, although it does process > documents and it does reasonably fast clear pending documents after > Dovecot closes the connection. > > On the surface it looks like Dovecot is too impatient. > > /PeterIf this value was made configurable people could set it to what they want. However the underlying problem is likely on solr configuration. Is this a problem only on initial indexing or an ongoing problem after initial indexing? The parameters that the solr plugin are using are designed to make documents visible to searches quickly. If there is an issue on initial indexing, where you are not really concerned about qucik visibility but just getting things into the index efficiently, a better approach would be for dovecot plugin not to send any commit or softCommit (or waitSearcher either) and that should speed things up. You'd need to configure solr with a long autoSoftCommit maxTime and a reasonable autoCommit maxTime, which you could then reconfigure when the load was done. If you're using dovecot built from source code it should be possible to test that by some minor modification of the code in? fts-backend-solr.c. John
On 4/13/2019 4:29 AM, John Fawcett via dovecot wrote:> If this value was made configurable people could set it to what they > want. However the underlying problem is likely on solr configuration.The Jetty that is included in Solr has its idle timeout set to 50 seconds. But in practice, I have not seen this timeout trigger ... and if the OP is seeing a 60 second timeout, then the 50 second idle timeout in Jetty must not be occurring. There may be a socket timeout configured on inter-server requests -- distributed queries or the load balancing that SolrCloud does. I can never remember whether this is the case by default. I think it is.> If there is an issue on initial indexing, where you are not really > concerned about qucik visibility but just getting things into the index > efficiently, a better approach would be for dovecot plugin not to send > any commit or softCommit (or waitSearcher either) and that should speed > things up. You'd need to configure solr with a long autoSoftCommit > maxTime and a reasonable autoCommit maxTime, which you could then > reconfigure when the load was done.Solr ships with autoCommit set to 15 seconds and openSearcher set to false on the autoCommit. The autoSoftCommit setting is not enabled by default, but depending on how the index was created, Solr might try to set autoSoftCommit to 3 seconds ... which is WAY too short. I will usually increase the autoCommit time to 60 seconds, just to reduce the amount of work that Solr is doing. The autoSoftCommit time, if it is used, should be set to a reasonably long value ... values between two and five minutes would be good. Attempting to use a very short autoSoftCommit time will usually lead to problems. This thread says that dovecot is sending explicit commits. One thing that might be happening to exceed 60 seconds is an extremely long commit, which is usually caused by excessive cache autowarming, but might be related to insufficient memory. The max heap setting on an out-of-the-box Solr install (5.0 and later) is 512MB. That's VERY small, and it doesn't take much index data before a much larger heap is required. Thanks, Shawn