Hi,
in the latest weeks i'm working on the Solr integration and
immediately i've faced the assertion failure errors, on 2.0.19, 2.2.9 and
2.3.11.3 servers in our network.
Reading the thread on debian ML, I realize this issue is related to nested
MIME and it affects large mailboxes
In my case, the error in dovecot.log pairs with the following on solr.log
and it seems the rows value has the same value of the last UID recorded in
the mailbox.
For your reference, here is the Solr logs, where *2276996170* is the value
passed by Dovecot as rows number and it clearly don't fit with the rows
data type.
Have you had experienced the same behaviour? Is there a workaround?
Thanks
Antonino
2020-12-30 12:07:33.897 ERROR (qtp1162918744-918) [ x:qadisha]
o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: For input
string: "2276996170"
at org.apache.solr.common.params.SolrParams.getInt(SolrParams.java:236)
at org.apache.solr.search.QParser.getSortSpec(QParser.java:274)
at
org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:189)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:337)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:214)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2627)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:795)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:568)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
at
org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string:
"2276996170"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:583)
at java.lang.Integer.valueOf(Integer.java:766)
at org.apache.solr.common.params.SolrParams.getInt(SolrParams.java:233)
... 46 more
2020-12-30 12:07:33.897 INFO (qtp1162918744-918) [ x:qadisha]
o.a.s.c.S.Request [qadisha] webapp=/solr path=/select
params={q={!lucene+q.op%3DAND}from:mykeyword+OR+to:mykeyword+OR+subject:mykeyword&fl=uid,score&sort=uid+asc&fq=%
2Bbox:6d2c381de797e95fbb160000c46e3ae7+%2Buser:info at
qadisha.it&rows=2276996170&wt=xml}
status=400 QTime=1
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20201230/c7327aa6/attachment-0001.html>
On 30/12/2020 16:04, Antonino Esposito wrote:> Hi, > > in the latest weeks i'm working on the Solr integration and > immediately?i've faced the?assertion failure errors, on 2.0.19, 2.2.9 > and 2.3.11.3 servers in our network. > Reading the thread on debian ML, I realize this issue is related to > nested MIME and it affects large mailboxes > > In my case, the error in dovecot.log pairs with the following on > solr.log and it seems the rows value has the same value of the last > UID recorded in the mailbox.? > > For your reference, here is the Solr logs, where *2276996170* is the > value passed by Dovecot as rows number and it clearly don't fit with > the rows data type. > > Have you had experienced?the?same behaviour? Is there a workaround? > Thanks > Antonino >Hi Antonio out of curiosity, what does the dovecot log show for this issue? John -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20201231/342e0316/attachment.html>
John Fawcett
2020-Dec-31 08:15 UTC
Solr and FTS - assertion failure [proposed patch for upper bound on rows in solr search]
On 30/12/2020 16:04, Antonino Esposito wrote:> Hi, > > in the latest weeks i'm working on the Solr integration and > immediately?i've faced the?assertion failure errors, on 2.0.19, 2.2.9 > and 2.3.11.3 servers in our network. > Reading the thread on debian ML, I realize this issue is related to > nested MIME and it affects large mailboxes > > In my case, the error in dovecot.log pairs with the following on > solr.log and it seems the rows value has the same value of the last > UID recorded in the mailbox.? > > For your reference, here is the Solr logs, where *2276996170* is the > value passed by Dovecot as rows number and it clearly don't fit with > the rows data type. > > Have you had experienced?the?same behaviour? Is there a workaround? > Thanks > AntoninoWhatever the reason for this happening, it would make sense not to supply unbounded values to solr. The "rows" value that is passed for a lookup on a single mailbox is the value of the uidnext for the searched mailbox. For lookups on multiple mailboxes there is a hard coded value: #define SOLR_MAX_MULTI_ROWS 100000 If 100000 is a good maximum for lookups on multiple mailboxes it could also be a good upper bound for lookups on single mailboxes too. My proposed patch, which stops too large "rows" values going to solr is as follows. This doesn't solve the issue of why uidnext is so large in the first place for the specific mailbox. Nevertheless I think it makes sense both as a potential workaround to the original issue and to incorporate it as a safeguard. If the hard-coded value is too limiting, it could be made configurable. diff -ur dovecot-2.3.11.3-orig/src/plugins/fts-solr/fts-backend-solr.c dovecot-2.3.11.3/src/plugins/fts-solr/fts-backend-solr.c --- dovecot-2.3.11.3-orig/src/plugins/fts-solr/fts-backend-solr.c?????? 2020-08-12 14:20:41.000000000 +0200 +++ dovecot-2.3.11.3/src/plugins/fts-solr/fts-backend-solr.c??? 2020-12-31 09:05:07.681897716 +0100 @@ -838,7 +838,7 @@ ??????? str = t_str_new(256); ??????? str_printfa(str, "wt=xml&fl=uid,score&rows=%u&sort=uid+asc&q=%%7b!lucene+q.op%%3dAND%%7d", -?????????????????? status.uidnext); +?????????????????? I_MIN(status.uidnext,SOLR_MAX_MULTI_ROWS)); ??????? prefix_len = str_len(str); ??????? if (solr_add_definite_query_args(str, args, and_args)) { John -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20201231/9174a598/attachment-0001.html>