On 07/10/2019 15:13, Fabian Kuran via dovecot wrote:> Hi, > >> Am 23.09.2019 um 09:36 schrieb Filip Hanes <filip.hanes at ringier.sk >> <mailto:filip.hanes at ringier.sk>>: >> >> >> ?t 19. 9. 2019 o?15:30 Fabian via dovecot <dovecot at dovecot.org >> <mailto:dovecot at dovecot.org>> nap?sal(a): >> >> Are there any other hints or tips regarding this ?invalid uid >> ?0?"-message? >> >> from fts-solr source: >> ????case?SOLR_XML_CONTENT_STATE_UID: >> ????????if?(uint32_parse(str,?len,?&ctx->uid)?<?0?||?ctx->uid?==?0)?{ >> ????????????i_error("fts_solr:?received?invalid?uid?'%s'",?t_strndup(str,?len)); >> It means that plugin tries to parse '0' when in state of parsing uid >> in xml response from solr or there is a bug in xml response parser. >> I would enable rawlog_dir in config and look there for unusual >> response or extra tags near usual uid in xml response. For example: >> |plugin { fts = solr fts_solr = url=... rawlog_dir=/var/log/fts-solr/ }| >> > > Thanks for this tip!?The rawlog_dir option is not yet supported in our > version 2.3.4. What we could provide now would be the XML-Response > from Solr. The corresponding search query sometimes causes this error. > But Solr's answer seems to always remain the same - in my opinion this > already indicates a bug of Dovecot? > > The XML response can be downloaded here: > https://owncloud.eideo.de/index.php/s/8oyJaf5HBK6zdbrBased on the XML response above, I investigated this problem thoroughly and determined that this is a pretty severe bug in the Solr XML response parsing code. This occurs only when the response is rather large and the boundary between two read chunks falls in the middle of a numeric value (that happens to end in '0'). Tracking internally as DOP-1470. I think this may also explain some of the weird reports about invalid UIDs that we've seen in recent history on this mailing list. Regards, Stephan.
On 08/10/2019 02:52, Stephan Bosch via dovecot wrote:> > > On 07/10/2019 15:13, Fabian Kuran via dovecot wrote: >> Hi, >> >>> Am 23.09.2019 um 09:36 schrieb Filip Hanes <filip.hanes at ringier.sk >>> <mailto:filip.hanes at ringier.sk>>: >>> >>> >>> ?t 19. 9. 2019 o?15:30 Fabian via dovecot <dovecot at dovecot.org >>> <mailto:dovecot at dovecot.org>> nap?sal(a): >>> >>> ??? Are there any other hints or tips regarding this ?invalid uid >>> ??? ?0?"-message? >>> >>> from fts-solr source: >>> ????case?SOLR_XML_CONTENT_STATE_UID: >>> ????????if?(uint32_parse(str,?len,?&ctx->uid)?<?0?||?ctx->uid?==?0)?{ >>> ????????????i_error("fts_solr:?received?invalid?uid?'%s'",?t_strndup(str,?len)); >>> >>> It means that plugin tries to parse '0' when in state of parsing uid >>> in xml response from solr or there is a bug in xml response parser. >>> I would enable rawlog_dir in config and look there for unusual >>> response or extra tags near usual uid in xml response. For example: >>> |plugin { fts = solr fts_solr = url=... >>> rawlog_dir=/var/log/fts-solr/ }| >>> >> >> Thanks for this tip!?The rawlog_dir option is not yet supported in >> our version 2.3.4. What we could provide now would be the >> XML-Response from Solr. The corresponding search query sometimes >> causes this error. But Solr's answer seems to always remain the same >> - in my opinion this already indicates a bug of Dovecot? >> >> The XML response can be downloaded here: >> https://owncloud.eideo.de/index.php/s/8oyJaf5HBK6zdbr > > Based on the XML response above, I investigated this problem > thoroughly and determined that this is a pretty severe bug in the Solr > XML response parsing code. This occurs only when the response is > rather large and the boundary between two read chunks falls in the > middle of a numeric value (that happens to end in '0'). > > Tracking internally as DOP-1470. > > I think this may also explain some of the weird reports about invalid > UIDs that we've seen in recent history on this mailing list.Preliminary fix is here: https://github.com/stephanbosch/dovecot-core/commits/fix-solr-xml-parser It's big due to some code restructuring for making a proper unit test. I am not sure when this will be in a release. Regards, Stephan.
Is there a way to get this as a patchset against 2.3.8? On Tue, Oct 8, 2019 at 12:03 PM Stephan Bosch via dovecot < dovecot at dovecot.org> wrote:> > > On 08/10/2019 02:52, Stephan Bosch via dovecot wrote: > > > > > > On 07/10/2019 15:13, Fabian Kuran via dovecot wrote: > >> Hi, > >> > >>> Am 23.09.2019 um 09:36 schrieb Filip Hanes <filip.hanes at ringier.sk > >>> <mailto:filip.hanes at ringier.sk>>: > >>> > >>> > >>> ?t 19. 9. 2019 o 15:30 Fabian via dovecot <dovecot at dovecot.org > >>> <mailto:dovecot at dovecot.org>> nap?sal(a): > >>> > >>> Are there any other hints or tips regarding this ?invalid uid > >>> ?0?"-message? > >>> > >>> from fts-solr source: > >>> case SOLR_XML_CONTENT_STATE_UID: > >>> if (uint32_parse(str, len, &ctx->uid) < 0 || ctx->uid == 0) { > >>> > i_error("fts_solr: received invalid uid '%s'", t_strndup(str, len)); > > >>> > >>> It means that plugin tries to parse '0' when in state of parsing uid > >>> in xml response from solr or there is a bug in xml response parser. > >>> I would enable rawlog_dir in config and look there for unusual > >>> response or extra tags near usual uid in xml response. For example: > >>> |plugin { fts = solr fts_solr = url=... > >>> rawlog_dir=/var/log/fts-solr/ }| > >>> > >> > >> Thanks for this tip! The rawlog_dir option is not yet supported in > >> our version 2.3.4. What we could provide now would be the > >> XML-Response from Solr. The corresponding search query sometimes > >> causes this error. But Solr's answer seems to always remain the same > >> - in my opinion this already indicates a bug of Dovecot? > >> > >> The XML response can be downloaded here: > >> https://owncloud.eideo.de/index.php/s/8oyJaf5HBK6zdbr > > > > Based on the XML response above, I investigated this problem > > thoroughly and determined that this is a pretty severe bug in the Solr > > XML response parsing code. This occurs only when the response is > > rather large and the boundary between two read chunks falls in the > > middle of a numeric value (that happens to end in '0'). > > > > Tracking internally as DOP-1470. > > > > I think this may also explain some of the weird reports about invalid > > UIDs that we've seen in recent history on this mailing list. > > Preliminary fix is here: > https://github.com/stephanbosch/dovecot-core/commits/fix-solr-xml-parser > > It's big due to some code restructuring for making a proper unit test. I > am not sure when this will be in a release. > > Regards, > > Stephan. > >-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx at gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106 -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20191009/ac4856d9/attachment-0001.html>