Xapian discuss - Mar 2008 - Xapian performance testing

On 5-11-2007 14:40, Olly Betts wrote:
> BTW, I have implemented the hoisting of the positional information
> checking part of NEAR and PHRASE, so that the "AND" inside can be
> merged with other AND and FILTER operations.  This gave a big
> performance boost to the slow queries (~50% saving in time just
> from this one change) and a good boost to the other queries (~25%
> saving from just this change).
> 
> This optimisation and all the earlier ones are in 1.0.4, so once you
> upgrade to that, it would be interesting to see what the slow query log
> looks like with these new optimisations in place.
We finally found time to upgrade our 0.9.8 to 1.0.5 and reindex the 
whole database. The results so far are quite good.
When taking the daily average of our forum search result page it went 
down from about 0.55 seconds to about 0.38 seconds.
Looking at the log-files, the slow query log file (queries taking more 
than 2 seconds) dramatically reduced in size. Prior to the update we had 
3190 and 3112 lines in a week and now in the latest week it had only 863 
lines.

The slowest queries seem to be the single-term phrases with a single 
character attached to a common word like 'd-link', 's-video' and
variants on that with only a few additional terms. As expected, I don't 
see any version numbers anymore in the slow query log.

Best regards,

Arjen van der Meijden
Tweakers.net

I have encountered the same issue having slow queries when term
contain dash characters.

I do not have solution except that I replaced the '-' with
'dash'
inside of the index.
Example: 'd-link', 's-video' as temporary solution: '
ddashlink', ' sdashvideo'

Perhaps we could address this slowness issue of dash characters in the
near future.

Kevin Duraj
http://myhealthcare.com


On Tue, Mar 18, 2008 at 1:26 AM, Arjen van der Meijden
<acmmailing at tweakers.net> wrote:
> On 5-11-2007 14:40, Olly Betts wrote:
>  > BTW, I have implemented the hoisting of the positional information
>  > checking part of NEAR and PHRASE, so that the "AND" inside
can be
>  > merged with other AND and FILTER operations.  This gave a big
>  > performance boost to the slow queries (~50% saving in time just
>  > from this one change) and a good boost to the other queries (~25%
>  > saving from just this change).
>  >
>  > This optimisation and all the earlier ones are in 1.0.4, so once you
>  > upgrade to that, it would be interesting to see what the slow query
log
>  > looks like with these new optimisations in place.
>
>  We finally found time to upgrade our 0.9.8 to 1.0.5 and reindex the
>  whole database. The results so far are quite good.
>  When taking the daily average of our forum search result page it went
>  down from about 0.55 seconds to about 0.38 seconds.
>  Looking at the log-files, the slow query log file (queries taking more
>  than 2 seconds) dramatically reduced in size. Prior to the update we had
>  3190 and 3112 lines in a week and now in the latest week it had only 863
>  lines.
>
>  The slowest queries seem to be the single-term phrases with a single
>  character attached to a common word like 'd-link',
's-video' and
>  variants on that with only a few additional terms. As expected, I
don't
>  see any version numbers anymore in the slow query log.
>
>  Best regards,
>
>  Arjen van der Meijden
>  Tweakers.net
>
>  _______________________________________________
>  Xapian-discuss mailing list
>  Xapian-discuss at lists.xapian.org
>  http://lists.xapian.org/mailman/listinfo/xapian-discuss
>