search for: mysociety

Olly, Thanks again very much for helping me improve my understanding of Xapian and Omega. Thanks especially for pointing out that my idea of trying to generate a snippet from stemmed text lacking capitalization and punctuation would probably not produce a user-friendly result. But I'm still doubtful that expanding the sample size could be the right way to obtain excerpts from the document

Whoops, forgot to say, as Olly said, much of that could probably now be simplified with Xapian's snippet() function, which I can only assume did not exist back when all this was written! :) ATB, Matthew On Mon, 21 Sep 2020 at 09:28, Matthew Somerville <matthew at mysociety.org> wrote: > Hi, > > Ha, I was reading this thread thinking TheyWorkForYou (which I help > maintain) does highlight terms wherever they are, and then you mentioned it > :) > The code is open source; it is quite old and probably hair-raising, but as > you say it does basica...

...database stores terms/boolean terms/values for the text, and for the document itself stores only an identifier. It works by doing the Xapian search, then fetching the resultant IDs from the database, then it boils down to calling prepare_search_result_for_display on each result: https://github.com/mysociety/theyworkforyou/blob/master/www/includes/easyparliament/hansardlist.php#L1279-L1306 Which uses two functions to then work out the extract, position_of_first_word: https://github.com/mysociety/theyworkforyou/blob/master/www/includes/easyparliament/searchengine.php#L562 and highlight: https://github.c...

Hi, I'm getting quite odd results using get_matches_estimated() that I haven't seen before; we've just added a bunch of new data to the database. Xapian 1.0.7, checkatleast is set to 100. The database will get new stuff added to it automatically around 8.30am BST, so it's possible this might affect the links I provide, I guess. But I'll note what is currently happening as

Hi, We were running xapian-check on one of our Xapian indexes and it returns the following error: position: baseB blocksize=8K items=809896869 lastblock=2090419 revision=3161 levels=3 root=2084903 Failed to check B-tree: DatabaseError: Items not in sorted order The other tables verify without issue. It looks like our oldest backup of this database (a month old) has the same issue. Searching and