search for: mset

Olly Betts <olly at survex.com> wrote: > On Wed, Aug 23, 2023 at 01:53:27PM +0000, Eric Wong wrote: > > I'm already retrying the ->get_mset operations; but now I'm > > wondering where I'd hit DatabaseModifiedErrors while inside a > > Xapian::MSetIterator loop. > > > > I assume ->get_document is a place where it gets thrown; > > but once a document is retrieved, can iterating through > > t...

...ic API that don't get > used by end users. However because LTR is outside the Xapian build > tree, we can't easily give it privileged access to Xapian internals. > Sorry for a delayed response. The way I was thinking of performing reranking with updated weights was to add a class MSetRanker (basically containing a copy of MSet to be re-ranked) to Xapian public API, which would be a friend of MSet::Internal class so that it could access/update the weights and sort MSetItems accordingly, returning an updated MSet. As you pointed out, two obvious issues with this are: 1) Unnecessa...

On Wed, Aug 23, 2023 at 01:53:27PM +0000, Eric Wong wrote: > I'm already retrying the ->get_mset operations; but now I'm > wondering where I'd hit DatabaseModifiedErrors while inside a > Xapian::MSetIterator loop. > > I assume ->get_document is a place where it gets thrown; > but once a document is retrieved, can iterating through > terms in one document (using Te...

On Thu, Aug 17, 2023 at 09:28:26PM +0000, Eric Wong wrote: > In other words, is it possible to avoid duplicates if new > documents are inserted into the DB by another process in-between > ->get_mset calls when reusing Xapian::Enquire objects? The Database object itself effectively does (it works in a snapshot of the state of the database when you open it, or last called reopen() which updates that snapshot to what's currently committed). However we don't currently have any locking of...

In other words, is it possible to avoid duplicates if new documents are inserted into the DB by another process in-between ->get_mset calls when reusing Xapian::Enquire objects? I do some expensive processing on each mset window, so I always limit the results to limit heap usage even if I'm planning on going through a big chunk of the DB: $mset = $enq->get_mset(0, 1000); do_something_slow_with_mset($mset); $mset = $en...

On Sun, Jul 31, 2016 at 12:44:16AM +0100, Olly Betts wrote: > Would a method which swapped two elements of an MSet provide what you > need? That would provide a more generic way to adjust the ranking of > an MSet which for example could be used to implement a diversification > feature or something like SQL "GROUP BY". Isn't the most common use going to be that the client (letor or whate...

...:) I've just opened a pull request for the CJK tokenizer: https://github.com/xapian/xapian/pull/114 > I wonder if we can arrange suitable defaults to use your > implementation with the older API, and come up with a newer API that > allows a SnippetGenerator class to be used from the MSet. The FastMail snippet generator has been written when MSet didn't create snippets. I'll first compare both implementations to see if there is a good reason for them to coexist, or might just as well merge any additional features into MSet. > A good start would certainly be rebasing aga...

Hello I defined a weighting scheme to simulate a king of "euclidean" distance. To test it, i used a database with 1000 documents. If I run : enquire.set_weighting_scheme(MyWeight()); Xapian::MSet matches = enquire.get_mset(0, 1000); I have a correct list of results. But if I run Xapian::MSet matches = enquire.get_mset(0, 10); I don't have the top-10 results. If I run Xapian::MSet matches = enquire.get_mset(0, 20); I d'ont have the top-20 results and it's even not the same order...

Hi folks, I spotted an HTML escaping bug in Xapian::MSet::snippet() while working on the code. This issue has been assigned CVE-2018-0499 (though currently there's no useful information on cve.mitre.org for it). I've added a wiki page for it here: https://trac.xapian.org/wiki/SecurityFixes/2018-07-02 The intended behaviour is that the selecte...

...glass. > > > > The reverse order (ENQ_ASCENDING) is really fast - about 0.0001 seconds. > > This is because in that case we can just stop once we've found 200 > > matches. > > So that sounds like it's O(1) and independent of how many > documents are in the mset? It won't be independent of the number of documents in the MSet - something needs to be done for each document added to the MSet, so clearly it'll be at least O(n) in that, but if n is fixed at 200 then the O() complexity isn't relevant. It'll be fairly independent of the total nu...

...ASH(call_info,exten,${dest})}); the syntax doesn't seem to be happy: -- Executing [s at appel_deb:8] Set("SIP/maverick-00000000", "HASH(info)=,101,maverick,0,0,max,0,0,123456,123654") in new stack [Jan 9 22:07:25] WARNING[27801]: pbx.c:9107 pbx_builtin_setvar_multiple: MSet: ignoring entry '101' with no '=' (in s at appel_deb:8 [Jan 9 22:07:25] WARNING[27801]: pbx.c:9107 pbx_builtin_setvar_multiple: MSet: ignoring entry 'maverick' with no '=' (in s at appel_deb:8 [Jan 9 22:07:25] WARNING[27801]: pbx.c:9107 pbx_builtin_setvar_multiple:...

...o parts of functions: the training part and the ranking part. I?ll use SVMRanker as an example. The training part basically uses the libsvm library and training data to train a model, then save the model file. The ranking part will calculate score for each document in searching results (MSet) by using the trained model file. My question is for each of our three rankers: 1) SVMRanker 2) ListMLE 3) ListNet, do we need three different types of training part? (The ranking part for each of those have the same form I think) I?m not sure the parameters for these three different ra...

...1 seconds for git > master with glass. > > The reverse order (ENQ_ASCENDING) is really fast - about 0.0001 seconds. > This is because in that case we can just stop once we've found 200 > matches. So that sounds like it's O(1) and independent of how many documents are in the mset? Would it be possible to teach Xapian to optimize its storage for certain queries so it can stop once it's found 200 matches? >From what I recall, SQL implementations are pretty good at that.

...ars silence. Here is the full debug output (asterisk -rvvvvv): == Using SIP RTP CoS mark 5 -- Executing [xxxxxxx at phones:1] Goto("SIP/ATA-xxxxxxxxxx-L1-024b6d88", "1xxxxxxxxxx,1") in new stack -- Goto (phones,1xxxxxxxxxx,1) -- Executing [1xxxxxxxxxx at phones:1] MSet("SIP/ATA-xxxxxxxxxx-L1-024b6d88", "oldcidnum=0") in new stack -- Executing [1xxxxxxxxxx at phones:2] MSet("SIP/ATA-xxxxxxxxxx-L1-024b6d88", "CALLERID(name)=""") in new stack -- Executing [1xxxxxxxxxx at phones:3] MSet("SIP/ATA-xxxxxxxxx...

On Mon, Nov 05, 2018 at 09:04:23PM +0000, Olly Betts wrote: > Dave Courtois wrote: > > Hi there just a little word to tell you that Xapian 1.4.6 work in > > Golang 1.9. There is some issue with the MSet ESet, the namespace part > > of the name is drop by swig. There is also some error with define > > macro in .i files, but with minor change all work perfectly. I had a look and I think that at least the macro changes are only needed because the golang branch hasn't been updated for...

Hello, is there a way to optimize sorting by certain values for queries which return a huge amount of results? For example, I just want a simple query that gives me the 200 most recent emails out of millions. The elapsed time for get_mset increases as the number of documents ($n * 2000) increases. I suppose I could store a pre-sorted set using SQLite or similar. Thanks in advance for any advice/help you can provide. -----------8<-------- #!/usr/bin/perl -w use strict; use warnings; use Search::Xapian::Document; use Search::Xap...

I'm already retrying the ->get_mset operations; but now I'm wondering where I'd hit DatabaseModifiedErrors while inside a Xapian::MSetIterator loop. I assume ->get_document is a place where it gets thrown; but once a document is retrieved, can iterating through terms in one document (using TermIterator) also throw DB modi...

On Fri, Mar 30, 2018 at 05:21:43PM +0000, Eric Wong wrote: > Hello, is there a way to optimize sorting by certain values > for queries which return a huge amount of results? [...] > $enquire->set_sort_by_value_then_relevance(0, 1); If you're just wanting the 200 newest, it'll be faster not to calculate weights, so: $enquire->set_sort_by_value(0, 1);

> > Olly Betts <olly at survex.com> wrote: > > > > > > The reverse order (ENQ_ASCENDING) is really fast - about 0.0001 seconds. > > > This is because in that case we can just stop once we've found 200 > > > matches. With a few million documents, that ENQ_ASCENDING sounds promising :) So, it looks like if I had ideal ordering, I could do

...mp;)" (??4Document at Xapian@@QAEXABV01@@Z)??????? "public: void __thiscall Xapian::ValueIterator::operator=(class Xapian::ValueIterator const &)" (??4ValueIterator at Xapian@@QAEXABV01@@Z) ???? libnet.lib(serialise.obj) : error LNK2019: ????????? "public: __thiscall Xapian::MSet::~MSet(void)" (??1MSet at Xapian@@QAE at XZ)??????? "public: __thiscall Xapian::MSetIterator::~MSetIterator(void)" (??1MSetIterator at Xapian@@QAE at XZ) ???? libnet.lib(serialise.obj) : error LNK2019: ????????? "public: char const * __thiscall Xapian::Error::get_error_string(vo...