similar to: Fwd: Re: what is the fastest way to fetch results which are sorted by timestamp ?

what is the fastest way to fetch results which are sorted by timestamp ? i want to use xapian as my search engine , use add_boolean_term(something) and add_value(0,sortable_serialise(get_timestamp())) to a doc. search through enquire.set_weighting_scheme(xapian.BoolWeight()) and enquire.set_sort_by_value(0,True) to ensure that the results are sorted by the timestamp. This method is ok , but

Hello, I wonder if there is a way to cause Xapian to order a result set purely by docid. In other words, once the result set has been determined, I'd like the results to be returned to me ordered by their docid, as opposed to by their match relevance. The problem at hand is that I'm building a search engine for a mailing list and I would like to return matches sorted by date; ordering by

Olly Betts <olly at survex.com> wrote: > On Fri, Mar 30, 2018 at 05:21:43PM +0000, Eric Wong wrote: > > Hello, is there a way to optimize sorting by certain values > > for queries which return a huge amount of results? > [...] > > $enquire->set_sort_by_value_then_relevance(0, 1); > > If you're just wanting the 200 newest, it'll be faster not to

Hello, is there a way to optimize sorting by certain values for queries which return a huge amount of results? For example, I just want a simple query that gives me the 200 most recent emails out of millions. The elapsed time for get_mset increases as the number of documents ($n * 2000) increases. I suppose I could store a pre-sorted set using SQLite or similar. Thanks in advance for any

i have 300 millions records and my search file like this , i want the newest 10 results that match my query , so i use boolean search and "enquire.set_docid_order(enquire.DESCENDING)" , but this method seems a little slow . when i remove "enquire.set_docid_order(enquire.DESCENDING)" it run much faster . how can i fetch the newest 10 results as fast as possible? search.py

On Fri, Mar 30, 2018 at 05:21:43PM +0000, Eric Wong wrote: > Hello, is there a way to optimize sorting by certain values > for queries which return a huge amount of results? [...] > $enquire->set_sort_by_value_then_relevance(0, 1); If you're just wanting the 200 newest, it'll be faster not to calculate weights, so: $enquire->set_sort_by_value(0, 1);

On Sat, Mar 31, 2018 at 12:58:19AM +0000, Eric Wong wrote: > Olly Betts <olly at survex.com> wrote: > > If you're just wanting the 200 newest, it'll be faster not to calculate > > weights, so: > > > > $enquire->set_sort_by_value(0, 1); > > $enquire->set_weighting_scheme(new Xapian::BoolWeight()); > > > > For me, this drops the time

When going through the xapian bindings yesterday, I noticed that several of the methods were not wrapped for Ruby because they were deprecated at the time the ruby bindings were created. I filed a bug (#126) saying that they should be removed entirely, which led to the suggestion from Olly that it would be good to make a semi-formal policy about deprecating features. I've written such a

Using MultiValueSorter, I can sort by key1, key2, relevance; or relevance, key1, key2. But AFAIK, I can't sort by key1, relevance, key2. Unless I spool out the entire result set or write some C++. I wonder if we need a new 'sort by' function that accepts any combination of keys and relevance in any order? The function would make it's own optimisations (ie is relevance first or

I got some performance issue for document.get_data() and enquire.get_mset(). It costs 35 seconds for matches = enquire.get_mset(0,200), and 3 seconds for iterating all doc in matches to get_data. Is't normal? My index contains 30millions documents. I use python binding to operate xapian. Bellow it's my index structure # value: 0:date, 1:site # data: json message which contains: author,

Hi, all: This is the basic implementation of tf-idf scheme (basic scheme used in SMART) that can be used in the Xapian. It might still need some futher revision, but I believe it works anyway.:) I modified the weight.h to define a subclass Tf_idfWeight and add a new file tf_idf.cc in ../weight in the repo, to implement Tf_idfWeight. Here is the git diff patch: https://gist.github.com/2422049

On Sun, Apr 09, 2017 at 11:34:07PM +0530, Vivek Pal wrote: > > Each scheme already has a human-readable name, and Xapian::Registry > > can map that to an "examplar" object of the right type, so we > > could take a string like "bm25 1 0.8", see the first word is "bm25" > > and get a BM25Weight object, then call parse_params("1 0.8") on

> Each scheme already has a human-readable name, and Xapian::Registry > can map that to an "examplar" object of the right type, so we > could take a string like "bm25 1 0.8", see the first word is "bm25" > and get a BM25Weight object, then call parse_params("1 0.8") on it to > create the correct Weight object (broadly similar to how

On Mon, Apr 10, 2017 at 11:47:36PM +0530, Vivek Pal wrote: > > No, use Xapian::Registry to find the weighting scheme from the name > > like how Weight::unserialise() does (otherwise every caller would need > > code similar to that above). > > Okay, I looked into Xapian::Registry and it seems you are referring to using > the get_weighting_scheme method? (which expects a

Hi guys In our implementation of Xapian for one of our sites, we index the title, subtitle, summary and table of contents of around 200,000 products on ReportBuyer.com. When we create each Xapian doc to index this information, we apply a weighting to each of these 'fields' and add these to the doc using index_text with the second parameter passing in a weighting. I've been asked if

On Sat, Apr 08, 2017 at 09:11:22PM +0100, James Aylett wrote: > On 8 Apr 2017, at 19:15, Vivek Pal <vivekpal.dtu at gmail.com> wrote: > > >> and the details of which weighting schemes were available in which version > >> isn't a key part of the $set command itself. > > > > Do you suggest dropping that piece of information out? Since the reason behind

Hi guys, I resently used xapian to sort some documents by distance between 2 points. I implemented a MatchDecider which work well. I now tried to implement a Weight scheme to put my document in ascending order depending on the distance... My information to calcul distance is in values in the document. How I can access document values from Weight to be able to add some sum_extra weight ??

Hello I defined a weighting scheme to simulate a king of "euclidean" distance. To test it, i used a database with 1000 documents. If I run : enquire.set_weighting_scheme(MyWeight()); Xapian::MSet matches = enquire.get_mset(0, 1000); I have a correct list of results. But if I run Xapian::MSet matches = enquire.get_mset(0, 10); I don't have the top-10 results. If I run Xapian::MSet

Olly, the process size does stay constant with the results from one query set, but running other queries will cause it to grow (once). Is it possible that this is a bug with the SWIG python bindings? On Fri, Jul 5, 2013 at 6:18 AM, Olly Betts <olly at survex.com> wrote: > On Wed, Jul 03, 2013 at 03:59:21PM -0400, Jeff Rand wrote: > > Run the following code and monitor the

Hi everyone, I'm Akshay, an Information Science undergrad from Bangalore. I'm interested in Information Retrieval and I'd like to contribute to Xapian as a part of GSoC and later to feed my interests. I liked the idea of adding more weighting schemes (Project #2). I did a project last semester on Document Retrieval on Hadoop using TF-IDF and Cosine Similarity (the query had to be a