similar to: Query Expansion trial version ready

Hello Dan and Olly, this is the code for the base class for query expansion that I have written. The code will not compile as I have written only the base class until now. Have yet to use it. Please do tell me what you think of the base class and what changes you suggest I should make before I move forward with the project. https://github.com/xapian/xapian/pull/23 -Regards -Aarsh --------------

Hey guys ,Hi :) I am working on a hack which will allow the user to specify a weighing scheme (along with the parameters , if he does not not want to use the default values) to build the Eset (rather than using the hard coded TradWeight scheme with default k=1 ) as Olly had suggested that we can probably get better terms (a more relevant Eset) for query expansion if we use say something

Hello Guys. I have updated the code,tests,documentation,makefile entries and the registry entry of the* *TfIdf patch as per the feedback.Please do let me know if any additional changes are required before the patch can be merged, -Regards -Aarsh On Sun, Mar 3, 2013 at 2:50 PM, aarsh shah <aarshkshah1992 at gmail.com> wrote: > Hello guys.I have sent a pull request for the code and

Hello, I am beginning work on the perf test module. The initial steps that I aim to accomplish are :- -> Download the wikipedia dumps for multiple languages . -> Write python scripts to tokenize the dump (will probably use something like nltk which has powerful inbuilt tokenizers) -> Discuss and finalize the design of the search and query expansion perf tests as I want to complete them

Hey guys,I have added a python indexer example to the SampleCode page of our wiki.Please do have a look.The code can also be found here :- https://github.com/aarshkshah1992/xapian/blob/efcf443527b74326119bbc0935fc41a002ce60db/xapian-bindings/python/docs/examples/simpleindexgrep.py/ Thanks :) -Regards -Aarsh -------------- next part -------------- An HTML attachment was scrubbed... URL:

Hello guys. I just ran the updated tests for PL2 and they are not giving the mset order I expect.Now,the thing is, dfr's behavior is a bit hard to predict and so even if I expect a particular order ,it may give another order and still be correct.So,the only way to write correct tests for PL2 is to manually calculate the weight of the documents to decide the expected order.For that,I need to

Hello guys.I have sent a pull request for the code and tests of the Tf-Idf weighting scheme. Please do let me know if any changes are required.Meanwhile,Ill begin working on implementing normalizations which require additional statistics and on the DFR schemes. https://github.com/xapian/xapian/pull/6 On Tue, Feb 26, 2013 at 5:30 PM, <xapian-devel-request at lists.xapian.org>wrote: >

Hello guys.As discussed on IRC,I have sent a pull request for a test for testing TradWeight with an Rset. On Fri, Mar 1, 2013 at 5:30 PM, <xapian-devel-request at lists.xapian.org>wrote: > Send Xapian-devel mailing list submissions to > xapian-devel at lists.xapian.org > > To subscribe or unsubscribe via the World Wide Web, visit >

Hi Parth, I implemented DFR algorithms in Xapian as a part of GSOC last year under the mentorship of Olly. This year, I want to work on analyzing and optimizing the performance of the DFR algorithms and comparing them with BM25.I also want to work on profiling the query expansion schemes and test the relevance(precision and recall) / speed(time taken) of the

Hello guys,I have added code for tfidf to the weight.cc file in omega/ . Here is the patch : - https://github.com/aarshkshah1992/xapian/commit/5ff41a15f574e6780cc61e67e7f3da3d97ff4ec8 It compiles well and I think it'll work well. Here's the link to the documentation file omegascript.rst where I've added tfidf.

Hey guys,I am a 3rd year Computer Science undergrad student.I a extremely interested in contributing code to the XAPIAN project. The work you people do sounds extremely fascinating and interesting.Can someone just give me a brief overview of how to proceed ?. I Can code in C,C++ and Python and have experience in Natural Lanuage Processing.Am also quite comfortable with NLTK and using Wordnet.Am

In matcher/expandweight.cc we have: OmExpandBits operator+(const OmExpandBits &bits1, const OmExpandBits &bits2) { OmExpandBits sum(bits1); sum.multiplier += bits2.multiplier; sum.rtermfreq += bits2.rtermfreq; // FIXME - try to share this information rather than pick half of it if (bits2.dbsize > sum.dbsize) { DEBUGLINE(WTCALC,

Hello guys, as discussed on IRC, I have written some code for better parsing of BM25 parameters in Omega. If no parameters are specified ,it defaults all of them. However, if there some are specified and some are not or if the invalid values are given for any of them,it throws an error. https://github.com/aarshkshah1992/xapian/commit/ac0a11f5d8ff975fad1e96e63764eab9b04dfcfb -Regards -Aarsh

Hello guys.I just read up about tf-idf schemes and want to implement it in Xapian (with some frequently used normalizations) as it will also give me a good hang of implementing a weighting scheme before I start working on implementing DFR schemes. I read the following as references and I think Ive understood it well and can write the hack :- 1.)

I was adding the calculations for a lower bound to get_sumpart() (DLH has no term independent component) when I realized that the same lower bound will be calculated for each term-docment pair that get_sumpart is called pair which basically reduces efficiency. How do I calculate the lower bound for a term only once and then use it ? -Regards -Aarsh On Fri, Jun 21, 2013 at 4:41 PM, Olly Betts

hello Can anyone tell me about the concept of query expansion? or else point to resources wich explain wat exactly it is? in xapian, query expansion facility is available n i used it too. but didnt understand its significance. thanks doubtfire40008 -------------- next part -------------- An HTML attachment was scrubbed... URL:

Hi olly, I rewrote the test for PL2 after adding code to deal with negative weights. It passes on all backends other than inmemory . I see a different value of weight for inmemory backend. The code to calculate the lower bound is implemented in init().Please can I get some help with this ? -Regards -Aarsh -------------- next part -------------- An HTML attachment was scrubbed... URL:

Hi, While searching for a project which matches my interest andskill level, I found this project named Matcher Optimization. This project is really challenging and excting from my view point and I would like to be a part of this project. Optimization techniques metioned in the reference links provided will take some time for me to have a good understanding about them. But I am trying to get my

Hello guys. I am currently working on the DLH weighting scheme .The formula for DLH is very complex and it ends up giving negative weights to some documents because of the formula.Due to this,inspite of having occurence/occurences of the keyword, the documents with negative weights don't show up in the results at all. Please can I get some help on how to deal with this ? Or should I just leave

Hey guys hi.I am trying to understand how Xapian works .I read the Theoretical Background to Xapian doc and the report by Salton and Jones.I still cant seem to understand how Eset works How exactly does Xapian add terms to expand a query ? Assuming we have a list of the k most important terms, how do we decide which term to add to the query and will be in context with the query ? And to decide r