similar to: Starting work on Perf Test Module

Hi Parth, I implemented DFR algorithms in Xapian as a part of GSOC last year under the mentorship of Olly. This year, I want to work on analyzing and optimizing the performance of the DFR algorithms and comparing them with BM25.I also want to work on profiling the query expansion schemes and test the relevance(precision and recall) / speed(time taken) of the

Hey guys,I am a 3rd year Computer Science undergrad student.I a extremely interested in contributing code to the XAPIAN project. The work you people do sounds extremely fascinating and interesting.Can someone just give me a brief overview of how to proceed ?. I Can code in C,C++ and Python and have experience in Natural Lanuage Processing.Am also quite comfortable with NLTK and using Wordnet.Am

Hey guys,I have added a python indexer example to the SampleCode page of our wiki.Please do have a look.The code can also be found here :- https://github.com/aarshkshah1992/xapian/blob/efcf443527b74326119bbc0935fc41a002ce60db/xapian-bindings/python/docs/examples/simpleindexgrep.py/ Thanks :) -Regards -Aarsh -------------- next part -------------- An HTML attachment was scrubbed... URL:

Hello Dan and Olly, this is the code for the base class for query expansion that I have written. The code will not compile as I have written only the base class until now. Have yet to use it. Please do tell me what you think of the base class and what changes you suggest I should make before I move forward with the project. https://github.com/xapian/xapian/pull/23 -Regards -Aarsh --------------

Hi, While searching for a project which matches my interest andskill level, I found this project named Matcher Optimization. This project is really challenging and excting from my view point and I would like to be a part of this project. Optimization techniques metioned in the reference links provided will take some time for me to have a good understanding about them. But I am trying to get my

Hello guys.I just read up about tf-idf schemes and want to implement it in Xapian (with some frequently used normalizations) as it will also give me a good hang of implementing a weighting scheme before I start working on implementing DFR schemes. I read the following as references and I think Ive understood it well and can write the hack :- 1.)

Hello, I wanted to decide the dataset that should be used for Letor stabilisation project. I think 2009 INEX Wikipedia Collection <http://www.mpi-inf.mpg.de/departments/databases-and-information-systems/software/inex/> should work fine. It's a collection of 2,666,190 XML articles, 115 topics <http://inex.mmci.uni-saarland.de/protected/adhoc/2009-topics.zip>, 50,275 qrel

Thanks for your reply! For the third question: In https://inex.mmci.uni-saarland.de/data/documentcollection.jsp, I can find inex2010-article.qrels in 2010 assessment, but can?t find query files. Could you send me the link? I have registered on INEX website. And I also need to download ``INEX 2009 collection without annotation tags: (unofficial)`` on

Ah. If FIRE doesn't have something that can show this suitably, then > maybe Parth can advise on access to TREC, as I know he's used some of > them in the past. > ?I can say FIRE is also a reliable source but INEX/TREC are better. INEX can give you free access and TREC is not freely available. I had used INEX for xapian in the past and some details are here:

Hi Parth, I?ve implemented SVMRanker class and also sorted out most of current Letor APIs. Now I?m trying to use INEX dataset to verify my implement. But I stuck in the indexing part. You said in the documentation that we have to add prefix when indexing. Also I notice that you set some metadata in omindex.cc of your version. But the omindex.cc has changed since 2011. I think that?s why my result

Hi, I've read the code of letor module. And I have some questions about it: 1. In https://github.com/rishabhmehrotra/xapian/blob/master/xapian-letor/letor_internal.cc#L299, there is a write_to_file method, which save RankList into ?train.txt?. But the format for ?train.txt? is different from the one mentioned in http://trac.xapian.org/wiki/GSoC2011/LTR/Notes#QueryLevelNorm. And in

I was adding the calculations for a lower bound to get_sumpart() (DLH has no term independent component) when I realized that the same lower bound will be calculated for each term-docment pair that get_sumpart is called pair which basically reduces efficiency. How do I calculate the lower bound for a term only once and then use it ? -Regards -Aarsh On Fri, Jun 21, 2013 at 4:41 PM, Olly Betts

Hi, Sorry for the long hiatus. I was caught with some other deadlines. Could some one tell me some specific expectations from Letor for GSOC 2015? I am currently trying out some learning algorithm implementations but need some help with it. Can Parth, James and others related to LETOR please tell me the IRC availability for the next few days. Regards Karthik On Wed, Oct 22, 2014 at 3:39 AM,

Hi, Before starting my proposal, I wanted to know what is the expected output of Letor module. Is it for transfer learning (i.e you learn from one dataset and leverage it to predict the rankings of other dataset) or is it for supervised learning? For instance - Xapian currently powers the Gmane search which is by default based on BM25 weighting scheme and now suppose we want to use LETOR to rank

Hi, I?m trying to use Omega to index INEX dataset for Letor. But omindex told me these xml files are unknown. Olly told me I could tell omindex to handle them as HTML. (Thanks Olly :) ) Is it appropriate? Parth, could you give me some suggestions? Thank you! Jiarong Wei

Hello guys. I am currently working on the DLH weighting scheme .The formula for DLH is very complex and it ends up giving negative weights to some documents because of the formula.Due to this,inspite of having occurence/occurences of the keyword, the documents with negative weights don't show up in the results at all. Please can I get some help on how to deal with this ? Or should I just leave

Hi, I am unable to build the letor module. I am generating the configure file using autoconf. The configure file generated is throwing the error ./configure: line 2057: syntax error near unexpected token `1.10.1' ./configure: line 2057: `AM_INIT_AUTOMAKE(1.10.1 -Wportability tar-ustar)` I am not too sure what to do with this. Need help with this. Thank You Regards Karthik On Mon, Sep 29,

Hey guys hi.I am trying to understand how Xapian works .I read the Theoretical Background to Xapian doc and the report by Salton and Jones.I still cant seem to understand how Eset works How exactly does Xapian add terms to expand a query ? Assuming we have a list of the k most important terms, how do we decide which term to add to the query and will be in context with the query ? And to decide r

Hi everyone, I am thinking of working on the following ideas for my GSOC proposal based on my discussions with Olly and my own understanding. Rather than focusing on an entire perftest module, I have decided to focus on implementing performance tests for weighting schemes based on a wikipedia dump and in addition to that, build a framework to measure the

Hello Guys. I have updated the code,tests,documentation,makefile entries and the registry entry of the* *TfIdf patch as per the feedback.Please do let me know if any additional changes are required before the patch can be merged, -Regards -Aarsh On Sun, Mar 3, 2013 at 2:50 PM, aarsh shah <aarshkshah1992 at gmail.com> wrote: > Hello guys.I have sent a pull request for the code and