similar to: xapian-letor refactoring and adding tests

1.Could you explain why are these libraries included in all the xapian-letor headers? #include<xapian/intrusive_ptr.h> #include<xapian/types.h> #include<xapian/visibility.h> Or just provide me with the documentation of these header. I looked into these header files but couldn't anything substantial. 2.

Hi, I have several questions regarding the letor module,I looked at the framework of learning to rank in xapian http://rishabhmehrotra.com/gsoc/17.png, I am a little confused. Why using deep learning to find unsupervised features in test data? Since in my understanding, learning to rank model usually learn features from the training data then apply the model to the test data? Why test set and

> > > > The approach I was thinking would look something like this: > > * instead of Features, which is really a namespace implemented as a > class, we separate out the calculation of the different features > into distinct subclasses of Feature, whose only job is to calculate > a single feature. Currently the FeatureManager calls these (via >

Hi all, Thank you for giving me the opportunity to work with Xapian :) I am Jiarong Wei, a third year undergraduate student in Zhejiang University, China. In GSoC 2014, I will work on Letor module with Hanxiao Sun. Here are some questions I encountered these days, 1. In letor.cc, we have two parts of functions: the training part and the ranking part. I?ll use SVMRanker as an example. The

Hi, I've read the code of letor module. And I have some questions about it: 1. In https://github.com/rishabhmehrotra/xapian/blob/master/xapian-letor/letor_internal.cc#L299, there is a write_to_file method, which save RankList into ?train.txt?. But the format for ?train.txt? is different from the one mentioned in http://trac.xapian.org/wiki/GSoC2011/LTR/Notes#QueryLevelNorm. And in

Hello Rishabh, The attached diff file should help you to refactor the existing letor code and plug in your code easily. I have defined the header files with the necessary flow. I have also specified the ranker class, which should be implemented by a new LTR models, ListNet and ListMLE in your case. The evaluation file should be used for validating the performance of the algorithm in the training

Thanks for your reply! For the third question: In https://inex.mmci.uni-saarland.de/data/documentcollection.jsp, I can find inex2010-article.qrels in 2010 assessment, but can?t find query files. Could you send me the link? I have registered on INEX website. And I also need to download ``INEX 2009 collection without annotation tags: (unofficial)`` on

Hello, I'm Ayush from New Delhi, India. I am interested in Letor Stabilisation project for GSoC. I have a good background in machine learning. Sorry for getting in so late, university exams were holding me back. I'll try to cover as much as I can in the coming week. I am following the plan of attack suggested on the project page. Following are the things that I have completed: 1.

Hi, Before starting my proposal, I wanted to know what is the expected output of Letor module. Is it for transfer learning (i.e you learn from one dataset and leverage it to predict the rankings of other dataset) or is it for supervised learning? For instance - Xapian currently powers the Gmane search which is by default based on BM25 weighting scheme and now suppose we want to use LETOR to rank

Hi everyone, My name is Shreedhar Pawar. I have already introduced myself on Xapian-discuss... I feel that the Xapian Search/Letor Algorithm, can speed up using Parallel computing. Techniques like Map-reduce, 'compact n split', radix sort, scan, parallel hashing n much more can be used to speed up the learning algorithms as well as the search... support vector machines in the Letor

Hello everyone, I have completed introducing some code from v-hasu's branch into mine, mainly for Features, FeatureVector and FeatureManager classes. I have pushed the changes to https://github.com/ayshtmr/xapian/tree/letor-update. I am now proceeding to write unit tests for feature modules. There are a few things that I wanted to clarify: 1. I have introduced a lot of code in a single

Hi all, While building from git with xapian-letor not ignored in bootstrap, I am getting the following make error: In function `main': /home/ayush/Desktop/xapian/xapian-letor/bin/xapian-letor-update.cc:98: undefined reference to `Xapian::Internal::str(unsigned int)' /home/ayush/Desktop/xapian/xapian-letor/bin/xapian-letor-update.cc:99: undefined reference to

Hi Parth, I?ve implemented SVMRanker class and also sorted out most of current Letor APIs. Now I?m trying to use INEX dataset to verify my implement. But I stuck in the indexing part. You said in the documentation that we have to add prefix when indexing. Also I notice that you set some metadata in omindex.cc of your version. But the omindex.cc has changed since 2011. I think that?s why my result

Hello, I wanted to decide the dataset that should be used for Letor stabilisation project. I think 2009 INEX Wikipedia Collection <http://www.mpi-inf.mpg.de/departments/databases-and-information-systems/software/inex/> should work fine. It's a collection of 2,666,190 XML articles, 115 topics <http://inex.mmci.uni-saarland.de/protected/adhoc/2009-topics.zip>, 50,275 qrel

> > > I'd prefer to avoid adding things to the public API that don't get > used by end users. However because LTR is outside the Xapian build > tree, we can't easily give it privileged access to Xapian internals. > Sorry for a delayed response. The way I was thinking of performing reranking with updated weights was to add a class MSetRanker (basically containing a

Hi I am on Debian verson 7.7, GCC 4.7.2 and make 3.8.1. I cloned the code : git clone https://github.com/v-hasu/xapian git checkout gsoc2014-evaluation rm xapian-letor/nobootstrap ./bootstrap ./configure --no-documentation make CXXFLAGS='-w' and I get a different set of errors --- ranker.cc: In member function 'std::string Xapian::Ranker::get_cwd()': ranker.cc:80:36: error:

Hi Rishabh, I think its better not to expose RankiList to Letor.h and make it better user friendly. So my suggestion is to convert RankList to the following statement in this method. std::map<Xapian::docid, double> letor_score(const Xapian::MSet & mset); So just convert the RankList in std::map<Xapian::docid, double> format in the methods where you need to return. Parth. On

I use Ubuntu12.04, GNU Make 3.81 and I am not sure if I use some special configure options. The commands I use after clone the repo are as follows: cd xapian ./bootstarp ./configure make cd xapian-letor aclocal -I ../xapian-core/m4 libtoolize --force autoconf autoheader automake --add-missing ./configure make Is this normal? 2014-12-07 22:37 GMT+08:00 James Aylett <james-xapian at

Hi, Wanted to know if other normalization techniques like normalization by standard-deviation have been tried to normalize the Feature-list in Letor. Regards, Ayush Pandey.

Hi~ Manu I was glad to see you were interested in Xapian, especially the learning to rank project. There are two branch in my repo. The one named gsoc2014 is followed the thought of Parth's work in which we not assign the test set for each ranking and in another one gsoc2014-evaluation we assign the training set and test set explicitly so that we can get the evaluation score of each ranking