similar to: New Idea on Ranking in IR

Dear Sir, I am Pankaj Singhal from Jaipur, India. I am very much interested and strongly looking forward in getting involved in this project Learning-to-Rank. My previous experience in this field is good. Last semester I did a similar job of ranking the URLs of the given huge dataset based on their attribute values. The dataset consisted hundreds of thousands of URLs and each url

For unsupervised approaches like BM25 this approach works well but letor does not need special weighting for title in this form as it itself assigns weights to title features separately. But I see your concern it would be a problem when BM25 is used on the index with this setup. Hence its preferable to take a note of this uplift in title weight for xapian-letor and normalize it everywhere

Hi Parth, I implemented DFR algorithms in Xapian as a part of GSOC last year under the mentorship of Olly. This year, I want to work on analyzing and optimizing the performance of the DFR algorithms and comparing them with BM25.I also want to work on profiling the query expansion schemes and test the relevance(precision and recall) / speed(time taken) of the

Hello, I am Mohiuddin Abdul Qader, final year student from dept of CSE in Bangladesh University of Engineering & Technology(BUET). My major was artificial intelligence & i finished my course on Machine Learning and Pattern Recognition this year. I am very keen to contribute in open source community. I have just completed my thesis on 'Location Based Structured Web Search'. For the

Hi, Before starting my proposal, I wanted to know what is the expected output of Letor module. Is it for transfer learning (i.e you learn from one dataset and leverage it to predict the rankings of other dataset) or is it for supervised learning? For instance - Xapian currently powers the Gmane search which is by default based on BM25 weighting scheme and now suppose we want to use LETOR to rank

Hi, I am Abhishek Currently Xapian::Weight follows BM25 scheme, many models such as the Divergence from Randomness (DfR) family of models, Unigram Language Model and the Bi-gram Language Model implemented two years ago in GSoc 2012 yet not merged to the master. The new weighing schemes or improvement in implementing the previous models to change the default scheme of BM25 from SMART with

Hello, I would like to work with Orange as part of GSoC 2012(and continue henceforth). Apologies for joining in a bit late- i was waiting to get a proper grasp of things before discussing it here. Currently I am a Masters students in Mathematics with my bachelors in Computer Science[integrated dual degree]. Over the last year and a half, I have worked on a few ML projects and have a couple of

Hello All, This time we are working on a weighting scheme "Learning to Rank" which involves machine learning and its a supervised ranking scheme unlike unsupervised schemes like BM25 under GSoC project. This mail intends to discuss the framework of the Learning to Rank in Xapian as a whole. I have thought of the following framework, pour in your insights or issues for the same. This is

Hi James, > We probably don't want them committed in git where they're evaluation > runs (because we can recreate them); a gist might be more appropriate. Sorry, I have moved results files over to gist for each individual weighting scheme. Link: https://gist.github.com/ivmarkp/secret > I can't tell, but are some of those files from FIRE? If so, they > shouldn't be

Hi everyone, I am thinking of working on the following ideas for my GSOC proposal based on my discussions with Olly and my own understanding. Rather than focusing on an entire perftest module, I have decided to focus on implementing performance tests for weighting schemes based on a wikipedia dump and in addition to that, build a framework to measure the

Tf-idf is most used used weighting scheme is easy to understand and has been used in other frameworks like lucene and many other places. okapi bm25(implemented in xapian) is theoretically better/improved measure than tf-idf and i am looking into various other weighting scheme which are there in xapian or can be implemented like TF-ICF(term frequecy inverse corpus frequency),TF-RF(term

Hello James, Parth, Following our discussion on IRC and on code review, the way FeatureVector class works needs some discussion. Presently, the FeatureVector class is defined as follows, with a fixed number of feature count (19): class FeatureVector::Internal : public Xapian::Internal::intrusive_base{ friend class FeatureVector; double label; double score;

Hi Olly, Wouldn't setting the weight of terms in title back to normal (e.g. 5 to 1) by below line, automatically adjust the wdfs and field lengths? indexer.index_text(title, 5, "S"); -> indexer.index_text(title, 1, "S"); if it does not then we should include that part in the patch too. I like to create a patch for xapian-letor for resolving common code of xapian.

Hello all, I am very sorry I did not include xapian-devel mailing list in my previous mail. Thanks for responding my mail. Mohd Azeem NIT UK ________________________________ From: Olly Betts <olly at survex.com> To: Mohd Azeem <azeem201001 at yahoo.in> Cc: Parth Gupta <parthg.88 at gmail.com> Sent: Saturday, 31 March 2012 11:40 AM Subject: Re: GSoC, Xapian Project Weighting

Hello, I am Mudit Raj Gupta, fourth year student of M.S. (Hons.) Chemistry and B.E. (Hons.) Electronics and Instrumentation at BITS-Pilani ( http://www.bits-pilani.ac.in/). I am interested in *Machine Learning and Computaional Intelligence*. I have an interest in implementing various existing algorithms and developing modified/new algorithms related to machine learning and computational

As Olly has already pointed out the 2012 branch is not merged. I think there are some compilation errors in the branch. The code in branch is better refactored. The Ranker and FeatureManager classes are well defined and implemented. Parth. On Wed, Sep 25, 2013 at 9:02 AM, Olly Betts <olly at survex.com> wrote: > On Tue, Sep 24, 2013 at 08:34:10PM +0800, jiangwen jiang wrote: >

On Mon, Aug 26, 2013 at 09:41:07AM +0800, jiangwen jiang wrote: > > For now, using weighting schemes which don't use document length is > > probably the simplest answer. > > There's tf-idf weighting scheme on svn master, is it suitable for lucene > backend? Yes - TfIdfWeight doesn't ever use the document length (at least with the normalisations currently

Hello, I've been working on the KMeans clustering algorithm recently and since the past week, I have been stuck on a problem which I'm not able to find a solution to. Since we are representing documents as Tf-idf vectors, they are really sparse vectors (a usual corpus can have around 5000 terms). So it gets really difficult to represent these sparse vectors in a way that would be

HI, Parth: Thanks for your reply. There's xapian-letor folder in current master trunk, is this code merged in 2011? the letor code of 2012 is not merged to the trunk? Regards 2013/9/23 Parth Gupta <pargup8 at gmail.com> > Dear Jiangwen Jiang, > > The current state of the project can be reached from the repo: > http://trac.xapian.org/wiki/GSoC2012/LTR > > I would

Hi, guys: I found this project idea http://trac.xapian.org/wiki/GSoCProjectIdeas#Project:LearningtoRank If it is need to be improved, I will try to handle it thanks Regards -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.xapian.org/pipermail/xapian-devel/attachments/20130923/d0ced0d6/attachment-0002.html>