similar to: Sent a pull request for the Tf-Idf Weighting scheme

Hello guys.I have sent a pull request for the code and tests of the Tf-Idf weighting scheme. Please do let me know if any changes are required.Meanwhile,Ill begin working on implementing normalizations which require additional statistics and on the DFR schemes. https://github.com/xapian/xapian/pull/6 On Tue, Feb 26, 2013 at 5:30 PM, <xapian-devel-request at lists.xapian.org>wrote: >

Hello Guys. I have updated the code,tests,documentation,makefile entries and the registry entry of the* *TfIdf patch as per the feedback.Please do let me know if any additional changes are required before the patch can be merged, -Regards -Aarsh On Sun, Mar 3, 2013 at 2:50 PM, aarsh shah <aarshkshah1992 at gmail.com> wrote: > Hello guys.I have sent a pull request for the code and

Hello guys.I just read up about tf-idf schemes and want to implement it in Xapian (with some frequently used normalizations) as it will also give me a good hang of implementing a weighting scheme before I start working on implementing DFR schemes. I read the following as references and I think Ive understood it well and can write the hack :- 1.)

Hi, all: This is the basic implementation of tf-idf scheme (basic scheme used in SMART) that can be used in the Xapian. It might still need some futher revision, but I believe it works anyway.:) I modified the weight.h to define a subclass Tf_idfWeight and add a new file tf_idf.cc in ../weight in the repo, to implement Tf_idfWeight. Here is the git diff patch: https://gist.github.com/2422049

Hello guys,I have added code for tfidf to the weight.cc file in omega/ . Here is the patch : - https://github.com/aarshkshah1992/xapian/commit/5ff41a15f574e6780cc61e67e7f3da3d97ff4ec8 It compiles well and I think it'll work well. Here's the link to the documentation file omegascript.rst where I've added tfidf.

Hello guys,I've modified the TfIdf patch as per the feedback I got on it and have added the code to the pull request. Please do have a look and let me now what you'll think. https://github.com/xapian/xapian/pull/6 Also,I read somewhere that I need to register this weighting scheme with Xapian. Please can you'll throw some light on that ? -Regards -Aarsh -------------- next part

Hello guys, :) Please do take a look at the pull request for the TfIdf patch Ive sent because I want to start working on writing DFR schemes for us and want to incorporate the feedback into making a good hack for the DFR schemes.The patch incorporates all normalizations possible with our current statistics and passed all the tests I wrote for it.Have also attached the tests with the pull request.

Hello guys, as discussed on IRC, I've added an example of a weighting scheme that requires statistics to the howto manual of our Getting Started guide.Please do let me know if any modifications are required. The pull request is here : https://github.com/jaylett/xapian-docsprint/pull/3 -Regards -Aarsh. -------------- next part -------------- An HTML attachment was scrubbed... URL:

I was adding the calculations for a lower bound to get_sumpart() (DLH has no term independent component) when I realized that the same lower bound will be calculated for each term-docment pair that get_sumpart is called pair which basically reduces efficiency. How do I calculate the lower bound for a term only once and then use it ? -Regards -Aarsh On Fri, Jun 21, 2013 at 4:41 PM, Olly Betts

Hello guys. I am currently working on the DLH weighting scheme .The formula for DLH is very complex and it ends up giving negative weights to some documents because of the formula.Due to this,inspite of having occurence/occurences of the keyword, the documents with negative weights don't show up in the results at all. Please can I get some help on how to deal with this ? Or should I just leave

Hello guys.As discussed on IRC,I have sent a pull request for a test for testing TradWeight with an Rset. On Fri, Mar 1, 2013 at 5:30 PM, <xapian-devel-request at lists.xapian.org>wrote: > Send Xapian-devel mailing list submissions to > xapian-devel at lists.xapian.org > > To subscribe or unsubscribe via the World Wide Web, visit >

Hi, Evaluation of pivoted normalization ("PPP") of tf-idf weighting scheme is also complete now. I have also evaluated the default tf-idf normalization ("ntn") and other normalizations combinations involving pivoted normalization in wdfn, idfn and wtn component as "Pxx", "xPx" and "xxP" normalization strings respectively to have a clear idea about

Hello guys. I just ran the updated tests for PL2 and they are not giving the mset order I expect.Now,the thing is, dfr's behavior is a bit hard to predict and so even if I expect a particular order ,it may give another order and still be correct.So,the only way to write correct tests for PL2 is to manually calculate the weight of the documents to decide the expected order.For that,I need to

Hello guys.I am working on implementing the PL2 weighting scheme of the DFR framework by Gianni Amati. It uses the Poisson approximation of the Binomial as the probabilistic model (P), the Laplace law of succession to calculate the after effect of sampling or the risk gain (L) and within document frequency normalization H2(2) (as proposed by Amati in his PHD thesis). The formula for w(t,d) in

Hello, I am Mohiuddin Abdul Qader, final year student from dept of CSE in Bangladesh University of Engineering & Technology(BUET). My major was artificial intelligence & i finished my course on Machine Learning and Pattern Recognition this year. I am very keen to contribute in open source community. I have just completed my thesis on 'Location Based Structured Web Search'. For the

Hi Parth, I implemented DFR algorithms in Xapian as a part of GSOC last year under the mentorship of Olly. This year, I want to work on analyzing and optimizing the performance of the DFR algorithms and comparing them with BM25.I also want to work on profiling the query expansion schemes and test the relevance(precision and recall) / speed(time taken) of the

Hello, I am beginning work on the perf test module. The initial steps that I aim to accomplish are :- -> Download the wikipedia dumps for multiple languages . -> Write python scripts to tokenize the dump (will probably use something like nltk which has powerful inbuilt tokenizers) -> Discuss and finalize the design of the search and query expansion perf tests as I want to complete them

I was not sharing it on maling list because i thought that someone can use all ideas i proposed in their GSOC proposal. Surely i will contribute to xapian project. sorry if that was against the rules The algorithm is not developed by me but after having much research on various clustering techniques. I found that there is a new algorithm called CLUBS(Clustering Using Binary Splitting) which gives

Hello Everyone, I am a second year graduate student at IIIT-Bangalore and my interest is in the field of Information Retrieval. I have successfully compiled Xapian from source and have implemented some examples. While going through the project list Weighting Schemes project is the one I was looking to contribute to. So i went through the xapian-core/weight where most of the schemes are already

Hi, While searching for a project which matches my interest andskill level, I found this project named Matcher Optimization. This project is really challenging and excting from my view point and I would like to be a part of this project. Optimization techniques metioned in the reference links provided will take some time for me to have a good understanding about them. But I am trying to get my