Ah. If FIRE doesn't have something that can show this suitably, then> maybe Parth can advise on access to TREC, as I know he's used some of > them in the past. >?I can say FIRE is also a reliable source but INEX/TREC are better. INEX can give you free access and TREC is not freely available. I had used INEX for xapian in the past and some details are here: https://trac.xapian.org/wiki/GSoC2011/LTR/Notes#IREvaluationofLetorrankingscheme I roughly remember that there was a discussion with our this year GSOC student Ayush about INEX data. He had also obtained it, this would also be a good way to collaborate with him :) and try to establish a common evaluation dataset for future. Cheers Parth> > Certainly until we have something where evaluation shows an > improvement, we shouldn't change the default. It does sound like it > should be possible to find a suitable dataset to demonstrate this on, > though. > > J > > -- > James Aylett, occasional trouble-maker > xapian.org > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.xapian.org/pipermail/xapian-devel/attachments/20160728/75646a8d/attachment-0001.html>
On Thu, Jul 28, 2016 at 12:25:38PM +0530, Parth Gupta wrote:> ?I can say FIRE is also a reliable source but INEX/TREC are > better. INEX can give you free access and TREC is not freely > available. > > I roughly remember that there was a discussion with our this year > GSOC student Ayush about INEX data. He had also obtained it, this > would also be a good way to collaborate with him :) and try to > establish a common evaluation dataset for future.I'd forgotten about INEX; it doesn't seem to be running any more, but there's still a range of datasets available. It looks like the bigger datasets are likely to survive a while, although if we come to use these regularly we should contact the hosting providers so we get warning if they'll disappear. The INEX supporting software is on Google Code, which will disappear at the end of this year, so ideally someone would convert that to git and make it available longer-term. (I've grabbed a download of the subversion repository for now.) I think ideally we'd have notes on using a variety of datasets, since they all seem to cover slightly different scenarios, that would be ideal. Probably best to start with creating an 'evaluation' page on the wiki to state which datasets have been used, any notes on them, and as somewhere to drop the results for the time being. J -- James Aylett, occasional trouble-maker xapian.org
Hi,
Evaluation of pivoted normalization ("PPP") of tf-idf weighting scheme
is
also complete now. I have also evaluated the default tf-idf normalization
("ntn") and other normalizations combinations involving pivoted
normalization in wdfn, idfn and wtn component as "Pxx",
"xPx" and "xxP"
normalization strings respectively to have a clear idea about which one
does better job of retrieving relevant documents.
All results of evaluation runs can be easily accessed here:
https://gist.github.com/ivmarkp
Comparing the MAP of "PPP" with that of "ntn" normalization,
we get results
as follows:
PPP : 0.0607107
ntn : 0.109525
Clearly, the default normalization does a better job here than pivoted
normalization but since we intended to have support for pivoted
normalization in Xapian rather making a replacement of default
normalization with pivoted normalization, I think this comparison may not
come as a big surprise.
Similarly, the MAP of Ptn, nPn and ntP which represent "Pxx",
"xPx" and
"xxP" normalization strings respectively are as follows:
ntP: 0.0747668
nPn: 0.0676789
Ptn: 0.11379
Interestingly, Ptn normalization does fairly good job than all other
normalizations and the default normalization ("ntn") as well. So, I
think
it can be recommended for applications based on news corpus to definitely
use Ptn normalization if exploring options beyond default tf-idf
normalization.
As a small side note -- now I'm planning to take up additional tasks we
were looking to work on in the end but before that I was wondering if this
is the right time to complete the documentation part of BM25+, PL2+, Dir+
and Piv+ weighting schemes and also if PRs for these weighting schemes can
be merged upstream finally? Please let me know if there are any loose ends
that might need some work before PRs can be merged.
Regards,
Vivek
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xapian.org/pipermail/xapian-devel/attachments/20160807/a357e2d2/attachment.html>