Xapian devel - Jul 2016 - Weighting Schemes: Implementing Piv+ Normalization

Hi,

I have added support for Piv normalization in Tf-Idf weighting scheme as a
intermediate step to implementing the support for Piv+ normalization. All
tests pass.

But I'm running into some issues with Piv+ normalization. In the Piv+
formula  , there are two parameters (s and delta) that control the weight
assigned. I think the way I'm serialising and unserialising these
parameters has some issues which is causing a few errors,
http://pastebin.com/yCyuvF1X.  Any suggestion is appreciated!

Also, I was wondering if we want to have pivoted normalization weighting as
a separate weighting scheme in Xapian as it will make a way to add support
of different variations of pivoted normalization in the future?

Piv+ weighting formula:
https://trac.xapian.org/attachment/wiki/GSoC2016/Weighting/ProjectPlan/Piv%2B.png
)
Commit of Piv changes:
https://github.com/ivmarkp/xapian/commit/a02942fe3c3cf6cfbdd3c14685c2ff5f8a6b8b7c
Commit Piv+ changes:
https://github.com/ivmarkp/xapian/commit/e31a46e5a5ee1bdd3931de25e25722e46df056fe

Thanks,
Vivek
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xapian.org/pipermail/xapian-devel/attachments/20160727/a2395be3/attachment.html>

On Wed, Jul 27, 2016 at 11:38:58PM +0530, Vivek Pal wrote:
> But I'm running into some issues with Piv+ normalization. In the Piv+
> formula  , there are two parameters (s and delta) that control the weight
> assigned. I think the way I'm serialising and unserialising these
> parameters has some issues which is causing a few errors,
> http://pastebin.com/yCyuvF1X.  Any suggestion is appreciated!
Two of those are compile errors, suggesting you aren't pulling in the
right header file (it's in common/serialise-double.h I believe).

This one:

----------------------------------------------------------------------
weight/tfidfweight.cc: In member function 'virtual
Xapian::TfIdfWeight* Xapian::TfIdfWeight::unserialise(const string&)
const':

weight/tfidfweight.cc:87:37: error: conversion from 'const char' to
non-scalar type 'const string {aka const
std::__cxx11::basic_string<char>}' requested

     const string normals = (*(ptr)++);
                                     ^
----------------------------------------------------------------------

I can't tell for sure without seeing the diff. You may mean just
`ptr++`? But it could be something else, depending on what you're
trying to do.
> Also, I was wondering if we want to have pivoted normalization weighting as
> a separate weighting scheme in Xapian as it will make a way to add support
> of different variations of pivoted normalization in the future?
This is more of a question for Olly, who I believe should be back on
reliable internet later this week.

J

-- 
  James Aylett, occasional trouble-maker
  xapian.org

> Two of those are compile errors, suggesting you aren't pulling in the
> right header file (it's in common/serialise-double.h I believe).
Thanks, fixed those errors.
> I can't tell for sure without seeing the diff. You may mean just
> `ptr++`? But it could be something else, depending on what you're
> trying to do.
I'm trying to unserialise normalization strings (e.g. "nfn",
"nbsl"  etc.)
along with the new double parameters (s and delta) but
it isn't turning out to be smooth because there's no method for
unserialising strings in serialise-double.h

Although, doing just

const string normals = ptr++; or, const string normals = static_cast<const
string>ptr++; fixes compile errors.

But tfidfweight3 test case is failing with remote backends :-

$ ./runtest gdb ./apitest -v tfidfweight3

Running test: tfidfweight3... SerialisationError: REMOTE:Bad encoded
double: short mantissa (context: remote:prog(../bin/xapian-progsrv -t300000
.glass/db=apitest_simpledata)

I'm wondering if I need to introduce a new method in serialise-double.h for
string parameters (normalizations in this case)? To be honest, I have
little idea about that part of Xapian so probably a workaround might be
better. :)

Thanks,
Vivek



On Thu, Jul 28, 2016 at 3:14 AM, James Aylett <james-xapian at
tartarus.org>
wrote:
> On Wed, Jul 27, 2016 at 11:38:58PM +0530, Vivek Pal wrote:
>
> > But I'm running into some issues with Piv+ normalization. In the
Piv+
> > formula  , there are two parameters (s and delta) that control the
weight
> > assigned. I think the way I'm serialising and unserialising these
> > parameters has some issues which is causing a few errors,
> > http://pastebin.com/yCyuvF1X.  Any suggestion is appreciated!
>
> Two of those are compile errors, suggesting you aren't pulling in the
> right header file (it's in common/serialise-double.h I believe).
>
> This one:
>
> ----------------------------------------------------------------------
> weight/tfidfweight.cc: In member function 'virtual
> Xapian::TfIdfWeight* Xapian::TfIdfWeight::unserialise(const string&)
> const':
>
> weight/tfidfweight.cc:87:37: error: conversion from 'const char' to
> non-scalar type 'const string {aka const
> std::__cxx11::basic_string<char>}' requested
>
>      const string normals = (*(ptr)++);
>                                      ^
> ----------------------------------------------------------------------
>
> I can't tell for sure without seeing the diff. You may mean just
> `ptr++`? But it could be something else, depending on what you're
> trying to do.
>
> > Also, I was wondering if we want to have pivoted normalization
weighting
> as
> > a separate weighting scheme in Xapian as it will make a way to add
> support
> > of different variations of pivoted normalization in the future?
>
> This is more of a question for Olly, who I believe should be back on
> reliable internet later this week.
>
> J
>
> --
>   James Aylett, occasional trouble-maker
>   xapian.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xapian.org/pipermail/xapian-devel/attachments/20160729/3a87559f/attachment.html>