Xapian discuss - Sep 2012 - C++ PostingSource with Python bindings

Hi all,

I want to use a custom PostingSource to adjust weightings of my search 
results, but an implementation in Python is proving to be very slow for 
large result sets.  So I'm looking to write the custom PostingSource in C++.

The basic strategy would be:

q = xapian.Query(...)
q = mycustommodule.add_posting_source(q)


I can acquire the Query pointer easily enough (it's just int(q.this)) 
and cast that to Xapian::Query, and then construct a new Xapian::Query 
with the custom posting source mixed in with OP_AND_MAYBE.

The challenge is getting the new C++ query object back into Python 
space.  SWIG documents say that although int(swigobj) gives you the 
underlying pointer, the inverse operation of constructing a SWIG object 
from a pointer isn't possible. [1]  I'm hoping there might be some 
Xapian-specific way, or that this documented limitation doesn't apply to 
native C/C++ code.

Is there a solution, or some alternative approach?

Thanks,
Jason.

[1] http://www.swig.org/Doc2.0/Python.html#Python_nn18

On Fri, Aug 31, 2012 at 08:23:06PM -0400, Jason Tackaberry
wrote:
> I want to use a custom PostingSource to adjust weightings of my search  
> results, but an implementation in Python is proving to be very slow for  
> large result sets.  So I'm looking to write the custom PostingSource in
> C++.
Ideally there would just be an easy way to wrap extra C++ classes for
Python (and other languages) such that they work as if they were part
of the Xapian API.

It should be possible to wrap classes in this way, but there's not
an easy way to do it currently.  So your approach is probably simpler
to implement right now.
> The basic strategy would be:
>
> q = xapian.Query(...)
> q = mycustommodule.add_posting_source(q)
>
> I can acquire the Query pointer easily enough (it's just int(q.this))  
> and cast that to Xapian::Query, and then construct a new Xapian::Query  
> with the custom posting source mixed in with OP_AND_MAYBE.
>
> The challenge is getting the new C++ query object back into Python  
> space.  SWIG documents say that although int(swigobj) gives you the  
> underlying pointer, the inverse operation of constructing a SWIG object  
> from a pointer isn't possible. [1]  I'm hoping there might be some
> Xapian-specific way, or that this documented limitation doesn't apply
to
> native C/C++ code.
>
> Is there a solution, or some alternative approach?
I'd try making the way you'd call this from Python:

    q = xapian.Query(...)
    mycustommodule.add_posting_source(q)

And then in the C++ code, construct your new Query object and swap it
with the old one:

    Xapian::Query * q = (Xapian::Query*)(...);
    Xapian::Query q_new(...);
    std::swap(q_new, *q);

Query objects are reference counted handles, so I think that should make
q in the python code refer to the new object.

If not, then you could try (though this isn't part of the public API so
might not work in the future even if it does now):

    q->internal = q_new.internal;

Cheers,
    Olly