Xapian discuss - Feb 2016 - RESTful xapian server needs help

For the last year we have been working hard in releasing an open source RESTful
xapian server named Xapiand (https://github.com/Kronuz/Xapiand) that will offer
clustering, replication and some other neat features (similar to what
elasticsearch offers, but the final goals go beyond it); however, for our
Xapiand server to work, we need to have a few changes in xapian-core that are
**essential**.

Please help us. We have two options, but this is very important for us to get
either of one (preferably option A) committed for the next version of
xapian-core, otherwise the server wouldn't compile or wouldn't be fully
featured with out-of-the-box xapian and our efforts would have been useless for
the community for the whole time until the next stable version of xapian.


Option A: Re-implementing the remote protocol ourselves in our server, these are
the things that need to be patched in xapian-core (not a whole lot to change,
but there are some important things which would be essential)

1) **IMPORTANT** Add get_revision_info() in Database API (optionally guaranty
that, if returned, strings could be compared between revisions to know which
revision comes first, using string::compare or strcmp)

2) **IMPORTANT** Make a public MultiMatch API (to be able to re-implement the
remote protocol ourselves)

3) **IMPORTANT** Send selected database path using MSG_SELECT to the remote
protocol if REPLY_UPDATE returns with no database selected (that's what our
protocol would do)

4) *OPTIONAL* Add MSG_SELECT to remote protocol (as an example of what it's
for and to reserve the command number in the protocol)



Option B: Using a public RemoteProtocol API, these are the things that would
need to be patched in xapian-core (the remote protocol would be a rather big
change)

1) **IMPORTANT** Add get_revision_info() in Database API (optionally guaranty
that, if returned, strings could be compared between revisions to know which
revision comes first, using string::compare or strcmp)

2) **IMPORTANT** Make a public RemoteProtocol API (big change, moving files
around and making a new public API)

3) **IMPORTANT** Send selected database path using MSG_SELECT to the remote
protocol if REPLY_UPDATE returns with no database selected (that's what our
protocol would do)

4) *OPTIONAL* Add MSG_SELECT to remoteserver (as an example of what it's for
and to reserve the command number in the protocol)


As I said, I think I'd prefer option A and we are prepared to make a pull
request for either of the two. Please help us make this happen. We feel strongly
about getting this in xapian-core and about it allowing our Xapiand server to
lift off.


German M. Bravo (Kronuz)

> For the last year we have been working hard in releasing an open source
RESTful xapian server named Xapiand

Reading the Portability Notes section at the following link is informative.

http://search.cpan.org/dist/EV/libev/ev.pod

On Sun, Feb 28, 2016 at 09:43:41AM -0600, German M. Bravo
wrote:
> Please help us. We have two options, but this is very important for us
> to get either of one (preferably option A) committed for the next
> version of xapian-core, otherwise the server wouldn't compile or
> wouldn't be fully featured with out-of-the-box xapian and our efforts
> would have been useless for the community for the whole time until the
> next stable version of xapian.
I think most of this has been addressed by or obsoleted by more recent
discussions on IRC.

But I think the key points from those discussions are worth noting here,
as not everyone idles on IRC or reads the channel logs:

* It's really way too late to be trying to push for major new features
  to go in before 1.4.0 - this email back in October was effectively the
  last call: http://thread.gmane.org/gmane.comp.search.xapian.general/9891
  We've already made a number of hard-nosed decisions about things to
  cut.

* However, almost everything you want to do can be achieved by adding
  API methods, so can be added in 1.4.x.

* You need to focus on anything which can't be done in 1.4.x, and you
  don't have much time - we're probably a few weeks away from a release
  at this point.  If someone has a sane patch including tests and
  documentation ready to go, I'm unlikely to reject it, but I'm also
  unlikely to hold the release for it.  By "sane" I mean a patch I
don't
  have to iterate much on, which has a low risk/reward ratio, etc.  I'm
  likely to get more conservative as we close in on 1.4.0.

* Open tickets in trac for things!  And with a sensible granularity.  If
  there's not a ticket in trac for something, it essentially doesn't
  exist from the point of view of release planning.

Cheers,
    Olly