Xapian discuss - Jul 2008 - Xapian across multiple servers

Hi,

We're looking to deploy Xapian as a search solution for a website
that's run across a number of load-balanced web servers.  However,
I've not seen any clear indication in the documentation as to how it's
recommended to operate Xapian in a load balanced / high availability
configuration, or how to distribute the content-changes into the
indexer.  Has anyone got any particular recommendations?  My feeling
is that the following are all possibilities:

1) Run Xapian locally on each front-end webserver, and distribute
content changes via existing messaging infrastructure.  This will
clearly work, but it's not necessarily optimal.
2) Run Xapian locally on each front-end webserver, but storing the
index on shared storage.  This will be I/O intensive, but doesn't
involve syncing changes to out to each front-end.
3) Run Xapian centrally, and access it using the remote protocol from
each front end.  This obviously makes it easier to construct the
index, but it's not obvious how to make this scalable / HA.
4) Something else?

I'd be really grateful if someone can share their success stories (or
otherwise).

Cheers,


Rob

On Tue, Jul 22, 2008 at 10:54:44AM +0100, Rob S wrote:
> We're looking to deploy Xapian as a search solution for a website
> that's run across a number of load-balanced web servers.  However,
> I've not seen any clear indication in the documentation as to how
it's
> recommended to operate Xapian in a load balanced / high availability
> configuration, or how to distribute the content-changes into the
> indexer.  Has anyone got any particular recommendations?  My feeling
> is that the following are all possibilities:
It partly depends on the loads on the servers from other things, and
from Xapian.
   
You may find it better to keep search on separate servers, particularly
if you have a large Xapian index and a lot of searches.  This also means
you can add more search or webservers independently.

Also relevant is how key search is to your application.  For some sites,
having the search unavailable is less of a problem than the site being
unavailable, so redundancy in search may be less critical.
> 1) Run Xapian locally on each front-end webserver, and distribute
> content changes via existing messaging infrastructure.  This will
> clearly work, but it's not necessarily optimal.
Indeed.
> 2) Run Xapian locally on each front-end webserver, but storing the
> index on shared storage.  This will be I/O intensive, but doesn't
> involve syncing changes to out to each front-end.
Probably OK if the search load is low.  A high search load on a large
database will probably get ugly.
> 3) Run Xapian centrally, and access it using the remote protocol from
> each front end.  This obviously makes it easier to construct the
> index, but it's not obvious how to make this scalable / HA.
Should involve less network traffic than mounting the database over NFS
or similar, but it does put all the search load on one server.
> 4) Something else?
SVN trunk has a database replication feature which is aimed at this sort
of situation.  It's not been released yet, and could still change before
it is, but it's being used on at least one live site I believe.

Cheers,
    Olly