Xapian discuss - Jan 2010 - Xapian under 360 MB VPS

Hello,

I wish to get some feedback on the use of Xapian in a virtual machine hosting
plan with 360MB. The processes to share the 360MB will be the following:

0. nginx web server as front (estimated 5MB)

1. custom C++ FastCGI for dynamic requests (estimated 10MB)

2. Xapian writer (1 process and 1 thread)

3. Xapian readers (1 process with n threads for n readers)

4. PostgreSQL (estimated 50MB or lower)


That leaves about 300MB for Xapian and the rest of the Linux OS. The main UI
will be a Google style search box.

Questions:

0. How would you configure Xapian for such low memory systems (e.g. how many
readers, flush threshold for writer)?

1. Will file handle limitation be a problem for multithreaded Xapian reader?

2. What are advantages of multiprocess readers (compared to multithreaded) aside
from crash isolation


Thanks so much!
Marlon

2010/1/21 Marlon Baculio <mbaculio at
hotmail.com>:
> That leaves about 300MB for Xapian and the rest of the Linux OS. The main
UI will be a Google style search box.
> 0. How would you configure Xapian for such low memory systems (e.g. how
many readers, flush threshold for writer)?
Totally depends on the load, and the size of documents.  To work out
the flush threshold, I'd probably do an index run, logging the number
of documents processed and watching the memory use in "top", and set
the flush threshold to the number processed when the indexer memory
use reaches about half the available memory use (so space is left for
the OS disk block cache, and the other processes).
> 1. Will file handle limitation be a problem for multithreaded Xapian
reader?
Depends on search load.  Each reader keeps about 5 filehandles open,
so multiply that by the number of concurrent readers you want.  If it
comes close to the per-process fd limit, you've got a problem.
> 2. What are advantages of multiprocess readers (compared to multithreaded)
aside from crash isolation
I can't think of any significant ones off the top of my head.  You
can't access a reader concurrently from multiple threads, so it
doesn't make much difference to Xapian whether the reader is in a
separate thread or a separate process.

You might find it easier to pool connections for reuse if you use
threads, but a process pool is perfectly feasible in theory too.

The readers should have very little memory overhead - they don't cache
anything, leaving that up to the OS disk cache (so it'll be shared
between all readers automatically).

--
Richard