Xapian discuss - Jan 2005 - File Descriptors

We are currently having problem with "too many open files" on our
system. We
have a busy server that runs Apache, MySQL, and PHP on Red Hat Enterprise AS
2.1. The most "exotic" thing the system runs is Xapian 0.8.3, which is
accessed via the PHP-SWIG binding. I'm currently investigating the cause,
and I thought I'd check with this list to see if there was anything obvious
that had to do with Xapian.

We're averaging something like 18,000 Xapian queries a day. Does each Xapian
query create a lot of open files? Is there anything special I need to do to
make sure those files get closed? Are there any special considerations since
I'm using Xapian via the PHP-SWIG binding?

Thanks for your input. I learn something new about Linux every day, though
it's usually the hard way. :)

Mike.

On Fri, 2005-01-28 at 11:19 -0500, Mike Boone wrote:
> We are currently having problem with "too many open files" on our
system. We
> have a busy server that runs Apache, MySQL, and PHP on Red Hat Enterprise
AS
> 2.1. The most "exotic" thing the system runs is Xapian 0.8.3,
which is
> accessed via the PHP-SWIG binding. I'm currently investigating the
cause,
> and I thought I'd check with this list to see if there was anything
obvious
> that had to do with Xapian.
>
> We're averaging something like 18,000 Xapian queries a day. Does each
Xapian
> query create a lot of open files? Is there anything special I need to do to
> make sure those files get closed? Are there any special considerations
since
> I'm using Xapian via the PHP-SWIG binding?
Each open Xapian database uses 5 file descriptors (and a 6th is used
whilst the database is being opened, or when the database is being
flushed).  This could lead to quite a large number of filedescriptors
being used.

On Linux I believe there are two file descriptor limits - one is a hard
limit set by the operating system (but is quite a high value).  The
other is a lower, per-process limit, which can be changed.  Under bash,
"ulimit -n" displays the current per-process limit, which may be
modified if you have sufficient permissions (usually root).

All file descriptors used by a Xapian database should be released when
the database is closed, so as long as there isn't a bug in PHP-SWIG
which is failing to delete databases after use, there shouldn't be a
problem that raising the limit won't solve.  I haven't looked at the PHP
bindings in detail, but it looks like you are meant to explicitly delete
database (and other Xapian objects) after use, so it would be worth
making sure that you're doing that.

-- 
Richard Boulton <richard@tartarus.org>