Hi all, I have a Xapian index set up and a mutithreaded xmlrpc server serving results. Inside the xmlrpc server I only open the xapian database for reading once. This causes requests to "stay in line" in order to get results and sometimes a one operation can take up to 1 second to complete (a time-taking operation includes text analisys and some more operations besides the actual Xapian interaction) thus delaying all the queued requests. I read lots of documents about xapian in mutithreaded environment and concluded that even for read access one cannot open the same database more than once and perform operations at the same time. Because I did not find a straight answer to this I am hoping one of you guys can clear this once and for all. I am using Xapian 0.9.6, Python bindings and Twisted Python XMLRPC package. Thanks, Lucian Nicolescu
Lucian Nicolescu wrote:> Inside the xmlrpc server I only open the xapian database for reading once. > This causes requests to "stay in line" in order to get results and sometimes > a one operation can take up to 1 second to complete (a time-taking operation > includes text analisys and some more operations besides the actual Xapian > interaction) thus delaying all the queued requests. > > I read lots of documents about xapian in mutithreaded environment and > concluded that even for read access one cannot open the same database more > than once and perform operations at the same time.This is a misconception - for read access, you can open the same database as many times as you like concurrently (well, as many as your operating system will allow - eventually, you'll run out of file handles). However, each of the Database objects you get by opening the database must not be accessed concurrently. A typical model might be to create N threads, and use N Database objects, one for each thread.> I am using Xapian 0.9.6, Python bindings and Twisted Python XMLRPC package.Just to note - Xapian 0.9.6 is rather old - the latest release is 1.0.2. 1.0.2 has the benefit that the Python bindings are set up to release the GIL whenever a potentially long-running xapian operation is in progress, which will give better performance in a concurrent environment. -- Richard
> I read lots of documents about xapian in mutithreaded environment and > concluded that even for read access one cannot open the same database more > than once and perform operations at the same time.You can open up multiple (ReadOnly) instances of the database. The problem is the concurrency between write & read operations with different connections to a Xapian database. If your read operations need to be in-sync with committed write operations, you will need to re-open the ReadOnly instances before each read. alan
Took care of that, the database is reopened every 60 minutes. Thanks, Lucian Nicolescu> -----Original Message----- > From: xapian-discuss-bounces@lists.xapian.org > [mailto:xapian-discuss-bounces@lists.xapian.org] On Behalf Of > alan runyan > Sent: Wednesday, August 01, 2007 4:23 PM > To: xapian-discuss@lists.xapian.org > Subject: [Xapian-discuss] Re: Multithreaded read access > > > I read lots of documents about xapian in mutithreaded > environment and > > concluded that even for read access one cannot open the > same database more > > than once and perform operations at the same time. > > You can open up multiple (ReadOnly) instances of the > database. The problem is > the concurrency between write & read operations with > different connections to a > Xapian database. If your read operations need to be in-sync > with committed > write operations, you will need to re-open the ReadOnly > instances before each > read. > > alan > > > _______________________________________________ > Xapian-discuss mailing list > Xapian-discuss@lists.xapian.org > http://lists.xapian.org/mailman/listinfo/xapian-discuss >