Xapian devel - Jun 2014 - About memory index/search in multithread program

hi,
Why xapian don't support memory  index/search ?

I know there is a method can create memory datebase, like this:
Xapian::WritableDatabase db(Xapian::InMemory::open());

*But, if i use these in multithread program,  i need create many
datebases!!*
Xapian::WritableDatabase db1(Xapian::InMemory::open());  //used in thread1
Xapian::WritableDatabase db2(Xapian::InMemory::open());  //used in thread2

because WritableDatabase object isn't thread-safe. And use lock is slowly.

*So, is there some solutions that One database, many thread can use??*
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xapian.org/pipermail/xapian-devel/attachments/20140619/1e52ad10/attachment.html>

On Thu, Jun 19, 2014 at 10:52:13PM +0800, Yanxiong Lu
wrote:
> Why xapian don't support memory  index/search ?
> 
> I know there is a method can create memory datebase, like this:
> Xapian::WritableDatabase db(Xapian::InMemory::open());
While the inmemory backend is handy for some things, in its current
incarnation some operations really aren't very time efficient, and it
doesn't even try to be very space efficient.

The longer term plan is to replace it, probably sharing code with the
disk based backends (which would make it fairly easy to do neat things
like migrating data from "inmemory" to disk).
> *But, if i use these in multithread program,  i need create many
> datebases!!*
> Xapian::WritableDatabase db1(Xapian::InMemory::open());  //used in thread1
> Xapian::WritableDatabase db2(Xapian::InMemory::open());  //used in thread2
> 
> because WritableDatabase object isn't thread-safe. And use lock is
slowly.
Pushing those locks inside the Xapian library won't make things any
faster - likely just the opposite.  If the API user locks, they
can choose the granularity, while if Xapian locks, it has to lock
and unlock around every API method called.

I guess a lock-free, thread-safe inmemory backend implementation might
be possible, but realistically if you want one any time soon, you're
probably going to need to contribute most of the code for it.
> *So, is there some solutions that One database, many thread can use??*
The simplest solution currently is probably to create a RAM disk and use
a disk-based backend in it.

Cheers,
    Olly

There may be some solutions?for example
class XAPIAN
{
    *static* int InitDatabase();  //for reading only, do not need lock, but
if writing use lock
    int Search(); //safe in one object, do not need lock
};
XIPIAN xp[ THREAD_NUM ];

one thread use one object, they use one database.
these can be in memory with one database.


2014-06-24 20:48 GMT+08:00 Olly Betts <olly at survex.com>:
> On Thu, Jun 19, 2014 at 10:52:13PM +0800, Yanxiong Lu wrote:
> > Why xapian don't support memory  index/search ?
> >
> > I know there is a method can create memory datebase, like this:
> > Xapian::WritableDatabase db(Xapian::InMemory::open());
>
> While the inmemory backend is handy for some things, in its current
> incarnation some operations really aren't very time efficient, and it
> doesn't even try to be very space efficient.
>
> The longer term plan is to replace it, probably sharing code with the
> disk based backends (which would make it fairly easy to do neat things
> like migrating data from "inmemory" to disk).
>
> > *But, if i use these in multithread program,  i need create many
> > datebases!!*
> > Xapian::WritableDatabase db1(Xapian::InMemory::open());  //used in
> thread1
> > Xapian::WritableDatabase db2(Xapian::InMemory::open());  //used in
> thread2
> >
> > because WritableDatabase object isn't thread-safe. And use lock is
> slowly.
>
> Pushing those locks inside the Xapian library won't make things any
> faster - likely just the opposite.  If the API user locks, they
> can choose the granularity, while if Xapian locks, it has to lock
> and unlock around every API method called.
>
> I guess a lock-free, thread-safe inmemory backend implementation might
> be possible, but realistically if you want one any time soon, you're
> probably going to need to contribute most of the code for it.
>
> > *So, is there some solutions that One database, many thread can use??*
>
> The simplest solution currently is probably to create a RAM disk and use
> a disk-based backend in it.
>
> Cheers,
>     Olly
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xapian.org/pipermail/xapian-devel/attachments/20140626/d34f7e2c/attachment-0002.html>