Xapian devel - Jun 2014 - About memory index/search in multithread program

There may be some solutions?for example
class XAPIAN
{
    *static* int InitDatabase();  //for reading only, do not need lock, but
if writing use lock
    int Search(); //safe in one object, do not need lock
};
XIPIAN xp[ THREAD_NUM ];

one thread use one object, they use one database.
these can be in memory with one database.


2014-06-24 20:48 GMT+08:00 Olly Betts <olly at survex.com>:
> On Thu, Jun 19, 2014 at 10:52:13PM +0800, Yanxiong Lu wrote:
> > Why xapian don't support memory  index/search ?
> >
> > I know there is a method can create memory datebase, like this:
> > Xapian::WritableDatabase db(Xapian::InMemory::open());
>
> While the inmemory backend is handy for some things, in its current
> incarnation some operations really aren't very time efficient, and it
> doesn't even try to be very space efficient.
>
> The longer term plan is to replace it, probably sharing code with the
> disk based backends (which would make it fairly easy to do neat things
> like migrating data from "inmemory" to disk).
>
> > *But, if i use these in multithread program,  i need create many
> > datebases!!*
> > Xapian::WritableDatabase db1(Xapian::InMemory::open());  //used in
> thread1
> > Xapian::WritableDatabase db2(Xapian::InMemory::open());  //used in
> thread2
> >
> > because WritableDatabase object isn't thread-safe. And use lock is
> slowly.
>
> Pushing those locks inside the Xapian library won't make things any
> faster - likely just the opposite.  If the API user locks, they
> can choose the granularity, while if Xapian locks, it has to lock
> and unlock around every API method called.
>
> I guess a lock-free, thread-safe inmemory backend implementation might
> be possible, but realistically if you want one any time soon, you're
> probably going to need to contribute most of the code for it.
>
> > *So, is there some solutions that One database, many thread can use??*
>
> The simplest solution currently is probably to create a RAM disk and use
> a disk-based backend in it.
>
> Cheers,
>     Olly
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xapian.org/pipermail/xapian-devel/attachments/20140626/d34f7e2c/attachment-0002.html>

On Thu, Jun 26, 2014 at 01:19:20PM +0800, Yanxiong Lu
wrote:
> There may be some solutions?for example
> class XAPIAN
> {
>     *static* int InitDatabase();  //for reading only, do not need lock, but
> if writing use lock
>     int Search(); //safe in one object, do not need lock
> };
> XIPIAN xp[ THREAD_NUM ];
> 
> one thread use one object, they use one database.
> these can be in memory with one database.
As I already tried to explain, if you're concerned about performance,
the current inmemory backend isn't the answer you are looking for.
Eliminating the need for the user to lock around searches isn't going to
magically solve that.

I don't really see how your 6 line sketch above helps either.  Xapian
makes *NO* thread-safety guarantees - even in the "reading only" case
you should be locking.  Even if it happens to work currently, there's
no guarantee it'll work in the next release.

Cheers,
    Olly

That code can't work, it is pseudo-code.

As I already tried to explain, if you're concerned about performance,
the current inmemory backend isn't the answer you are looking for.

I will write a memory based SE with WAND merge algorithm and proximity
similarity
algorithm(BM25 don't use position).
Thank you very much.


2014-06-26 14:33 GMT+08:00 Olly Betts <olly at survex.com>:
> On Thu, Jun 26, 2014 at 01:19:20PM +0800, Yanxiong Lu wrote:
> > There may be some solutions?for example
> > class XAPIAN
> > {
> >     *static* int InitDatabase();  //for reading only, do not need
lock,
> but
> > if writing use lock
> >     int Search(); //safe in one object, do not need lock
> > };
> > XIPIAN xp[ THREAD_NUM ];
> >
> > one thread use one object, they use one database.
> > these can be in memory with one database.
>
> As I already tried to explain, if you're concerned about performance,
> the current inmemory backend isn't the answer you are looking for.
> Eliminating the need for the user to lock around searches isn't going
to
> magically solve that.
>
> I don't really see how your 6 line sketch above helps either.  Xapian
> makes *NO* thread-safety guarantees - even in the "reading only"
case
> you should be locking.  Even if it happens to work currently, there's
> no guarantee it'll work in the next release.
>
> Cheers,
>     Olly
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xapian.org/pipermail/xapian-devel/attachments/20140627/f3500ffd/attachment-0002.html>