Xapian discuss - Apr 2024 - Xapian concurrency

Dear list,

I'm trying to use Xapian in a multi-threaded environment; and generally
be carefully that database is not accessed concurrently (as per [1])

However, during indexing I want to create Xapian documents in multiple
threads (without referencing the database in any way). Actually adding
them to the database is done in a single thread (and with locks etc.)

Is this is a safe way to implement this?

Thanks in advance,
Dirk.

[1]
https://getting-started-with-xapian.readthedocs.io/en/latest/concepts/concurrency.html

-- 
Dirk-Jan C. Binnema                  Helsinki, Finland
e:djcb at djcbsoftware.nl           w:www.djcbsoftware.nl
gpg: 6987 9CED 1745 9375 0F14 DA98 11DD FEA9 DCC4 A036

On Sun, Apr 28, 2024 at 11:10:12PM +0300, Dirk-Jan C. Binnema
wrote:
> I'm trying to use Xapian in a multi-threaded environment; and generally
> be carefully that database is not accessed concurrently (as per [1])
> 
> However, during indexing I want to create Xapian documents in multiple
> threads (without referencing the database in any way). Actually adding
> them to the database is done in a single thread (and with locks etc.)
> 
> Is this is a safe way to implement this?
It should be if you're careful about how the object is handed off from
one tread to another - you want to make sure that two different threads
won't try to modify the internal reference count simultaneously.

I'd suggest looking at using C++11 std::move() to safely transfer the
Xapian::Document object between the two threads.  I've not tried it
myself, but German M.  Bravo reported using this approach in the (sadly
apparently now defunct) xapiand project.

We should probably document this approach in the page you linked to so
if you try it, some feedback would be useful.

Cheers,
    Olly