Xapian discuss - Jun 2018 - output debug information of LOGCALL_CTOR/LOGCALL_VOID/...

Hi,

I have read the concurrency webpage from the Xapian documentation:

http://getting-started-with-xapian.readthedocs.io/en/latest/concepts/concurrency.html

But it is still not clear to me how to ensure thread-safety when using
libxapian (C++ API). Usually when doing multi-threading many threads can
read the same variable concurrently without locking provided none of the
threads modifies the variable.

1) Does this apply for Xapian as well? I.e. Can many thread search the
database provided each thread uses their own Xapian database object and
none of the threads updates/indexes the database?

2) What about when one thread indexes a new document? Do I need to
ensure (by locking) that no other thread reads or write the database until
the first thread has finished indexing?

Thanks,
Kim

On 8 Feb 2018, at 15:18, Kim Walisch <kim.walisch at gmail.com> wrote:
> I have read the concurrency webpage from the Xapian documentation:
> 
>
http://getting-started-with-xapian.readthedocs.io/en/latest/concepts/concurrency.html
> 
> But it is still not clear to me how to ensure thread-safety when using
> libxapian (C++ API).
> 
> 1) Can many thread search the
> database provided each thread uses their own Xapian database object and
> none of the threads updates/indexes the database?
This is covered explicitly in the document you linked to:
> Xapian doesn’t maintain any global state, so you can safely use Xapian in a
multi-threaded program provided you don’t share objects between threads. In
practice this restriction is often not a problem - each thread can create its
own xapian.Database object, and everything will work fine.

So the answer is 'yes'.
> 2) What about when one thread indexes a new document? Do I need to
> ensure (by locking) that no other thread reads or write the database until
> the first thread has finished indexing?
This isn't a problem either; Xapian takes care of lock management for you.
It's covered in a different part of the manual:
https://getting-started-with-xapian.readthedocs.io/en/latest/concepts/indexing/databases.html#concurrent-access
> Currently, all the backends only support a single writer existing at a
given time; attempting to open another writer on the same database will throw
xapian.DatabaseLockError to indicate that it wasn’t possible to acquire a lock.
Multiple concurrent readers are supported (in addition to the writer).

Note the caution there about the number of versions that can exist in
Xapian's MVCC, and how to deal with DatabaseModifiedError.

J

-- 
 James Aylett, occasional troublemaker & project governance
 xapian.org

Awesome, thanks!

On Feb 9, 2018 8:33 PM, "James Aylett" <james-xapian at
tartarus.org> wrote:
> On 8 Feb 2018, at 15:18, Kim Walisch <kim.walisch at gmail.com>
wrote:
>
> > I have read the concurrency webpage from the Xapian documentation:
> >
> > http://getting-started-with-xapian.readthedocs.io/en/
> latest/concepts/concurrency.html
> >
> > But it is still not clear to me how to ensure thread-safety when using
> > libxapian (C++ API).
> >
> > 1) Can many thread search the
> > database provided each thread uses their own Xapian database object
and
> > none of the threads updates/indexes the database?
>
> This is covered explicitly in the document you linked to:
>
> > Xapian doesn’t maintain any global state, so you can safely use Xapian
> in a multi-threaded program provided you don’t share objects between
> threads. In practice this restriction is often not a problem - each thread
> can create its own xapian.Database object, and everything will work fine.
>
>
> So the answer is 'yes'.
>
> > 2) What about when one thread indexes a new document? Do I need to
> > ensure (by locking) that no other thread reads or write the database
> until
> > the first thread has finished indexing?
>
> This isn't a problem either; Xapian takes care of lock management for
you.
> It's covered in a different part of the manual:
> https://getting-started-with-xapian.readthedocs.io/en/
> latest/concepts/indexing/databases.html#concurrent-access
>
> > Currently, all the backends only support a single writer existing at a
> given time; attempting to open another writer on the same database will
> throw xapian.DatabaseLockError to indicate that it wasn’t possible to
> acquire a lock. Multiple concurrent readers are supported (in addition to
> the writer).
>
>
> Note the caution there about the number of versions that can exist in
> Xapian's MVCC, and how to deal with DatabaseModifiedError.
>
> J
>
> --
>  James Aylett, occasional troublemaker & project governance
>  xapian.org
>
>

On Thu, Feb 08, 2018 at 04:18:12PM +0100, Kim Walisch
wrote:
> But it is still not clear to me how to ensure thread-safety when using
> libxapian (C++ API). Usually when doing multi-threading many threads can
> read the same variable concurrently without locking provided none of the
> threads modifies the variable.
That's true for simple types, but breaks down for classes because they
may have mutable members - e.g. for caching values computed lazily:

class FactorialFactory {
  private:
     mutable int r = -1;
     mutable int n;
  public:
     FactorialFactory() {}

     int calc(int v) const {
         if (r < 0 || n != v) {
	     r = n;
	     n = v;
	     for (int i = n - 1; i > 1; --i) {
	         r *= i;
	     }
	 }
	 return r;
     }
};

It's not safe to concurrently call f.calc() from different threads, even
though conceptually calc() is a read-only method.

Cheers,
    Olly

Hi,

We have a search service based on xapian, when receving a request, we create
several threads to create xapian-query and search in the corresponding
databases, and then merge all results together.

In some case, we use Xapian::Query::MatchAll to create the query, but it always
has Segmentation fault (core dumped). It looks like some pointers are double
freed.

This is the function that we use to create xapian-query.

std::shared_ptr<Xapian::Query> constructQuery() {

    Xapian::Query black_list("BLt1");

    return std::make_shared<Xapian::Query>(Xapian::Query::OP_AND_NOT,
Xapian::Query::MatchAll, black_list);

}

I have wrote a demo to reproduce our problems, you can click this link for the
detail. 
https://github.com/xiangqianzsh/xapian_leaning/tree/master/matchall_coredump

How we solve this problems?

On Mon, Apr 09, 2018 at 12:49:11AM +0800, 张少华 wrote:
> In some case, we use Xapian::Query::MatchAll to create the query, but
> it always has Segmentation fault (core dumped). It looks like some
> pointers are double freed.
Xapian::Query::MatchAll is a static object, but that means that it's a
Xapian object that can get used from multiple threads concurrently which
we don't support.  The result is that the reference count can get
updated by more than one thread at the same time and get out of step.

I don't currently see a nice way to fix this while providing the same
API, but there is at least a simple workaround - just use
Xapian::Query(std::string()) instead of Xapian::Query::MatchAll in
multi-threaded code.

Xapian::Query::MatchNothing is also a static object, but in that case
the internal object is NULL so there isn't a reference count and it
should actually be safe in multi-threaded code.
> This is the function that we use to create xapian-query.
> 
> std::shared_ptr<Xapian::Query> constructQuery() {
> 
>     Xapian::Query black_list("BLt1");
> 
>     return std::make_shared<Xapian::Query>(Xapian::Query::OP_AND_NOT,
Xapian::Query::MatchAll, black_list);
> 
> }
This isn't connected with the problem here, but there's no need to use
std::shared_ptr<> here if you're just wanting to reference count the
object as Xapian::Query is itself a reference counted pointer to an
internal implementation object.

Cheers,
    Olly

HI,


Many functions in xapian can be traced when they are called using
LOGCALL_CTOR/LOGCALL_VOID/... and I want to output the debug information.
So, I first install xapian and enable log, using commands as follow
```
cd xapian-core-1.4.5  # xapian source code downloaded
./configure --enable-log=yes --enable-assertions=yes
make
sudo make install
```
But there isn't any debug information when I run
xapian-core-1.4.5/examples/simplesearch.cc


Also, I notice -DXAPIAN_REALLY_NO_DEBUG_LOG is added in
xapian-core-1.4.5/Makefile, so I deleted this options, but I can not compile
successfully then.


How I can output the debug information.

On 9 Jun 2018, at 07:37, 张少华 <xiangqianzsh at 163.com> wrote:
> Many functions in xapian can be traced when they are called using
LOGCALL_CTOR/LOGCALL_VOID/... and I want to output the debug information.
In xapian-core/HACKING in the source code there's a section on this. As well
as pass --enable-log to configure, you also need to:

 * set XAPIAN_DEBUG_LOG to be the path to a file that you would like debugging
   output to be appended to, or to the special value ``-`` to indicate that you
   would like debugging output to be sent to stderr.  Unless XAPIAN_DEBUG_LOG
   is set, no debug logging will be performed.  Occurrences of %p in
   XAPIAN_DEBUG_LOG will be replaced with the current process-id.
> Also, I notice -DXAPIAN_REALLY_NO_DEBUG_LOG is added in
xapian-core-1.4.5/Makefile, so I deleted this options, but I can not compile
successfully then.
That symbol is to stop debug calls being available when compiling things that
use Xapian. It's used in common/debuglog.h, which says:

// In places where we include library code in non-library contexts, we can't
// have debug logging enabled, as the support functions aren't visible

which sounds like exactly what you've encountered.

J

-- 
 James Aylett
 devfort.com — spacelog.org — tartarus.org/james/