Xapian devel - Oct 2014 - about 'DatabaseModifiedError' and 'reopen'

Hello! I found a problem when write and read at the same time.
When I read the Database, there will appear 'DatabaseModifiedError' at
sometimes. then I had to catch the 'DatabaseModifiedError', reopen the
Database and repeat the last operation. But when the number of concurrent of the
write is very high, the error 'DatabaseModifiedError' will often appear.
So, it seems not a good idea to catch and reopen the Database. Is there another
method to resolve the problem??
?????
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xapian.org/pipermail/xapian-devel/attachments/20141024/9bd69f4f/attachment-0002.html>

On Thu, Oct 23, 2014, at 10:52 PM, ???? wrote:
> Hello! I found a problem when write and read at the same time. When I
> read the Database, there will appear 'DatabaseModifiedError' at
> sometimes. then I had to catch the 'DatabaseModifiedError', reopen
the
> Database and repeat the last operation. But when the number of
> concurrent of the write is very high, the error
> 'DatabaseModifiedError' will often appear. So, it seems not a good
> idea to catch and reopen the Database. Is there another method to
> resolve the problem??
We just use a separate external locking system, which is kind of sad -
but it gets the job done :)

Here's the full comment before we take the lock (which used to pass
/*read*/0 as the last param)

/* we grab an activefile writelock to index. Strictly we don't need it,
but * doing this guarantees we never write under a client which is
reading, which * avoids this: * * IOERROR: Xapian: caught exception: :
DatabaseModifiedError: The revision * being read has been discarded -
you should call Xapian::Database::reopen() * and retry the operation * *
in theory, this will go away eventually, and we can switch back to
write: 0 * in this code. * *
http://grokbase.com/t/xapian/xapian-discuss/0667ppbks8/#20060608j8x5aeept49dv5fm8d02xkczgr
* * "This is almost invariably caused by updating a database while
reading * from it. If two updates are committed before the read
completes, you * get this error (it's DatabaseModifiedError). It's a bit
of a pain * and will be going away in the future, but it's not too hard
to design * to avoid it happening at least." */ active
activefile_open(mailbox->name, mailbox->part, &tr->activefile,
/*write*/1);

Cheers,

Bron.

--
Bron Gondwana brong at fastmail.fm


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xapian.org/pipermail/xapian-devel/attachments/20141025/ebebdbb1/attachment-0002.html>

On Fri, Oct 24, 2014 at 10:52:03AM +0800, ???? wrote:
> When I read the Database, there will appear 'DatabaseModifiedError'
at
> sometimes. then I had to catch the 'DatabaseModifiedError', reopen
the
> Database and repeat the last operation. But when the number of
> concurrent of the write is very high, the error
> 'DatabaseModifiedError' will often appear. So, it seems not a good
> idea to catch and reopen the Database. Is there another method to
> resolve the problem?
My suggestion would be to batch up your updates (or batch them up more
than you currently are).

It's inherently much more efficient to apply updates in batches, but in
particular it means that the time between commits will be greater, so
searches will get a chance to complete before the revision they are
reading gets yanked away from underneath them.

There's work in progress to allow readers to lock the revision being
read, which would allow you to avoid DatabaseModifiedError.  However,
batching up updates is still a generally a good idea.

Cheers,
    Olly