Xapian devel - May 2007 - multiple writers and remotetcp backends

Hi guys,
  We've been in discussion with Richard and Olly on this issue, in various
different forums, but as the correct answer isn't immediately obvious,
I'm
opening it up for wider discussion and comment.

The problem is that a xapian tcp-server in 'writable' mode makes no
attempt
to ensure only one 'active' connection at a time is trying to modify the
database.  If multiple connections are made to a writable server, the
behaviour is undefined (or even it is was defined, it is unlikely to be
defined in a way that would make it useful).  While some applications can
ensure internally that only a single connection is made to such a server,
some applications are architected such that multiple processes, possibly
even multiple machines, must coordinate this "single writer" approach.
This
becomes quite difficult without support inside xapian itself.

It seems there are 2 general solutions we can implement:

* Only ever allow a single connection to the writable server.  When a second
connection is attempted, we either refuse the connection, or allow the
connection just to send back an authorative 'writer already connected'
response, and then close the connection.

* Implement a kind of 'queue' or some other way to block the incoming
connection.  In this case we would accept the connection, respond with a
message indicating they are in a queue (your call is important!) and then
block until the first writer is complete.  The client side of the connection
then has a choice regarding waiting in the queue, or hanging up and trying
again later.


In my opinion, the second option sounds the "best", but the first
option
seems "good enough" and easier to implement.  I'm sure there are
both other
options and opinions on this topic, so I'm soliciting all feedback on this,
with the intention of opening a xapian bug to track the status, and
ultimately end up with a patch.

Thanks,

Mark

Mark Hammond wrote:
> It seems there are 2 general solutions we can implement:
> 
> * Only ever allow a single connection to the writable server.  When a
second
> connection is attempted, we either refuse the connection, or allow the
> connection just to send back an authorative 'writer already
connected'
> response, and then close the connection.
> 
> * Implement a kind of 'queue' or some other way to block the
incoming
> connection.  In this case we would accept the connection, respond with a
> message indicating they are in a queue (your call is important!) and then
> block until the first writer is complete.  The client side of the
connection
> then has a choice regarding waiting in the queue, or hanging up and trying
> again later.
 >
> In my opinion, the second option sounds the "best", but the first
option
> seems "good enough" and easier to implement.
I agree. If we implement the first solution, I imagine that many clients 
will need to emulate the behaviour of the second solution by repeatedly 
opening a connection to poll for the lock.  While the first option is 
slightly simpler to implement, I think that the majority of the 
implementation work is likely to be in enforcing the single-writer 
constraint.  I recommend attempting to implement the first solution 
first anyway, since the second solution requires everything that it 
does: once we've got reliable checking of the locks on the connection, 
we can implement a queue on top of that.

One thing which both of these will probably need is a change to the 
remote protocol to allow a connection to specify whether it is writable 
or readonly at the time the connection is opened.  This would allow the 
lock on the database to be checked and obtained at this point for 
writable database, rather than waiting for the connection to attempt a 
write operation.

In theory, Xapian itself is meant to prevent there being two instances 
of a Writable database for the same path in existence concurrently: 
however, the tcpserver avoids the check for this by opening the database 
in its server process, and then passing it through a fork to the 
sub-processes.  I don't think there's any way we can check for this 
state in the core Xapian library, so the tcpserver itself needs to 
enforce the single-writer constraint.

I'm not sure what happens for the windows variant of the server (which 
is a threaded implementation), but I imagine that there the same 
instance of the database is accessed by each connection: if this is 
correct, there are likely to be other problems since the database is not 
safe for concurrent access.  A threaded implementation needs to create a 
new instance of the database for each thread.

 > I'm sure there are both other
> options and opinions on this topic, so I'm soliciting all feedback on
this,
> with the intention of opening a xapian bug to track the status, and
> ultimately end up with a patch.
Opening a bug sooner than later would probably be advisable, so that the 
history of this discussion is easy to find in future.

In particular, discussion of this bug may make it clear whether it 
should block the 1.0.0 release, and if so we

-- 
Richard

On Tue, May 01, 2007 at 12:36:54PM +1000, Mark Hammond
wrote:
> The problem is that a xapian tcp-server in 'writable' mode makes no
attempt
> to ensure only one 'active' connection at a time is trying to
modify the
> database.  If multiple connections are made to a writable server, the
> behaviour is undefined (or even it is was defined, it is unlikely to be
> defined in a way that would make it useful).
I'd not appreciated this happened from the previous discussion - this
is certainly a bug.  I understood the issue was just that of trying to
marshal multiple processes wanting to write to the same remote server
in a sane way.

Looking at the code, I believe it's also wrong that we open the database
and then fork multiple processes which can make use of it, even for a
read-only Database.  We certainly don't promise that you can use the same
Xapian object from different threads.  I think similar rules ought to
apply over fork.

But this matters much more for writers - with the current backends, it
happens to work OK for readers I think.

So we should probably leave the reader issue for now, as it can be fixed
without API or ABI changes, but fix the writer issue.

Cheers,
    Olly