Xapian devel - Mar 2012 - GSoC 2012: Erlang Bindings

Hello,

I am Uvarov Michael.
I am studding at Southern Federal University in Russia from 2008.
My github profile is here: https://github.com/freeakk
I want to implement a binding as an Erlang driver. I already have an
experience with NIFs (Native Implemented Functions)
(Code is here https://github.com/freeakk/i18n).

I implemented basic operations with Unicode both in pure Erlang and as NIF.
I think, that Erlang will be a good choice for binding, because it has a nice
inter-node communication interface. This interface can be used as a transport
for access to a remote database.

I have few questions and ideas about Xapian:

First of all, how does Xapian handle concurrent access to shared resources?
How many connections can be established to one database for reading and writing?
Who does coordinate parallel access to same database?
Are opening and closing a database expensive operations or not?
Is a minimal example of using Xapian with a real data set?
Are there datasets for an efficiently testing?

I think it is a good idea for implementing this binding as a linked-in driver.
A driver provides a queue and Xapian classes are not thread-safe. So, it will
be a mapping: one driver port -- one opening connection to a base.
NIFs will require an implementing of a thread pool in C, but it is not
a trivial
task.
Calls of NIFs can block a schedule of an virtual machine. It is also not good,
because other green processes will wait.

I will wait for answers and your questions.

--
Best regards,
Uvarov Michael

Hi Michael,

On Sun, Mar 18, 2012 at 08:50:32PM +0300, Michael Uvarov
wrote:
> I have few questions and ideas about Xapian:
> 
> First of all, how does Xapian handle concurrent access to shared resources?
> How many connections can be established to one database for reading
> and writing?
One writer.  The number of readers is only limited by OS resource
limits.

The way the Btree update work is that readers can detect if a writer
has overwritten the snapshot they are working on.
> Who does coordinate parallel access to same database?
There is no daemon or other central authority.

Writer locking is done using fcntl, with the lock held by a child
process to protect us from some unhelpful semantics of fcntl.
> Are opening and closing a database expensive operations or not?
Opening to read is pretty cheap.  Opening to write a bit more expensive.
Closing is cheap, except for a writer it will implicitly flush any
pending changes (unless you're in a transaction).  It's the flush which
is the expensive operation though, not the actual closing.
> Is a minimal example of using Xapian with a real data set?
There's an example of indexing museum exhibits in the getting started
guide:

http://getting-started-with-xapian.readthedocs.org/
> Are there datasets for an efficiently testing?
There's a "perftest" which was aimed to test performance, but it
uses
randomly synthesised data, so the results may not translate well to real
situations.

We don't have anything set up with real data.  It would certainly be
good to, and could certainly be part of the project if you wanted.
> I think it is a good idea for implementing this binding as a linked-in
driver.
> A driver provides a queue and Xapian classes are not thread-safe.
I think "not thread-safe" is not a totally helpful way to think about
it
as it tends to suggest more issues than there are.  In particular,
everything should be re-entrant (there's no global state).

It's concurrent accesses to the same C++ object which aren't guaranteed
to be supported.  But, for example, you can open the same database in
several different threads at the same time (as those are different C++
Database objects).
> So, it will
> be a mapping: one driver port -- one opening connection to a base.
> NIFs will require an implementing of a thread pool in C, but it is not
> a trivial
> task.
> Calls of NIFs can block a schedule of an virtual machine. It is also not
good,
> because other green processes will wait.
I don't know enough about Erlang to comment on this, but Lenz may have
some thoughts.

Cheers,
    Olly