Xapian discuss - Jun 2004 - Threaded test (in C++) to reproduce our database problems

> This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.

--B_3170265892_25193571
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit

Okay, we finally broke down and write a quickie threaded test in C++ to
reproduce the database problems we've been having.  On my dual-processor Mac
running OS X, this test fails every time.

The code is below, and should be fairly straight forward.  1 writer thread
that constantly add's documents to a Xapian::WritableDatabase, and a bunch
of reader threads that execute queries.

Ignoring the lack of synchronization around stderr/stdout, here's the
output:

--------------------------------------------
$ ./build/xapian_threads
Writer:  flush()
Reader: started
RRReeeaaadddeeerrrs::   ssstttaaarrrttteeeddd


Reader:  ERROR=Error reading all bytes: Bad file descriptor (fd=7)
Reader:  ERROR=Error reading block 40: Bad file descriptor (fd=5)
Writer:  flush()
Reader:  ERROR=Db block overwritten
Writer:  ERROR=Db block overwritten
Writer thread unexpectedly exited...
--------------------------------------------

So you can see, our reader threads bomb out, then the writer thread dies
too.  :(

Any ideas?  We wrote this little C++ test pretty quickly (could be buggy),
but it reproduces what we're seeing from the Java-side.

eric


--B_3170265892_25193571
Content-type: application/octet-stream; name="xapian_threads.cpp"
Content-disposition: attachment;
	filename="xapian_threads.cpp"
Content-transfer-encoding: base64

I2luY2x1ZGUgPHhhcGlhbi5oPgojaW5jbHVkZSA8cHRocmVhZC5oPgojaW5jbHVkZSA8dW5p
c3RkLmg+CiNpbmNsdWRlIDxpb3N0cmVhbT4KI2luY2x1ZGUgInhhcGlhbl90aHJlYWRzLmgi
CgojZGVmaW5lIERBVEFCQVNFICIvdG1wL3hhcGlhbl90aHJlYWRzLmRiIgojZGVmaW5lIE1B
WF9USFJFQURTIDMKCnVzaW5nIG5hbWVzcGFjZSBzdGQ7CgpwdGhyZWFkX3QgX3dyaXRlcjsK
cHRocmVhZF90IF9yZWFkZXJzW01BWF9USFJFQURTXTsKCnZvaWQgZG9fd3JpdGVzKCkgewog
ICAgdW5saW5rKERBVEFCQVNFIi9kYl9sb2NrIik7CglYYXBpYW46OldyaXRhYmxlRGF0YWJh
c2UgZGIgPSBYYXBpYW46OkF1dG86Om9wZW4oREFUQUJBU0UsIFhhcGlhbjo6REJfQ1JFQVRF
X09SX09WRVJXUklURSk7CgljaGFyICpybmQgPSAoY2hhciAqKSBtYWxsb2MoMjU1KTsKCW1l
bXNldChybmQsIDAsIDI1NSk7Cglsb25nIGNudCA9IDA7CgkKCXRyeSB7CgkJLy8gZW5kbGVz
c2x5IGFkZCBhIDIwMDAgdGVybSBkb2N1bWVudCB0byB0aGUgZGF0YWJhc2UKCQl3aGlsZSAo
dHJ1ZSkgewoJCQlYYXBpYW46OkRvY3VtZW50IGRvYzsKCQkJaW50IGk9MDsKCgkJCWZvciAo
OyBpPDEwMDA7IGkrKykgewoJCQkJZG9jLmFkZF9wb3N0aW5nKCJyYW5kb20iLCBpKTsKCgkJ
CQlzcHJpbnRmKHJuZCwgIiVkIiwgcmFuZCgpKTsKCQkJCWRvYy5hZGRfcG9zdGluZyhybmQs
IGkpOwoJCQl9CgkJCgkJCWRiLmFkZF9kb2N1bWVudChkb2MpOwoJCQkKCQkJaWYgKCsrY250
ICUgMTAgPT0gMCkgewoJCQkJLy8gZmx1c2ggZXZlcnkgMTAgZG9jdW1lbnRzCgkJCQlkYi5m
bHVzaCgpOwoJCQkJY291dCA8PCAiV3JpdGVyOiAgZmx1c2goKSIgPDwgZW5kbDsKCQkJfQoJ
CX0KCX0gY2F0Y2ggKFhhcGlhbjo6RXJyb3IgJmVycikgewoJCWNlcnIgPDwgIldyaXRlcjog
IEVSUk9SPSIgPDwgZXJyLmdldF9tc2coKSA8PCBlbmRsOwoJfQoJY2VyciA8PCAiV3JpdGVy
IHRocmVhZCB1bmV4cGVjdGVkbHkgZXhpdGVkLi4uIiA8PCBlbmRsOwoJcHRocmVhZF9leGl0
KE5VTEwpOwp9Cgp2b2lkIGRvX3JlYWRzKCkgewoJY291dCA8PCAiUmVhZGVyOiBzdGFydGVk
IiA8PCBlbmRsOwoJdHJ5IHsKCQkvLyBjcmVhdGUgdGhlIHJlYWRlciBkYXRhYmFzZSBvbmNl
CgkJLy8gYW5kIGNvbnRpbnVlIHRvIHF1ZXJ5IGl0LgoJCVhhcGlhbjo6RGF0YWJhc2UgZGIg
PSBYYXBpYW46OkF1dG86Om9wZW4oREFUQUJBU0UpOwoJCQoJCXdoaWxlICh0cnVlKSB7CgkJ
CS8vIE5PVEU6ICBDcmVhdGluZyB0aGUgZGF0YWJhc2UgaGVyZSBjYXVzZXMgdGhlIHRlc3QK
CQkJLy8gdG8gZmFpbCB0b28sIGJ1dCBpbiBkaWZmZXJlbnQgYW5kIG15c3RlcmlvdXMgd2F5
cwoJCQlYYXBpYW46OkVucXVpcmUgZW5xKGRiKTsKCQkJWGFwaWFuOjpRdWVyeSBxdWVyeSgi
cmFuZG9tIik7CgkJCVhhcGlhbjo6TVNldCBzZXQ7CgkJCQoJCQllbnEuc2V0X3F1ZXJ5KHF1
ZXJ5KTsKCQkJc2V0ID0gZW5xLmdldF9tc2V0KDAsIDI1MDApOwoJCQkKCQkJLy8gd2FsayB0
aGUgcmVzdWx0cyB0byBleGVyY2l6ZSB0aGUgbXNldCBpdGVyYXRvcgoJCQlmb3IgKFhhcGlh
bjo6TVNldEl0ZXJhdG9yIGkgPSBzZXQuYmVnaW4oKTsgaSAhPSBzZXQuZW5kKCk7ICsraSk7
CgkJfQoJfSBjYXRjaCAoWGFwaWFuOjpFcnJvciAmZXJyKSB7CgkJY2VyciA8PCAiUmVhZGVy
OiAgRVJST1I9IiA8PCBlcnIuZ2V0X21zZygpIDw8IGVuZGw7Cgl9CgkKCXB0aHJlYWRfZXhp
dChOVUxMKTsKfQoKdm9pZCBzZXR1cF93cml0ZXJfdGhyZWFkKCkgewoJcHRocmVhZF9jcmVh
dGUoJl93cml0ZXIsIE5VTEwsIGRvX3dyaXRlcywgTlVMTCk7Cn0KCnZvaWQgc2V0dXBfcmVh
ZGVyX3RocmVhZHMoKSB7CglpbnQgaSA9IDA7CgkKCWZvciAoOyBpPE1BWF9USFJFQURTOyBp
KyspIHsKCQlpbnQgc3VjY2VzcyA9IHB0aHJlYWRfY3JlYXRlKCYoX3JlYWRlcnNbaV0pLCBO
VUxMLCBkb19yZWFkcywgTlVMTCkgPT0gMDsKCQkKCQlpZiAoIXN1Y2Nlc3MpIHsKCQkJY2Vy
ciA8PCAic2V0dXBfcmVhZGVyX3RocmVhZHM6ICBDb3VsZCBub3QgY3JlYXRlIHRocmVhZCAj
IiA8PCBpIDw8IGVuZGw7CgkJfQoJfQp9CgppbnQgbWFpbihpbnQgYXJndiwgY2hhciAqKmFy
Z2MpIHsKCWludCBpID0gMDsKCQoJc2V0dXBfd3JpdGVyX3RocmVhZCgpOwoJLy8gbGV0IHRo
ZSB3cml0ZXIgdGhyZWFkIGdldCBnb2luZyBiZWZvcmUgCgkvLyB3ZSBzdGFydCBhIGJ1bmNo
IG9mIHJlYWRlcnMKCXNsZWVwKDIpOyAgIAoJCgkKCXNldHVwX3JlYWRlcl90aHJlYWRzKCk7
Cgljb3V0IDw8ICJSZWFkZXJzIHN0YXJ0ZWQiIDw8IGVuZGw7CgkKCS8vIHNpdCBhbmQgd2Fp
dCBmb3IgZXZlcnl0aGluZyB0byBmaW5pc2gKCS8vIGluIGEgcGVyZmVjdCB3b3JsZCwgdGhl
eSdsbCBuZXZlciBmaW5pc2gKCXB0aHJlYWRfam9pbihfd3JpdGVyLCBOVUxMKTsJCglmb3Ig
KDsgaTxNQVhfVEhSRUFEUzsgaSsrKSB7CgkJcHRocmVhZF9qb2luKF9yZWFkZXJzW2ldLCBO
VUxMKTsKCX0KCQoJcmV0dXJuIDA7Cn0KCg=
--B_3170265892_25193571--

On Wed, Jun 16, 2004 at 09:24:50PM -0400, Eric B. Ridge
wrote:
> Okay, we finally broke down and write a quickie threaded test in C++ to
> reproduce the database problems we've been having.  On my
dual-processor Mac
> running OS X, this test fails every time.
> 
> The code is below, and should be fairly straight forward.  1 writer thread
> that constantly add's documents to a Xapian::WritableDatabase, and a
bunch
> of reader threads that execute queries.
Great - this should be very helpful.
> Reader:  ERROR=Error reading all bytes: Bad file descriptor (fd=7)
That's very odd.  That's either a Btree base file, or the meta file.
> Reader:  ERROR=Error reading block 40: Bad file descriptor (fd=5)
And a DB file this time.
> Writer:  flush()
> Reader:  ERROR=Db block overwritten
That's less odd.  After the writer has flushed twice, readers will get
this exception - they need to call reopen() and restart the operation.
> Writer:  ERROR=Db block overwritten
But the writer thread shouldn't get this unless there's another writer.
> Any ideas?  We wrote this little C++ test pretty quickly (could be buggy),
> but it reproduces what we're seeing from the Java-side.
I can't see anything wrong from a quick read through.  I'm right in the
middle of something right now, but I'll give this a whirl in the next
day or so.

Cheers,
    Olly

Compiling now.  I'll report back in a few.  Been up all night dealing with a
(scheduled) power-outage in our datacenter.  Always fun to see which servers
won't boot.

Hopefully, CVS HEAD will fix this, but as another data-point:  The test (poorly
written as it is) also fails on our test Gentoo system, which is a dual
processor hyperthreaded Xeon, running the 2.6.6 Linux kernel.  So it ain't
just my Mac.

As an aside, is it necessary to compile Xapian with -pthread or -lpthread or
-D_REENTRANT or -D_THREAD_SAFE or something?  My thinking is no, but I'm
just a novice when it comes to pthreads.

I really appreciate the time y'all have spent looking into this.  A welcomed
surprise when power came back on!

eric

-----Original Message-----
From:	Olly Betts [mailto:olly@survex.com]
Sent:	Thu 6/17/2004 8:28 AM
To:	xapian-discuss@lists.xapian.org
Cc:=09
Subject:	Re: [Xapian-discuss] Threaded test (in C++) to reproduce our database
problems
On Thu, Jun 17, 2004 at 12:52:36PM +0100, Olly Betts
wrote:
> On Thu, Jun 17, 2004 at 12:25:48PM +0100, Olly Betts wrote:
> > OK, I can reproduce this with vanilla 0.8.0, on x86 using 2.4.24. 
It's
> > a 4 CPU box - perhaps that makes a difference, or maybe CVS HEAD fixes
> > it.  That's my next thing to try.
>=20
> Still happens with CVS HEAD.
Actually, the dynamic linker was picking up the wrong version of the
shared library.  It looks like CVS HEAD fixes this.

Eric - can you try a CVS snapshot, and let us know if that fixes this.
You can get them from:

http://www.oligarchy.co.uk/xapian/HEAD/

Cheers,
    Olly

_______________________________________________
Xapian-discuss mailing list
Xapian-discuss@lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-discuss