Hi guys I have implemented xapian on a website, and it currently has about 2M items in its index. Its all been working quite nicely so far, until I tried removing some old items from the index (removing items when the index was smaller was no problems at all). When I try to remove them now (using writabledatabase_delete_document() via php), it halfway freezes up the machine, and the apache httpd runs amok spawning more and more children, until I break the php script that is trying to remove documents from xapian.>From my laymens point of view, it seems that the xapian delete documentfunction freezes up the OS on a filesystem level. Is this a correct assessment? Any ideas on how I can get xapian _not_ to freeze up the system like this when deleting documents? I'm using php 5.2.0, and xapian 0.9.9. This is a 'ls -lha' of the xapian index dir: total 1.3G drwxr-xr-x 2 malte users 408 Dec 1 17:22 . drwxr-xr-x 16 malte users 400 Nov 15 20:21 .. -rw------- 1 malte users 0 Dec 1 17:22 db_lock -rw-r--r-- 1 malte users 10 Nov 24 00:39 meta -rw-r--r-- 1 malte users 468M Dec 1 17:22 position_DB -rw-r--r-- 1 malte users 7.4K Dec 1 16:46 position_baseB -rw-r--r-- 1 malte users 324M Dec 1 17:23 postlist_DB -rw-r--r-- 1 malte users 5.1K Dec 1 16:46 postlist_baseB -rw-r--r-- 1 malte users 61M Dec 1 17:22 record_DB -rw-r--r-- 1 malte users 956 Dec 1 16:46 record_baseB -rw-r--r-- 1 malte users 339M Dec 1 17:22 termlist_DB -rw-r--r-- 1 malte users 5.4K Dec 1 16:46 termlist_baseB -rw-r--r-- 1 malte users 99M Dec 1 17:22 value_DB -rw-r--r-- 1 malte users 1.6K Dec 1 16:46 value_baseB Thanks for your help. Alec
Alexander Lind
2006-Dec-01 17:44 UTC
[Xapian-devel] Re: writabledatabase_delete_document()
I should add that I use xapians own document_id as the argument to writabledatabase_delete_document(). I have made it so the indexing part of my script saves the xapian document_id in the sql db. I just read a post somewhere on the net about how you can use term names to ID and delete items out of an xapian index instead of using a document_id. Is this faster? (would seem strange if it was). Alec Alexander Lind wrote:> Hi guys > > I have implemented xapian on a website, and it currently has about 2M > items in its index. > > Its all been working quite nicely so far, until I tried removing some > old items from the index (removing items when the index was smaller was > no problems at all). > > When I try to remove them now (using writabledatabase_delete_document() > via php), it halfway freezes up the machine, and the apache httpd runs > amok spawning more and more children, until I break the php script that > is trying to remove documents from xapian. > > >From my laymens point of view, it seems that the xapian delete document > function freezes up the OS on a filesystem level. Is this a correct > assessment? > > Any ideas on how I can get xapian _not_ to freeze up the system like > this when deleting documents? > > I'm using php 5.2.0, and xapian 0.9.9. > > This is a 'ls -lha' of the xapian index dir: > total 1.3G > drwxr-xr-x 2 malte users 408 Dec 1 17:22 . > drwxr-xr-x 16 malte users 400 Nov 15 20:21 .. > -rw------- 1 malte users 0 Dec 1 17:22 db_lock > -rw-r--r-- 1 malte users 10 Nov 24 00:39 meta > -rw-r--r-- 1 malte users 468M Dec 1 17:22 position_DB > -rw-r--r-- 1 malte users 7.4K Dec 1 16:46 position_baseB > -rw-r--r-- 1 malte users 324M Dec 1 17:23 postlist_DB > -rw-r--r-- 1 malte users 5.1K Dec 1 16:46 postlist_baseB > -rw-r--r-- 1 malte users 61M Dec 1 17:22 record_DB > -rw-r--r-- 1 malte users 956 Dec 1 16:46 record_baseB > -rw-r--r-- 1 malte users 339M Dec 1 17:22 termlist_DB > -rw-r--r-- 1 malte users 5.4K Dec 1 16:46 termlist_baseB > -rw-r--r-- 1 malte users 99M Dec 1 17:22 value_DB > -rw-r--r-- 1 malte users 1.6K Dec 1 16:46 value_baseB > > > Thanks for your help. > Alec > >