Xapian::WritableDatabase: commit changes depending on the buffer size

john veter
Sat, 06 Feb 2016 16:39:15 -0000

Hi.  I have a lot of documents with different filesize. While indexing, i call
commit() every 1000 documents (this is the default value, user can change it).
The problem is the following: the indexing process runs  smoothly while indexing
small files. The indexer uses about half of the available RAM. But one moment it
hits a bunch of bigger documents.  As a result, the RAM usage increases
drastically. Finally, i just run out of memory.

I think that the solution is to call commit() depending on the buffer actual
size (in megabytes), but not based on the number of the indexed documents. So,
is there any way to estimate the size of the buffer of the
Xapian::WritableDatabase object?

P.S.  may be somebody have other suggestions how to solve the problem?