Xapian discuss - Mar 2023 - manual flushing thresholds for deletes?

Olly Betts <olly at survex.com> wrote:
> On Fri, Mar 24, 2023 at 10:37:41AM +0000, Eric Wong wrote:
> > Realizing I had documents of hugely varying sizes (0.5KB..20MB)
> > and little RAM, I instead tracked the number of raw bytes in the
> > text being indexed and flushed whenever I'd seen a configurable
> > byte count.  Not the most scientific way, but it seems to work
> > well enough on low-end systems.
> > 
> > Now, I'm dealing with many deletes at once and hitting OOM
> > again.  Since the raw text is no longer available and I didn't
> > store its original size anywhere, would calculating something
> > based on get_doclength be a reasonable approximation?
> > 
> > I'm wondering if something like:
> > 
> >   get_doclength * (1 + 3) * mean_term_length
> > 
> >   where 1x is for the mean term length itself,
> >   and 3x for the position overhead
> 
> If I follow you want an approximation to the number of raw bytes in the
> text to match the non-delete case, so I think you want something like:
> 
> get_doclength() / 2 * (mean_word_length + 1)
> 
> The /2 is assuming you're indexing both stemmed and unstemmed terms
> since with the default indexing strategy one word in the document
> generates one of each.
> 
> The +1 is for the spaces between words in the text.  This is
> likely to underestimate due to punctuation and runs of whitespace,
> So perhaps +1.<something> is better (and perhaps better to
overestimate
> slightly and flush a little more often rather than risk OOM).
Thanks for the response.
> > And perhaps assume mean_term_length is 10 bytes, so maybe:
> > 
> >   get_doclength * 40
> 
> 10 seems too long.  You want the mean word length weighted by frequency
> of occurrence.  For English that's typically around 5 characters, which
> is 5 bytes.  If we go for +1 that's:
Actually, 10 may be too short in my case since there's a lot of
40-byte SHA-1 hex (and likely SHA-256 in the future) from git; and
also long function names, etc...

Without capitalized prefixes, I get a mean length from delve as 15.3571:

	xapian-delve -a -1 . | tr -d A-Z \
	  awk '{ d = length - mean; mean += d/NR } END { print mean }'

(that awk bit should be overflow-free)
> > I'm using Search::Xapian XS since it's in Debian stable;
> > and don't think there's a standard way to show the amount
> > of memory uncommitted changes are taking up.
> 
> We don't have an easy way to calculate this in any version.  It would
> need us to take more control of the allocation of the memory used to
> store these changes.  We probably need to do that as the threshold
> here really should be in terms of memory used to store the pending
> changes, but it's not a trivial change.
Understood.

On Mon, Mar 27, 2023 at 11:22:09AM +0000, Eric Wong
wrote:
> Olly Betts <olly at survex.com> wrote:
> > 10 seems too long.  You want the mean word length weighted by
frequency
> > of occurrence.  For English that's typically around 5 characters,
which
> > is 5 bytes.  If we go for +1 that's:
> 
> Actually, 10 may be too short in my case since there's a lot of
> 40-byte SHA-1 hex (and likely SHA-256 in the future) from git; and
> also long function names, etc...
> 
> Without capitalized prefixes, I get a mean length from delve as 15.3571:
> 
> 	xapian-delve -a -1 . | tr -d A-Z \
> 	  awk '{ d = length - mean; mean += d/NR } END { print mean }'
That's not weighted by frequency though, and short words tend to be more
frequent, so you're likely skewing the answer.  Also it'll include
boolean terms which didn't come from the document text.

You can take frequency into account with something like this:

xapian-delve -avv1 .|tr -d A-Z|awk '{t + = length($1)*$3; n += $3} END
{print t/n}'

This will also effectively ignore boolean terms, assuming you're giving
them wdf of 0 (because $3 here is the collection frequency, which is
sum(wdf(term)) over all documents).
> (that awk bit should be overflow-free)
I don't see how to do the above as a rolling mean, so to be accurate for
a large database it seems with awk you'll need to make two passes over
the data - one to calculate `n`, then use that in a second pass like you
do with NR.  Or use a language which supports arbitrary precision
numbers.

Cheers,
    Olly