Xapian discuss - Jun 2007 - Extremely high WDF's

Hi, I'm thinking of using the WDF of terms for something other than 
actual "frequency".

I have to index some pages over time. The actual content of the document 
isn't really that important, but it needs to find a term that may have 
only appeared in a 1minute interval over the 10-year life of the document.

 So I've devised a way to just extract terms, and associated
"life" of
that term, it could be contiguous, it could be popping in and out all 
the time.

Now I want to use the WDF to give more weight to terms that have 
appeared on that page throughout the life of the document, as opposed to 
terms that only appeared briefly. I thought of adding all the seconds 
that the term has appeared on that page, and that could be its WDF.

However, this would give me WDF's well into the millions.

Since it's already been more than 24hours since my last ludicrous idea, 
I'd thought it would be time for another one.

Plan B: normalize the time from 1..N where N is the number of terms that 
have ever appeared on the page and then just assign each term its order 
in that range.

On Wed, Jun 27, 2007 at 04:48:34PM +1000, David Morris-Oliveros
wrote:
> Now I want to use the WDF to give more weight to terms that have 
> appeared on that page throughout the life of the document, as opposed to 
> terms that only appeared briefly. I thought of adding all the seconds 
> that the term has appeared on that page, and that could be its WDF.
> 
> However, this would give me WDF's well into the millions.
You don't seem to actually have asked a question, but I guess you want
to know if this is reasonable, or if it is likely to cause problems.

If you have other terms with more normally set wdfs, the relative
weightings between the terms will probably not be sensible.

But if all the terms have similarly inflated wdfs, I can't really see
any major problems.  The worst I can see is that the document length is
the sum of the wdfs over all, and that is calculated in a 32 bit
unsigned integer.

However, you could presumably just calculate your "term exposed time"
and then scale it down by a constant factor (1000, say), to give smaller
wdfs and document lengths, which would avoid risk of overflow and result
in a smaller database (smaller values are stored in fewer bytes).
> Plan B: normalize the time from 1..N where N is the number of terms that 
> have ever appeared on the page and then just assign each term its order 
> in that range.
I think you'd have to try both approaches to find which (if either!)
gives good results.

Trying to factor in the lifetime of terms is certainly an interesting
idea - I'd love to hear how you get on.

Cheers,
    Olly