Xapian discuss - May 2008 - MatchSpy:ing on a large recordset

Hi All

I have a project in the works that will have a 10-15M records with a  
set of arbitrary attributes on each record.

I need to build a system where a user can filter the recordset by  
selecting attribute values and/or negating on them, and for each  
attribute value given, the amount of matching records needs to be  
calculated in realtime - 1-2 seconds lookup time is acceptable.

I must also be able to update the attributes of large sets of records  
without making the system fall over.

Can this be achieved with Xapian and the MatchSpy functionality?

I will have a pretty powerful machine for this project, so if a large  
amount of RAM etc is OK.

Cheers
Alec

On Wed, May 21, 2008 at 11:03:04PM -0700, alexander lind
wrote:
> I have a project in the works that will have a 10-15M records with a  
> set of arbitrary attributes on each record.
> 
> I need to build a system where a user can filter the recordset by  
> selecting attribute values and/or negating on them, and for each  
> attribute value given, the amount of matching records needs to be  
> calculated in realtime - 1-2 seconds lookup time is acceptable.
For the filtering options you describing, making each attribute a
term prefix and filtering on those terms would be the most efficient
approach I think.
> Can this be achieved with Xapian and the MatchSpy functionality?
You certainly could do it this way.  If there's enough RAM to cache
all the value data, you'll probably at least be near the performance
target, but without trying it I couldn't say for sure.  Using C++ here
is likely to help - calling from C++ to a scripting language and back
tens of millions of times will probably be a measurable overhead.

Cheers,
    Olly