Xapian discuss - Jul 2004 - [Scriptindex] Exception: Key too long

Hi,

For our forum searchengine I'm implementing Xapian.
Each topic is a new Document in Xapian.

My scriptindex config file is:

url   : field=url boolean=Q unique=Q
title : field=title weight=3 index index=XT
start : field=start weight=2 index index=XS truncate=200
body  : field=text index
users : boolean=XU
forum : boolean=XF

'Users' are userids which have placed a posting in a topic.
It is a space-separated string with unique userids. 
The problem is that I get this error:  Exception: Key too long: length was
5301 bytes, maximum length of a key is 252 bytes

I already found out that this is because the btree index is getting too
large, but how can I work around this...

I would like to search for userids, but a normal field=users index=XU does
not seem to work for me. After mapping user to XU in my Omega template,
searching for user:1 does not return results.

Regards

On 7-7-2004 11:32, Trebel @ LEdZ wrote:
> url   : field=url boolean=Q unique=Q
> title : field=title weight=3 index index=XT
> start : field=start weight=2 index index=XS truncate=200
> body  : field=text index
> users : boolean=XU
> forum : boolean=XF
Perhaps you'd put that field=start after the truncate=200, otherwise you 
won't get a truncated value (afaik) in that field.
Btw, have a look at the unhtml-command aswell, you may otherwise be 
inserting html, which is normally not really interesting tekst.
> 'Users' are userids which have placed a posting in a topic.
> It is a space-separated string with unique userids. 
> The problem is that I get this error:  Exception: Key too long: length was
> 5301 bytes, maximum length of a key is 252 bytes
Are you sure there are spaces in between?
Anyway, the exception is about a single (unsplittable) term that is much 
longer than 252 characters.

The easiest way to find out which term is too long, is probably to 
adjust the sourcecode of Xapian and have it spit out this too long key 
(or perhaps it does that if you set it in debug mode?? See the HACKING 
in xapian/doc)
> I already found out that this is because the btree index is getting too
> large, but how can I work around this...
No, it's not about that. It's about a single term that is too long, and 
since a term of 5301 characters isn't very common, the key-length of 252 
bytes wasn't raised above that limit (afaik it has something to do with 
the 255 "magic number") ;)
> I would like to search for userids, but a normal field=users index=XU does
> not seem to work for me. After mapping user to XU in my Omega template,
> searching for user:1 does not return results.
Did you put your setmap entirely at the front of the template?
We do it like so:

[here is the documentstart]
$set{stemmer,dutch}$setmap{prefix,title,XC,start,XA,topicstart,XA}
hits="$msize"
[and here is more omega-script code]

The setmap-prefix should be before any call to a few (it is documented 
in the omegascript.txt in the /doc-dir of omega) other commands.

Best regards,

Arjen (ACM)