Xapian discuss - Nov 2006 - UTF-8: what is done and what is not?

I'm aware of the UTF-8 branch here: 
http://www.oligarchy.co.uk/xapian/branches/utf8/ , but I'd like more 
information about what it contains and if it's enough for me.

I not not using omega, nor stemmers. I use Xapian from PHP as a replacement 
to MySQL full text search. The data submited by users is indexed live in 
Xapian. Users can search the data, on the search page, using (currently) 
limited syntax:

[ word1 ] => return data containing "word1 "
[ word1 word2 ] => return data containing "word1" AND
"word2"
[ "word1 word2" ] => return data containing the phrase "word1
word2"

No fancy stemmers, no result scoring, etc. But I do need to support UTF-8.

Currently, I wrote my own word spliter to index the data and my own 
queryparser. They are not perfect and I would like to use built-in Xapian 
objects instead.

So what exactly is available now in Xapian?

This is a thread I started some time ago: 
http://lists.tartarus.org/pipermail/xapian-discuss/2006-February/001674.html 
about the same topic.

Sorry in advance if you already answered those questions!

And thanks again for your really appreciated work.

JL

On Thu, Nov 02, 2006 at 08:46:36AM -0500, tata 668
wrote:
> I'm aware of the UTF-8 branch here: 
> http://www.oligarchy.co.uk/xapian/branches/utf8/ , but I'd like more 
> information about what it contains and if it's enough for me.
The current status is summarised here:

http://wiki.xapian.org/Utf8Support

I'm in the process of turning the release handle for 0.9.8 (to fix
various minor problems reported since 0.9.7), so I'm very close to
merging the utf-8 branch in and the rate of visible progress should pick
up.
> Currently, I wrote my own word spliter to index the data and my own 
> queryparser. They are not perfect and I would like to use built-in Xapian 
> objects instead.
There's not currently a word splitter in the core library, but
Xapian::QueryParser now works in utf-8 on the branch, so you can
probably use that now.

I've not tested utf-8 from any of the bindings yet.  Some languages
standardise on a particular internal representation, so there could
be issues here (I don't know how PHP handles such issues).  But I'd
certainly encourage you to try it and let us know if it works or if
there are problems.

Cheers,
    Olly