search for: 8108523ccd53

...tter is valid, just removing this block (or removing the parts of it which are Lu or Ll) should fix the problem as then tokenisation will switch mode - I tried this and it fixes your case at least: diff --git a/xapian-core/queryparser/word-breaker.cc b/xapian-core/queryparser/word-breaker.cc index 8108523ccd53..4fabc23f4b56 100644 --- a/xapian-core/queryparser/word-breaker.cc +++ b/xapian-core/queryparser/word-breaker.cc @@ -103,7 +103,7 @@ is_unbroken_script(unsigned p) // FE30..FE4F; CJK Compatibility Forms 0xFE30 - 1, 0xFE4F, // FF00..FFEF; Halfwidth and Fullwidth Forms - 0xFF00 - 1, 0xFFEF, + /...

...e halfwidth Katakana and Hangul codepoints as unbroken script and treats Latin characters, numbers, symbols and punctuation as broken script. There's a couple of unit tests that check for this. diff --git a/xapian-core/queryparser/word-breaker.cc b/xapian-core/queryparser/word-breaker.cc index 8108523ccd53..6122dcdccc97 100644 --- a/xapian-core/queryparser/word-breaker.cc +++ b/xapian-core/queryparser/word-breaker.cc @@ -102,8 +102,10 @@ is_unbroken_script(unsigned p) 0xF900 - 1, 0xFAFF, // FE30..FE4F; CJK Compatibility Forms 0xFE30 - 1, 0xFE4F, - // FF00..FFEF; Halfwidt...

I think I found a bug in Xapian 1.5 when using FLAG_WORD_BREAKS for input that contains characters in Unicode Halfwidth and Fullwidth Forms (https://unicode.org/charts/PDF/UFF00.pdf). Since I am undecided yet if and how to fix this in Xapian I haven't come up with a pull request. Because trac currently is offline, I could not file a bug. I hope it's OK to post my analysis here first,