search for: fe4f

...at least: diff --git a/xapian-core/queryparser/word-breaker.cc b/xapian-core/queryparser/word-breaker.cc index 8108523ccd53..4fabc23f4b56 100644 --- a/xapian-core/queryparser/word-breaker.cc +++ b/xapian-core/queryparser/word-breaker.cc @@ -103,7 +103,7 @@ is_unbroken_script(unsigned p) // FE30..FE4F; CJK Compatibility Forms 0xFE30 - 1, 0xFE4F, // FF00..FFEF; Halfwidth and Fullwidth Forms - 0xFF00 - 1, 0xFFEF, + //0xFF00 - 1, 0xFFEF, // 1AFF0..1AFFF; Kana Extended-B // 1B000..1B0FF; Kana Supplement // 1B100..1B12F; Kana Extended-A If we're fixing it this way we should check this...

...: <information> Activation (eth0) Stage 5 of 5 (IP Configure Commit) complete. Jan 15 05:35:35 localhost avahi-daemon[5154]: New relevant interface eth0.IPv6 for mDNS. Jan 15 05:35:35 localhost avahi-daemon[5154]: Joining mDNS multicast group on interface eth0.IPv6 with address fe80::21c:c0ff:fe4f:f2cf. -- Jan 15 05:35:35 localhost dnsmasq[5162]: using nameserver 13.13.13.3#53 Jan 15 05:35:35 localhost avahi-daemon[5154]: Registering new address record for fe80::21c:c0ff:fe4f:f2cf on eth0. Jan 15 05:37:11 localhost NetworkManager: <WARNING> nm_device_802_11_wireless_scan (): could not...

.../queryparser/word-breaker.cc b/xapian-core/queryparser/word-breaker.cc index 8108523ccd53..6122dcdccc97 100644 --- a/xapian-core/queryparser/word-breaker.cc +++ b/xapian-core/queryparser/word-breaker.cc @@ -102,8 +102,10 @@ is_unbroken_script(unsigned p) 0xF900 - 1, 0xFAFF, // FE30..FE4F; CJK Compatibility Forms 0xFE30 - 1, 0xFE4F, - // FF00..FFEF; Halfwidth and Fullwidth Forms - 0xFF00 - 1, 0xFFEF, + // FF00..FF60: Fullwidth Numbers, Latin Characters, Punctuation + // FF61..FF64: Halfwidth Punctuation + 0xFF65 - 1, 0xFFDC, // Halfwidth Katakan...

I think I found a bug in Xapian 1.5 when using FLAG_WORD_BREAKS for input that contains characters in Unicode Halfwidth and Fullwidth Forms (https://unicode.org/charts/PDF/UFF00.pdf). Since I am undecided yet if and how to fix this in Xapian I haven't come up with a pull request. Because trac currently is offline, I could not file a bug. I hope it's OK to post my analysis here first,

...0:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:50:da:4f:79:83 brd ff:ff:ff:ff:ff:ff inet6 fe80::250:daff:fe4f:7983/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:04:75:d5:3e:79 brd ff:ff:ff:ff:ff:ff inet 172.16.0.1/24 brd 172.16.0.255 scope global eth1 inet6 fe80::204:75ff:fed5:3e79/64 scope...

...]="NEMO" [2010/06/08 20:11:43, 2] lib/interface.c:340(add_interface) added interface em0 ip=fe80:1::216:e6ff:fe7f:972e bcast=fe80:1::ffff:ffff:ffff:ffff netmask=ffff:ffff:ffff:ffff:: [2010/06/08 20:11:43, 2] lib/interface.c:340(add_interface) added interface fxp0 ip=fe80:2::2d0:b7ff:fe4f:4629 bcast=fe80:2::ffff:ffff:ffff:ffff netmask=ffff:ffff:ffff:ffff:: [2010/06/08 20:11:43, 2] lib/interface.c:340(add_interface) added interface em0 ip=172.24.0.254 bcast=172.24.0.255 netmask=255.255.255.0 [2010/06/08 20:11:43, 2] lib/interface.c:340(add_interface) added interface fxp0 ip=172...

Hi, I am looking for Chinese Japanese and Korean tokenizer that could can be use to tokenize terms for CJK languages. I am not very familiar with these languages however I think that these languages contains one or more words in one symbol which it make more difficult to tokenize into searchable terms. Lucene has CJK Tokenizer ... and I am looking around if there is some open source that we