Rails - Oct 2005 - Ferret - Supported languages

Hi,

A few questions about available languages by ferret or lucene
1. what are the languages currently supported by the engine? Only english?
2. is there any plan to have western european languages like french, 
german, or italian?
3. is there any plan to have hungarian? Or does it exist already?

Thank you,
-- 
   Jean-Etienne Durand
   Mail: etienne dot durand at mail dot com
   Blog: http://spaces.msn.com/members/jetienne

On the pure Lucene side of things, it stores text entirely as Unicode  
(for the most part - the Kinosearch guy has identified that it''s the  
Java version of Unicode, but for all intents and purposes it''s  
Unicode).  So indexing and storing text containing any character  
possible works just fine.   So there is no problem indexing and  
searching for any of the languages you mention.

The complexity comes in how you _analyze_ the text to extract words.   
Lots of rhetorical questions... what to do about accented  
characters?  Flatten them to the standard unaccented alphabet?  Do  
you need stemming?   What about handling of transliterations and  
spell checking?

One good test - try out the StandardAnalyzer with some Hungarian text  
to see what happens.   To do this, the code from Lucene in Action can  
be downloaded and tried out easily (as long as you have Ant and a JDK  
- see the README for details) from http://www.lucenebook.com

Here''s a sample output:

$ ant AnalyzerDemo
Buildfile: build.xml

AnalyzerDemo:
      [echo]
      [echo]       Demonstrates analysis of sample text.
      [echo]
      [echo]       Refer to the "Analysis" chapter for much more on
this
      [echo]       extremely crucial topic.
      [echo]
     [input] Press return to continue...

     [input] String to analyze: [This string will be analyzed.]

      [echo] Running lia.analysis.AnalyzerDemo...
      [java] Analyzing "This string will be analyzed."
      [java]   WhitespaceAnalyzer:
      [java]     [This] [string] [will] [be] [analyzed.]

      [java]   SimpleAnalyzer:
      [java]     [this] [string] [will] [be] [analyzed]

      [java]   StopAnalyzer:
      [java]     [string] [analyzed]

      [java]   StandardAnalyzer:
      [java]     [this] [string] [will] [be] [analyzed]

      [java]   SnowballAnalyzer:
      [java]     [this] [string] [will] [be] [analyz]

      [java]   SnowballAnalyzer:
      [java]     [this] [string] [wil] [be] [analyzed]

      [java]   SnowballAnalyzer:
      [java]     [thi] [string] [will] [be] [analyz]


BUILD SUCCESSFUL
Total time: 10 seconds



On 26 Oct 2005, at 07:44, Jean-Etienne Durand wrote:
> Hi,
>
> A few questions about available languages by ferret or lucene
> 1. what are the languages currently supported by the engine? Only  
> english?
> 2. is there any plan to have western european languages like  
> french, german, or italian?
> 3. is there any plan to have hungarian? Or does it exist already?
>
> Thank you,
> -- 
>   Jean-Etienne Durand
>   Mail: etienne dot durand at mail dot com
>   Blog: http://spaces.msn.com/members/jetienne
> _______________________________________________
> Rails mailing list
> Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
>

Here are my answers for Ferret.

On 26 Oct 2005, at 07:44, Jean-Etienne Durand wrote:
> > A few questions about available languages by ferret or lucene
> > 1. what are the languages currently supported by the engine? Only
> > english?

Currently yes. Unfortunately, in the Regexp library in ruby 1.8, [:alpha:]
only matches the ascii alphabet. That is going to change in the next version
of Ruby with the Oniguruma Regexp library so tokenizers will be easier to
create then.

Having said that, it is really simple to create tokenizer in Ferret. If you
can describe a token (word) in your target language with a ruby regular
expression, you can create a tokenizer with a few lines of code.

 class IncompleteGermanTokenizer < Ferret::Analysis::RegExpTokenizer
def token_re() /[A-Za-zÄäéÖöÜüß/ end
end

Of course, if you want to do more with the next like stemming or converting
to ascii you''ll need to do a little more than that. But The Analysis
module
is definitely the easiest part of the library to understand. To see what the
Standard Analyzer does to text just try;

ruby /usr/lib/ruby/gems/1.8/gems/ferret-0.1.3
/lib/ferret/analysis/standard_tokenizer.rb

You''ll need to adjust the path for your machine.
> 2. is there any plan to have western european languages like
> > french, german, or italian?
> 3. is there any plan to have hungarian? Or does it exist already?
>
I''d definitely like people to contribute the code to support these
languages.

Dave


_______________________________________________
Rails mailing list
Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org
http://lists.rubyonrails.org/mailman/listinfo/rails