[Ferret-talk] Parallel indexing doesn''t work?

Noah M. Daniels
Wed, 09 Jan 2008 21:37:45 -0000

Thanks, Jens. Good catch; this little example works correctly after
fixing that change.

However, my ActsAsFerret index merging does _not_ work, and I''m
wondering if it''s something to do with AAF''s handling of
documents in
an index?

Let''s call my indexed class Company...


Company.find_by_contents(''*'')
=> #<ActsAsFerret::SearchResults:0x2b1699108878 @current_page=nil,
@total_hits=3, @results=[], @total_pages=1, @per_page=3>


yet on each partial index prior to merging, that query would return a
bunch of results as one would expect.

now, here''s how I''ve built that index... any idea why the
merged index
is broken?

module FerretHelpers
def merge_ferret_index_partitions(model)

     model_dir = File.basename(model.aaf_configuration[:ferret][:path])

     final_index_path =
"/tmp/merged_parallel_ferret_index/#{model_dir}"

     partial_index_path = "/tmp/partial_indices/#{model_dir}"

     paths = Dir.glob("#{partial_index_path}/*")

     paths.each do |path|
       i = Ferret::I.new(:path => path, :create => true)
       name = path.split(''/'').last
       i << {:name => name}
       i.close
     end

     readers = []
     paths.each {|path| readers << IndexReader.new(path) }
     index_writer = IndexWriter.new(:path => final_index_path)
     index_writer.add_readers(readers)
     index_writer.close()
     readers.each {|reader| reader.close()}
     index = Ferret::Index::Index.new(:path => final_index_path)
     index.optimize
     index.close


   end
end


On Jan 9, 2008, at 4:24 PM, Jens Kraemer wrote:

> Hi!
>
> seems to me you''re indexing strings starting with
''index'' but you''re
> searching for ''name''? Or maybe correcting this already
was one of your
> minimal changes?
>
> If not, try changing that line:
>> res = i.search(''name*'')
> to
>> res = i.search(''index*'')
>
> cheers,
> Jens
>
> On Wed, Jan 09, 2008 at 04:02:17PM -0500, Noah M. Daniels wrote:
>> Hi,
>>
>> I''m trying to get parallelized ferret indexing working for my
AAF
>> indices, based on the example in the O''Reilly Ferret shortcut.
>> However, the resulting indices after merging seem to have no actual
>> documents.
>>
>> I went and made minimal changes to the example in the Ferret shortcut
>> pdf, and indeed can''t get that to work either. I''d
appreciate any
>> help
>> anyone can give! Thanks!
>>
>> The example is below:
>>
>> #!/usr/bin/env ruby
>>
>> require ''rubygems''
>> require ''ferret''
>> include Ferret::Index
>>
>> 5.times do |i|
>>   name = "index#{i}"
>>   puts name
>>   i = Ferret::I.new(:path => "/tmp/#{i}", :create =>
true)
>>   i << {:name => name}
>>   i.close
>> end
>> readers = []
>> readers << IndexReader.new("/tmp/0")
>> readers << IndexReader.new("/tmp/1")
>> readers << IndexReader.new("/tmp/2")
>> readers << IndexReader.new("/tmp/3")
>> readers << IndexReader.new("/tmp/4")
>> index_writer = IndexWriter.new(:path => "/tmp/test")
>> index_writer.add_readers(readers)
>> index_writer.close()
>> readers.each {|reader| reader.close()}
>> i = Ferret::I.new(:path => ''/tmp/test'')
>> res = i.search(''name*'')
>> puts res.inspect # gives me: #<struct Ferret::Search::TopDocs
>> total_hits=0, hits=[], max_score=0.0,
>> searcher=#<Ferret::Search::Searcher:0x58a6ec>>
>>
>> puts res.hits.size # gives me: 0
>> _______________________________________________
>> Ferret-talk mailing list
>> Ferret-talk at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/ferret-talk
>>
>
> --
> Jens Kr?mer
> http://www.jkraemer.net/ - Blog
> http://www.omdb.org/     - The new free film database