[Ferret-talk] Parallel indexing doesn''t work?

Jens Kraemer
Fri, 11 Jan 2008 18:53:41 -0000

Hi!

On Wed, Jan 09, 2008 at 04:58:20PM -0500, Noah M. Daniels
wrote:
> Ok, further update -- there was an obvious and stupid bug in my code
> that was overwriting the partial indices. So now when that''s
fixed, I
> get the proper number of results for a search:
>
> Company.find_by_contents(''*'')
> => #<ActsAsFerret::SearchResults:0x2b0902ab4628 @current_page=nil,
> @total_hits=247, @results=[], @total_pages=1, @per_page=247>

strange. Did you try to access the merged index with plain Ferret to see
if this works? Additionally, are your partial indexes index ok and
deliver results with contents when you search only one of them?


Cheers,
Jens

>
> however, why is @results empty?
>
> Similarly, find_id_by_contents also returns empty documents, it seems:
>
>  >> Company.find_id_by_contents(''*'')
> => [247, [{:model=>"Company", :data=>{}, :score=>1.0,
:id=>nil},
> {:model=>"Company", :data=>{}, :score=>1.0,
:id=>nil},
> {:model=>"Company", :data=>{}, :score=>1.0,
:id=>nil},
> {:model=>"Company", :data=>{}, :score=>1.0,
:id=>nil},
> {:model=>"Company", :data=>{}, :score=>1.0,
:id=>nil},
> {:model=>"Company", :data=>{}, :score=>1.0,
:id=>nil},
> {:model=>"Company", :data=>{}, :score=>1.0,
:id=>nil},
> {:model=>"Company", :data=>{}, :score=>1.0,
:id=>nil},
> {:model=>"Company", :data=>{}, :score=>1.0,
:id=>nil},
> {:model=>"Company", :data=>{}, :score=>1.0,
:id=>nil}]]
>
> when I would have expected:
>
>  >> Company.find_id_by_contents(''*'')
> => [247, [{:model=>"Company", :score=>1.0,
:id=>"189", :data=>{}},
> {:model=>"Company", :score=>1.0, :id=>"2",
:data=>{}},
> {:model=>"Company", :score=>1.0, :id=>"192",
:data=>{}},
> {:model=>"Company", :score=>1.0, :id=>"4",
:data=>{}},
> {:model=>"Company", :score=>1.0, :id=>"6",
:data=>{}},
> {:model=>"Company", :score=>1.0, :id=>"7",
:data=>{}},
> {:model=>"Company", :score=>1.0, :id=>"8",
:data=>{}},
> {:model=>"Company", :score=>1.0, :id=>"37",
:data=>{}},
> {:model=>"Company", :score=>1.0, :id=>"13",
:data=>{}},
> {:model=>"Company", :score=>1.0, :id=>"21",
:data=>{}}]]
>
> thanks for the help, and sorry for the silly previous bugs :)
>
> On Jan 9, 2008, at 4:37 PM, Noah M. Daniels wrote:
>
> > Thanks, Jens. Good catch; this little example works correctly after
> > fixing that change.
> >
> > However, my ActsAsFerret index merging does _not_ work, and
I''m
> > wondering if it''s something to do with AAF''s
handling of documents in
> > an index?
> >
> > Let''s call my indexed class Company...
> >
> >
> > Company.find_by_contents(''*'')
> > => #<ActsAsFerret::SearchResults:0x2b1699108878
@current_page=nil,
> > @total_hits=3, @results=[], @total_pages=1, @per_page=3>
> >
> >
> > yet on each partial index prior to merging, that query would return a
> > bunch of results as one would expect.
> >
> > now, here''s how I''ve built that index... any idea
why the merged index
> > is broken?
> >
> > module FerretHelpers
> > def merge_ferret_index_partitions(model)
> >
> >     model_dir = File.basename(model.aaf_configuration[:ferret][:path])
> >
> >     final_index_path = "/tmp/merged_parallel_ferret_index/
> > #{model_dir}"
> >
> >     partial_index_path = "/tmp/partial_indices/#{model_dir}"
> >
> >     paths = Dir.glob("#{partial_index_path}/*")
> >
> >     paths.each do |path|
> >       i = Ferret::I.new(:path => path, :create => true)
> >       name = path.split(''/'').last
> >       i << {:name => name}
> >       i.close
> >     end
> >
> >     readers = []
> >     paths.each {|path| readers << IndexReader.new(path) }
> >     index_writer = IndexWriter.new(:path => final_index_path)
> >     index_writer.add_readers(readers)
> >     index_writer.close()
> >     readers.each {|reader| reader.close()}
> >     index = Ferret::Index::Index.new(:path => final_index_path)
> >     index.optimize
> >     index.close
> >
> >
> >   end
> > end
> >
> >
> > On Jan 9, 2008, at 4:24 PM, Jens Kraemer wrote:
> >
> >> Hi!
> >>
> >> seems to me you''re indexing strings starting with
''index'' but you''re
> >> searching for ''name''? Or maybe correcting this
already was one of
> >> your
> >> minimal changes?
> >>
> >> If not, try changing that line:
> >>> res = i.search(''name*'')
> >> to
> >>> res = i.search(''index*'')
> >>
> >> cheers,
> >> Jens
> >>
> >> On Wed, Jan 09, 2008 at 04:02:17PM -0500, Noah M. Daniels wrote:
> >>> Hi,
> >>>
> >>> I''m trying to get parallelized ferret indexing
working for my AAF
> >>> indices, based on the example in the O''Reilly Ferret
shortcut.
> >>> However, the resulting indices after merging seem to have no
actual
> >>> documents.
> >>>
> >>> I went and made minimal changes to the example in the Ferret
> >>> shortcut
> >>> pdf, and indeed can''t get that to work either.
I''d appreciate any
> >>> help
> >>> anyone can give! Thanks!
> >>>
> >>> The example is below:
> >>>
> >>> #!/usr/bin/env ruby
> >>>
> >>> require ''rubygems''
> >>> require ''ferret''
> >>> include Ferret::Index
> >>>
> >>> 5.times do |i|
> >>>  name = "index#{i}"
> >>>  puts name
> >>>  i = Ferret::I.new(:path => "/tmp/#{i}", :create
=> true)
> >>>  i << {:name => name}
> >>>  i.close
> >>> end
> >>> readers = []
> >>> readers << IndexReader.new("/tmp/0")
> >>> readers << IndexReader.new("/tmp/1")
> >>> readers << IndexReader.new("/tmp/2")
> >>> readers << IndexReader.new("/tmp/3")
> >>> readers << IndexReader.new("/tmp/4")
> >>> index_writer = IndexWriter.new(:path =>
"/tmp/test")
> >>> index_writer.add_readers(readers)
> >>> index_writer.close()
> >>> readers.each {|reader| reader.close()}
> >>> i = Ferret::I.new(:path => ''/tmp/test'')
> >>> res = i.search(''name*'')
> >>> puts res.inspect # gives me: #<struct
Ferret::Search::TopDocs
> >>> total_hits=0, hits=[], max_score=0.0,
> >>> searcher=#<Ferret::Search::Searcher:0x58a6ec>>
> >>>
> >>> puts res.hits.size # gives me: 0
> >>> _______________________________________________
> >>> Ferret-talk mailing list
> >>> Ferret-talk at rubyforge.org
> >>> http://rubyforge.org/mailman/listinfo/ferret-talk
> >>>
> >>
> >> --
> >> Jens Kr?mer
> >> http://www.jkraemer.net/ - Blog
> >> http://www.omdb.org/     - The new free film database
> >> _______________________________________________
> >> Ferret-talk mailing list
> >> Ferret-talk at rubyforge.org
> >> http://rubyforge.org/mailman/listinfo/ferret-talk
> >
> > _______________________________________________
> > Ferret-talk mailing list
> > Ferret-talk at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/ferret-talk
>
> _______________________________________________
> Ferret-talk mailing list
> Ferret-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/ferret-talk
>

--
Jens Kr?mer
http://www.jkraemer.net/ - Blog
http://www.omdb.org/     - The new free film database