I''ve been messing around with Ferret (no punn intended). After spending some time testing it out (indexing to file), I decided to index about 10% of the data I want to eventually index. It took several hours to complete the index on my local machine, but it was created without any problems and after optimising it the searches returned results at the sort of speed I was expecting. I then continuned to add a further 10% of the overall data to the index and it all ran through fine for a few hours until it bombed out with the error below. I''ve been running this on my local machine and the first run was left overnight, whereas the second run was going on while I was using my machine for other things. Which leads me to suspect that it''s a memory-related issue that made it bail out. so I have 2 questions: 1) does anyone know what caused the error or have any suggestions to prevent it in future? 2) I can''t search on the index without getting errors so is the index recoverable or will I need to recreate the index again? Thanks in advance, Derek EOFError c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:178:in `refill'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:94:in `read_byte'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:109:in `read_bytes'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:108:in `upto'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:108:in `read_bytes'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_reader.rb:286:in `get_norms_into'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_reader.rb:271:in `synchronize'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_reader.rb:271:in `get_norms_into'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:341:in `merge_norms'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:338:in `each'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:338:in `merge_norms'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:334:in `each_with_index'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/field_infos.rb:130:in `each_with_index'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/field_infos.rb:130:in `each'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/field_infos.rb:130:in `each_with_index'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/field_infos.rb:130:in `each_with_index'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:334:in `merge_norms'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:49:in `merge'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:403:in `merge_segments'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:351:in `flush_ram_segments'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:127:in `close'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:126:in `synchronize'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:126:in `close'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index.rb:173:in `close'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index.rb:168:in `synchronize'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index.rb:168:in `close'' -- Posted via http://www.ruby-forum.com/.
I''ve been messing around with Ferret (no punn intended). After spending some time testing it out (indexing to file), I decided to index about 10% of the data I want to eventually index. It took several hours to complete the index on my local machine, but it was created without any problems and after optimising it the searches returned results at the sort of speed I was expecting. I then continuned to add a further 10% of the overall data to the index and it all ran through fine for a few hours until it bombed out with the error below. I''ve been running this on my local machine and the first run was left overnight, whereas the second run was going on while I was using my machine for other things. Which leads me to suspect that it''s a memory-related issue that made it bail out. so I have 2 questions: 1) does anyone know what caused the error or have any suggestions to prevent it in future? 2) I can''t search on the index without getting errors so is the index recoverable or will I need to recreate the index again? Thanks in advance, Derek EOFError c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:178:in `refill'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:94:in `read_byte'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:109:in `read_bytes'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:108:in `upto'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:108:in `read_bytes'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_reader.rb:286:in `get_norms_into'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_reader.rb:271:in `synchronize'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_reader.rb:271:in `get_norms_into'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:341:in `merge_norms'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:338:in `each'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:338:in `merge_norms'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:334:in `each_with_index'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/field_infos.rb:130:in `each_with_index'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/field_infos.rb:130:in `each'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/field_infos.rb:130:in `each_with_index'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/field_infos.rb:130:in `each_with_index'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:334:in `merge_norms'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:49:in `merge'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:403:in `merge_segments'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:351:in `flush_ram_segments'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:127:in `close'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:126:in `synchronize'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:126:in `close'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index.rb:173:in `close'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index.rb:168:in `synchronize'' c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index.rb:168:in `close'' -- Posted via http://www.ruby-forum.com/.
Hi Derek, I''m not sure why this happens. But it seems to only happen with the pure Ruby version so I''d recommend waiting for the Windows port of the C version. You''ll also be done in minutes rather than hours with about 100 times speed up of the indexing. I will get around to trying to fix this bug in the pure Ruby version eventually but don''t hold your breath. Patches welcome! Cheers, Dave On 4/19/06, Derek Mailer <derek.mailer at iomart.com> wrote:> I''ve been messing around with Ferret (no punn intended). After spending > some time testing it out (indexing to file), I decided to index about > 10% of the data I want to eventually index. It took several hours to > complete the index on my local machine, but it was created without any > problems and after optimising it the searches returned results at the > sort of speed I was expecting. I then continuned to add a further 10% of > the overall data to the index and it all ran through fine for a few > hours until it bombed out with the error below. > > I''ve been running this on my local machine and the first run was left > overnight, whereas the second run was going on while I was using my > machine for other things. Which leads me to suspect that it''s a > memory-related issue that made it bail out. > > so I have 2 questions: > 1) does anyone know what caused the error or have any suggestions to > prevent it in future? > 2) I can''t search on the index without getting errors so is the index > recoverable or will I need to recreate the index again? > > Thanks in advance, > Derek > > EOFError > > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:178:in > `refill'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:94:in > `read_byte'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:109:in > `read_bytes'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:108:in > `upto'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/store/buffered_index_io.rb:108:in > `read_bytes'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_reader.rb:286:in > `get_norms_into'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_reader.rb:271:in > `synchronize'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_reader.rb:271:in > `get_norms_into'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:341:in > `merge_norms'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:338:in > `each'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:338:in > `merge_norms'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:334:in > `each_with_index'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/field_infos.rb:130:in > `each_with_index'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/field_infos.rb:130:in > `each'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/field_infos.rb:130:in > `each_with_index'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/field_infos.rb:130:in > `each_with_index'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:334:in > `merge_norms'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/segment_merger.rb:49:in > `merge'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:403:in > `merge_segments'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:351:in > `flush_ram_segments'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:127:in > `close'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:126:in > `synchronize'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index_writer.rb:126:in > `close'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index.rb:173:in > `close'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index.rb:168:in > `synchronize'' > c:/ruby/lib/ruby/gems/1.8/gems/ferret-0.9.0/lib/ferret/index/index.rb:168:in > `close'' > > -- > Posted via http://www.ruby-forum.com/. > _______________________________________________ > Ferret-talk mailing list > Ferret-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/ferret-talk >