I have a lot of crashes when I try to use term vectors. Here''s an
example, which crashes pretty consistently. This problem seems to be
somewhat sensitive to platform... people on other OS''s and ruby
versions
have reported no error. I have seen this with ferret 0.10.13 and 0.10.14
on debian stable using ruby 1.8.2, but I have observed the same problem
on various other systems as well. I''ve reported this issue here before,
but it was when David was gone.
program:
require ''rubygems''
require ''ferret''
#require ''zlib''
fields=Ferret::Index::FieldInfos.new
fields.add_field :text, :store=>:no#, :index=>:omit_norms
i = Ferret::I.new :field_infos=>fields
#:path=>''temp_index''
20.times{
i << {:text=>`man gcc`[0..135000]}
}
#i.close_writer
r=i.reader
#r.term_docs_for(:text, "example")
r.term_vector(0,:text)
example output:
$ ruby tvtest.rb
Reformatting gcc(1), please wait...
Reformatting gcc(1), please wait...
Reformatting gcc(1), please wait...
Reformatting gcc(1), please wait...
Reformatting gcc(1), please wait...
Reformatting gcc(1), please wait...
Reformatting gcc(1), please wait...
Reformatting gcc(1), please wait...
Reformatting gcc(1), please wait...
Reformatting gcc(1), please wait...
Reformatting gcc(1), please wait...
Reformatting gcc(1), please wait...
Reformatting gcc(1), please wait...
Reformatting gcc(1), please wait...
Reformatting gcc(1), please wait...
Reformatting gcc(1), please wait...
Reformatting gcc(1), please wait...
Reformatting gcc(1), please wait...
Reformatting gcc(1), please wait...
Reformatting gcc(1), please wait...
tvtest.rb:16: [BUG] Segmentation fault
ruby 1.8.2 (2005-04-11) [i386-linux]
Aborted
On Fri, Feb 16, 2007 at 02:52:28PM -0800, Caleb Clausen wrote:> I have a lot of crashes when I try to use term vectors. Here''s an > example, which crashes pretty consistently. This problem seems to be > somewhat sensitive to platform... people on other OS''s and ruby versions > have reported no error. I have seen this with ferret 0.10.13 and 0.10.14 > on debian stable using ruby 1.8.2, but I have observed the same problem > on various other systems as well. I''ve reported this issue here before, > but it was when David was gone. > > program: > > require ''rubygems'' > require ''ferret'' > #require ''zlib'' > > > fields=Ferret::Index::FieldInfos.new > fields.add_field :text, :store=>:no#, :index=>:omit_norms > i = Ferret::I.new :field_infos=>fields #:path=>''temp_index'' > > 20.times{ > i << {:text=>`man gcc`[0..135000]} > } > #i.close_writer > r=i.reader > #r.term_docs_for(:text, "example") > > r.term_vector(0,:text) > >[..]> tvtest.rb:16: [BUG] Segmentation fault > ruby 1.8.2 (2005-04-11) [i386-linux] > > Abortedsame here with Ubuntu 6.10 / Ruby 1.8.4. Jens -- Jens Kr?mer webit! Gesellschaft f?r neue Medien mbH Schnorrstra?e 76 | 01069 Dresden Telefon +49 351 46766-0 | Telefax +49 351 46766-66 kraemer at webit.de | www.webit.de Amtsgericht Dresden | HRB 15422 GF Sven Haubold, Hagen Malessa
>> tvtest.rb:16: [BUG] Segmentation fault >> ruby 1.8.2 (2005-04-11) [i386-linux] >> >> Aborted > > same here with Ubuntu 6.10 / Ruby 1.8.4.no problem on MacOSX 10.4, ruby 1.8.5, ferret (0.10.14)
Benjamin Krause wrote:>>> tvtest.rb:16: [BUG] Segmentation fault >>> ruby 1.8.2 (2005-04-11) [i386-linux] >>> >>> Aborted >>> >> same here with Ubuntu 6.10 / Ruby 1.8.4. >>Same on cygwin / Ruby 1.8.5, BUT if I turn off garbage collection (GC.disable) it doesn''t crash. I think this is related to: http://rubyforge.org/pipermail/ferret-talk/2007-February/002504.html and others... which David said he is working on. The following script always seems to die at the same point on my machine and may provide some extra insight. require ''rubygems'' require ''ferret'' fields = Ferret::Index::FieldInfos.new fields.add_field :text, :store => :no #GC.disable s = `man gcc` ix = 0 s.scan(/./m) do |c| puts "#{ix}: #{c}" i = Ferret::I.new :field_infos => fields i << {:text => s[0..ix+=1]} tv = i.reader.term_vector(0, :text) end Dies on character 357 on my machine... Cheers! Patrick -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/ferret-talk/attachments/20070219/805ea2b7/attachment.html
Some more on this issue. I can narrow the crash down to an index with
just one document, and ferret crashes after getting its term vector a
number of times, perhaps as few as 2 or 3. The version below is tuned to
crash quickly on my system, others may find it necessary to give other
numbers in the command line argument in order to make the crash happen
sooner. See below for the code.
Some fiddling with the ferret source code reveals that disabling the
call to tv_destroy() in frt_ir_term_vector() (which is the implmentation
of #term_vector) seems to make the problem go away. Experimenting with
tv_destroy(), I found that disabling just the frees of offsets and
positions is enough to keep the crash away. This suggests that there''s
a
mismanagement of the allocation of the associated variables, but if so I
was unable to spot it in the source....
This is further than I''ve gotten in investigating this problem in a
while, but I''m unsure where to go next.
require ''rubygems''
require ''ferret''
fields = Ferret::Index::FieldInfos.new
fields.add_field :text, :store => :no
scale=(ARGV.first||662).to_i #rand(1000)
s = {:text => "foo bar baz "*scale }
i = Ferret::I.new :field_infos => fields
i << s
9999999999.times do|j|
tv = i.reader.term_vector(0, :text)
print "."; STDOUT.flush
end
On 2/21/07, Caleb Clausen <caleb at inforadical.net> wrote:> Some more on this issue. I can narrow the crash down to an index with > just one document, and ferret crashes after getting its term vector a > number of times, perhaps as few as 2 or 3. The version below is tuned to > crash quickly on my system, others may find it necessary to give other > numbers in the command line argument in order to make the crash happen > sooner. See below for the code. > > Some fiddling with the ferret source code reveals that disabling the > call to tv_destroy() in frt_ir_term_vector() (which is the implmentation > of #term_vector) seems to make the problem go away. Experimenting with > tv_destroy(), I found that disabling just the frees of offsets and > positions is enough to keep the crash away. This suggests that there''s a > mismanagement of the allocation of the associated variables, but if so I > was unable to spot it in the source.... > > This is further than I''ve gotten in investigating this problem in a > while, but I''m unsure where to go next.Hi Caleb, After reading your first email I found the same things behavior as you describe here. This is very frustrating because in this case I create completely independent Ruby objects. They don''t reference the Ferret data space at all so this was the last place I expected to have garbage collection problems. It makes no sense to me at all that not freeing the offsets and positions arrays should make any difference at all. If you have any more ideas with regard to this problem I''d love to hear them as it has me a little stumped. -- Dave Balmain http://www.davebalmain.com/
On 2/23/07, David Balmain <dbalmain.ml at gmail.com> wrote:> On 2/21/07, Caleb Clausen <caleb at inforadical.net> wrote: > > Some more on this issue. I can narrow the crash down to an index with > > just one document, and ferret crashes after getting its term vector a > > number of times, perhaps as few as 2 or 3. The version below is tuned to > > crash quickly on my system, others may find it necessary to give other > > numbers in the command line argument in order to make the crash happen > > sooner. See below for the code. > > > > Some fiddling with the ferret source code reveals that disabling the > > call to tv_destroy() in frt_ir_term_vector() (which is the implmentation > > of #term_vector) seems to make the problem go away. Experimenting with > > tv_destroy(), I found that disabling just the frees of offsets and > > positions is enough to keep the crash away. This suggests that there''s a > > mismanagement of the allocation of the associated variables, but if so I > > was unable to spot it in the source.... > > > > This is further than I''ve gotten in investigating this problem in a > > while, but I''m unsure where to go next. > > Hi Caleb, > > After reading your first email I found the same things behavior as you > describe here. This is very frustrating because in this case I create > completely independent Ruby objects. They don''t reference the Ferret > data space at all so this was the last place I expected to have > garbage collection problems. It makes no sense to me at all that not > freeing the offsets and positions arrays should make any difference at > all. If you have any more ideas with regard to this problem I''d love > to hear them as it has me a little stumped.I''ve made a little more progress on this. By disabling the garbage collector while building the term_vector I can prevent the segfault. I guess I need to spend some time to really understand how the ruby garbage collector works. Adding the top and bottom lines bellow prevents any segfault; int old_dont_gc = rb_gc_disable(); rtv = frt_get_tv(tv); tv_destroy(tv); if (old_dont_gc == Qfalse) rb_gc_enable(); -- Dave Balmain http://www.davebalmain.com/
Dave Balmain wrote:> After reading your first email I found the same things behavior as you > describe here. This is very frustrating because in this case I create > completely independent Ruby objects. They don''t reference the Ferret > data space at all so this was the last place I expected to have > garbage collection problems. It makes no sense to me at all that notI''m so glad to hear it''s not just me!> freeing the offsets and positions arrays should make any difference at > all. If you have any more ideas with regard to this problem I''d love > to hear them as it has me a little stumped.I''ve long thought this problem is a just a pointer gone rampaging, not really a garbage collector issue so much, it''s just that that''s where it shows up..... the fact that disabling memory de-allocation code causes it to go into remission suggested to me that it was in fact a memory management issue. But it sounds like we both think the code that manages those variables is correct. (It''s not code that I''m familiar with, so it''s good that we concur on this point.) A workaround, even one that causes a memory leak, is important progress on this issue, as far as I''m concerned. It sounds like you came up with an even better workaround; I''ll have to try it. I''m afraid this is one of those really nassty c pointer bugs that is difficult to solve because the problem is very far away from the code that actually fails. Perhaps this is a situation that calls for valgrind, or something like that? Some kind of library or tool for debugging memory allocation problems could help prove that the code really really does operate the way the programmers think it should. I''ve played with using valgrind to debug ferret before; I think it was in connection with another bug. I couldn''t get very far in part because ruby pre-allocates pools of objects which it manages itself. That behavior would have to be disabled to permit valgrind to do its magic. Sticking lots of assertions into all the code that gets executed by the failing test script is another thing to try, I guess. That really more of a shot-gun approach... I''m sorry I can''t offer anything but these really generic suggestions.
On 2/25/07, Caleb Clausen <caleb at inforadical.net> wrote:> I''m afraid this is one of those really nassty c pointer bugs that is > difficult to solve because the problem is very far away from the code > that actually fails. Perhaps this is a situation that calls for > valgrind, or something like that? Some kind of library or tool for > debugging memory allocation problems could help prove that the code > really really does operate the way the programmers think it should. I''ve > played with using valgrind to debug ferret before; I think it was in > connection with another bug. I couldn''t get very far in part because > ruby pre-allocates pools of objects which it manages itself. That > behavior would have to be disabled to permit valgrind to do its magic.I use valgrind all the time. It is a godsend. Unfortunately it doesn''t work too well with Ruby as ruby''s garbage collector raises so many errors. I plan to make Ferret 2.0 scriptable so that more people can get involved with Ferret development and I would have liked to have used Ruby except for this problem with valgrind. Lua looks like a better alternative and much lighter weight. Anyway, it is a long way in the future so probably not worth mentioning.> Sticking lots of assertions into all the code that gets executed by the > failing test script is another thing to try, I guess. That really more > of a shot-gun approach... I''m sorry I can''t offer anything but these > really generic suggestions.This is the method I end up having to use when debugging Ferret''s Ruby bindings. Anyway, I probably should have posted it here but I did end up solving the problem; http://www.ruby-forum.com/topic/98709#210591 So term-vectors will be fine in the next release without introducing any memory leaks. Cheers, Dave -- Dave Balmain http://www.davebalmain.com/