A plea for help, here... The rails core team is hacking like mad this weekend at RubyConf, and my assignment has been to track down and fix the memory leak in development mode. Here''s what is known about this leak: 1. It only occurs in development mode (production mode does not exhibit the leak) 2. It only occurs under FastCGI (running under WEBrick does not exhibit the leak) 3. The leak is due to an accumulation of Proc objects used with define_method, primarily in ActiveRecord (the define_method''s in question are used to create the dynamic "has_one", "belongs_to", "has_many", and "habtm" accessors). The leak may also be due to define_method''s used in ActionMailer, as well. The problem seems strangely nondeterministic. Although it is easily reproducible, fixing it has been like a nightmare game of whack-a- mole. I''ll add a few lines of code somewhere to test a solution, and the problem will go away, but moving those lines elsewhere in the framework (even a line or two up or down) brings the problem back. And what "fixes" one application''s leak may have no effect at all on another application. There _must_ be a sane solution, however, because I have verified multiple times (in multiple applications) that there are no leaks when running the same application under WEBrick. The leak seems to be on the order of 10 procs per request, per declared ActiveRecord association. (Basecamp, as an example, leaks about 200 procs per request in development mode. A minimal app I wrote to reproduce the problem, with two AR subclasses with one association each, leaks 20 procs per request.) I won''t spam the lists with additional details, but if anyone is feeling particularly noble (and brave), please contact me off list and I''ll give you as much information about this as you can handle. (You''ll need to have edge rails and fastcgi working, either with lighttpd or apache. I can provide you with a sample app to play with, which requires sqlite.) - Jamis
Does it happen under cgi, or scgi? On 10/15/05, Jamis Buck <jamis-uHoyYlH2B+GakBO8gow8eQ@public.gmane.org> wrote:> A plea for help, here... The rails core team is hacking like mad this > weekend at RubyConf, and my assignment has been to track down and fix > the memory leak in development mode. Here''s what is known about this > leak: > > 1. It only occurs in development mode (production mode does not > exhibit the leak) > 2. It only occurs under FastCGI (running under WEBrick does not > exhibit the leak) > 3. The leak is due to an accumulation of Proc objects used with > define_method, primarily in ActiveRecord (the define_method''s in > question are used to create the dynamic "has_one", "belongs_to", > "has_many", and "habtm" accessors). The leak may also be due to > define_method''s used in ActionMailer, as well. > > The problem seems strangely nondeterministic. Although it is easily > reproducible, fixing it has been like a nightmare game of whack-a- > mole. I''ll add a few lines of code somewhere to test a solution, and > the problem will go away, but moving those lines elsewhere in the > framework (even a line or two up or down) brings the problem back. > And what "fixes" one application''s leak may have no effect at all on > another application. There _must_ be a sane solution, however, > because I have verified multiple times (in multiple applications) > that there are no leaks when running the same application under WEBrick. > > The leak seems to be on the order of 10 procs per request, per > declared ActiveRecord association. (Basecamp, as an example, leaks > about 200 procs per request in development mode. A minimal app I > wrote to reproduce the problem, with two AR subclasses with one > association each, leaks 20 procs per request.) > > I won''t spam the lists with additional details, but if anyone is > feeling particularly noble (and brave), please contact me off list > and I''ll give you as much information about this as you can handle. > (You''ll need to have edge rails and fastcgi working, either with > lighttpd or apache. I can provide you with a sample app to play with, > which requires sqlite.) > > - Jamis > _______________________________________________ > Rails mailing list > Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > http://lists.rubyonrails.org/mailman/listinfo/rails >
On Oct 15, 2005, at 1:06 PM, Kyle Maxwell wrote:> Does it happen under cgi, or scgi?Not under CGI, because that reloads the entire environment for each request. I haven''t tested scgi (anyone want to try that for me?) - Jamis> > On 10/15/05, Jamis Buck <jamis-uHoyYlH2B+GakBO8gow8eQ@public.gmane.org> wrote: > >> A plea for help, here... The rails core team is hacking like mad this >> weekend at RubyConf, and my assignment has been to track down and fix >> the memory leak in development mode. Here''s what is known about this >> leak: >> >> 1. It only occurs in development mode (production mode does not >> exhibit the leak) >> 2. It only occurs under FastCGI (running under WEBrick does not >> exhibit the leak) >> 3. The leak is due to an accumulation of Proc objects used with >> define_method, primarily in ActiveRecord (the define_method''s in >> question are used to create the dynamic "has_one", "belongs_to", >> "has_many", and "habtm" accessors). The leak may also be due to >> define_method''s used in ActionMailer, as well. >> >> The problem seems strangely nondeterministic. Although it is easily >> reproducible, fixing it has been like a nightmare game of whack-a- >> mole. I''ll add a few lines of code somewhere to test a solution, and >> the problem will go away, but moving those lines elsewhere in the >> framework (even a line or two up or down) brings the problem back. >> And what "fixes" one application''s leak may have no effect at all on >> another application. There _must_ be a sane solution, however, >> because I have verified multiple times (in multiple applications) >> that there are no leaks when running the same application under >> WEBrick. >> >> The leak seems to be on the order of 10 procs per request, per >> declared ActiveRecord association. (Basecamp, as an example, leaks >> about 200 procs per request in development mode. A minimal app I >> wrote to reproduce the problem, with two AR subclasses with one >> association each, leaks 20 procs per request.) >> >> I won''t spam the lists with additional details, but if anyone is >> feeling particularly noble (and brave), please contact me off list >> and I''ll give you as much information about this as you can handle. >> (You''ll need to have edge rails and fastcgi working, either with >> lighttpd or apache. I can provide you with a sample app to play with, >> which requires sqlite.) >> >> - Jamis >> _______________________________________________ >> Rails mailing list >> Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org >> http://lists.rubyonrails.org/mailman/listinfo/rails >> >> > _______________________________________________ > Rails mailing list > Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > http://lists.rubyonrails.org/mailman/listinfo/rails >
Hello Jamis, 2005/10/15, Jamis Buck <jamis@37signals.com>:> Not under CGI, because that reloads the entire environment for each > request. I haven't tested scgi (anyone want to try that for me?)I can say I have the problem under BOTH WEBrick & SCGI. I'm on Win32, so that might be part of the equation too. Jamis, if you want the code, send me a private E-Mail, and I can give you access to the repository. Bye, François _______________________________________________ Rails mailing list Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org http://lists.rubyonrails.org/mailman/listinfo/rails