Hello, I was wondering why my Unicorn master process''s memory use grows over time. As as I understand it, when I (re)start Unicorn a master process spins up which loads my Rails app. The master process then brings up worker processes which handle traffic to the app. As time passes I''m not surprised to see the workers use more memory: Ruby 1.9''s garbage collector doesn''t free as much memory as it could and it''s not inconceivable that my code is somewhat relaxed about creating objects. However if the workers are handling the traffic, why does the master process''s footprint grow? Is it simply the inefficient garbage collector or is there another reason which, hopefully, I could address? Thanks in advance. Yours, Andy Stewart _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying
Andrew Stewart <boss@airbladesoftware.com> wrote:> I was wondering why my Unicorn master process''s memory use grows over > time. > > As as I understand it, when I (re)start Unicorn a master process spins > up which loads my Rails app. The master process then brings up worker > processes which handle traffic to the app.Correct, the master should not grow. If you''re using "preload_app true", can you reproduce the growth without it?> As time passes I''m not surprised to see the workers use more memory: > Ruby 1.9''s garbage collector doesn''t free as much memory as it could and > it''s not inconceivable that my code is somewhat relaxed about creating > objects. > > However if the workers are handling the traffic, why does the master > process''s footprint grow? Is it simply the inefficient garbage > collector or is there another reason which, hopefully, I could address?If you''re using preload_app, I suspect it''s some background thread or hook causing it. Otherwise, can you reproduce this with a barebones application? _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying
On Wed, May 15, 2013, at 11:28 AM, Eric Wong wrote:> Correct, the master should not grow. If you''re using "preload_app true", > can you reproduce the growth without it?I am using "preload_app true". I''ll experiment without it and see what happens.> If you''re using preload_app, I suspect it''s some background thread > or hook causing it. Otherwise, can you reproduce this with a barebones > application?Do you mean something like Delayed Job via rake via Upstart? Or would using the Rails (3.0) console affect the master process somehow? I''ll try a barebones application and see if I can reproduce this. Yours, Andy Stewart _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying
Andrew Stewart <boss@airbladesoftware.com> wrote:> On Wed, May 15, 2013, at 11:28 AM, Eric Wong wrote: > > If you''re using preload_app, I suspect it''s some background thread > > or hook causing it. Otherwise, can you reproduce this with a barebones > > application? > > Do you mean something like Delayed Job via rake via Upstart? Or would > using the Rails (3.0) console affect the master process somehow?Do any of those connect to the master? Hopefully somebody else here knows more about Delayed Job and Rails more than I do. Also, if you have something constantly sending signals to the master in a loop, unicorn (and maybe Ruby itself) may have trouble from queueing signals up. _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying
> Do you mean something like Delayed Job via rake via Upstart? Or wouldusing the Rails (3.0) console affect the master process somehow? I don''t think so. Delayed job and rails console are independent processes. They won''t affect unicorn master process. There are some gem libraries that work in thread. For example, NewRelic and AirBrake use thread to report errors and metrics without blocking your application. Probably you want to look at libraries like them. Tatsuya On 15 May 2013 10:57, Eric Wong <normalperson@yhbt.net> wrote:> > Andrew Stewart <boss@airbladesoftware.com> wrote: > > On Wed, May 15, 2013, at 11:28 AM, Eric Wong wrote: > > > If you''re using preload_app, I suspect it''s some background thread > > > or hook causing it. Otherwise, can you reproduce this with a barebones > > > application? > > > > Do you mean something like Delayed Job via rake via Upstart? Or would > > using the Rails (3.0) console affect the master process somehow? > > Do any of those connect to the master? Hopefully somebody else here > knows more about Delayed Job and Rails more than I do. > > Also, if you have something constantly sending signals to the master in > a loop, unicorn (and maybe Ruby itself) may have trouble from queueing > signals up. > _______________________________________________ > Unicorn mailing list - mongrel-unicorn@rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-unicorn > Do not quote signatures (like this one) or top post when replying_______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying
On Wed, May 15, 2013, at 11:57 AM, Eric Wong wrote:> Andrew Stewart <boss@airbladesoftware.com> wrote: > > On Wed, May 15, 2013, at 11:28 AM, Eric Wong wrote: > > > If you''re using preload_app, I suspect it''s some background thread > > > or hook causing it. Otherwise, can you reproduce this with a barebones > > > application?I wonder whether New Relic could be doing this. I don''t really know how New Relic works but the language in its configuration file suggests it polls each app process in the background. It would be interesting to hear what other New Relic users observe happening to their master process''s memory footprint.> Also, if you have something constantly sending signals to the master in > a loop, unicorn (and maybe Ruby itself) may have trouble from queueing > signals up.I''m not aware of anything signalling the master in a loop. _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying
On Wed, May 15, 2013 at 6:22 PM, Andrew Stewart <boss@airbladesoftware.com> wrote:> I wonder whether New Relic could be doing this. I don''t really know how > New Relic works but the language in its configuration file suggests it > polls each app process in the background. > > It would be interesting to hear what other New Relic users observe > happening to their master process''s memory footprint.We''re running on Heroku and also using NewRelic. According to my observation, the master process is quite stable, staying roughly the same memory footprint within 24 hours. (Heroku auto-restarts each process each day, so I don''t know what would happen after 24 hours) We''re using preload_app=true as well. As far as I know, NewRelic''s agent is running in each worker processes, but does nothing in master process. (only a wild guess, I didn''t read the code there) _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying
On May 15, 2013, at 3:22 AM, boss at airbladesoftware.com (Andrew Stewart) wrote:> On Wed, May 15, 2013, at 11:57 AM, Eric Wong wrote: >> Andrew Stewart <boss at airbladesoftware.com> wrote: >>> On Wed, May 15, 2013, at 11:28 AM, Eric Wong wrote: >>>> If you''re using preload_app, I suspect it''s some background thread >>>> or hook causing it. Otherwise, can you reproduce this with a barebones >>>> application? > > I wonder whether New Relic could be doing this. I don''t really know how > New Relic works but the language in its configuration file suggests it > polls each app process in the background. > > It would be interesting to hear what other New Relic users observe > happening to their master process''s memory footprint. >This being software, anything is possible, but I strongly suspect that New Relic is not the cause here. The New Relic Ruby agent works by spawning a background thread in each instrumented process that periodically sends the data collected during request processing back to New Relic''s servers. In the case of unicorn, it actually doesn''t even spawn the background thread until we''re in a callback that gets called inside each of the unicorn worker processes after the master has forked them off. That is to say, there is no New Relic-spawned background thread in the unicorn master process. FWIW, my suggestions would be to either slowly whittle down a repro case by removing gems and parts of your application that you suspect might be related until you hit one that makes the problem disappear, or to use a tool like heap_dump (https://github.com/Vasfed/heap_dump) or hack something together manually using ObjectSpace.count_objects to see if there''s a particular kind of Object that''s accumulating in your unicorn master process. Ben _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying