Hi,
I''m working at Cookpad Inc.
- http://cookpad.com/
Cookpad is very famous cooking recipe service in Japan,
and largest rails application in Japan.
I applied GC.disable after fork, and a few changed Unicorn::OobGC.
# unicorn.conf.rb
after_fork do |server, worker|
GC.disable if RAILS_ENV == ''production''
# OobGC code
def process_client(client)
super(client) # Unicorn::HttpServer#process_client
if OOBGC_PATH =~ OOBGC_ENV[PATH_INFO] && ((@@nr -= 1) <= 0)
@@nr = OOBGC_INTERVAL
OOBGC_ENV.clear
disabled = GC.enable
GC.start
GC.disable if disabled
end
end
after applied,
application response time 130% fast!
and CPU usage to half!
- http://bit.ly/rhnVlg (response time)
- http://bit.ly/pgMN06 (CPU usage)
This is OobGC patch, apply please.
- http://bit.ly/roVLtZ (patch)
Also, my rails application have memory leaks ;(
I wrote process killing code because GC.disable create more memory leak.
- https://gist.github.com/1258681
------------------------------------------
Yuichi Tateno
https://github.com/hotchpotch
That looks promising. I''m curious what our app''s memory would climb to (at worst). Clifton
secondlife <hotchpotch at gmail.com> wrote:> after applied, > application response time 130% fast! > and CPU usage to half! > > - http://bit.ly/rhnVlg (response time) > - http://bit.ly/pgMN06 (CPU usage)Thanks for the feedback! I wonder if 1.9.3 with lazy sweep GC can provide similar improvements (without needing OobGC or GC.disable, even).> This is OobGC patch, apply please. > - http://bit.ly/roVLtZ (patch)Thanks! Applied as c7ba76a21c5d00fb5c173cd6aa847442bbc652cb with the following commit message: From: Yuichi Tateno <hotchpotch at gmail.com> Date: Mon, 3 Oct 2011 16:51:19 +0900 Subject: [PATCH] OobGC: force GC.start [ew: we need to explicitly enable GC if it is disabled and respect applications that disable GC] Acked-by: Eric Wong <normalperson at yhbt.net>> I wrote process killing code because GC.disable create more memory leak. > - https://gist.github.com/1258681I think that could be done entirely as Unicorn-specific Rack middleware and not hook into Unicorn internal methods like process_client. Also, the pid checks are unnecessary since that code never runs in the master.
> Thanks! Applied as c7ba76a21c5d00fb5c173cd6aa847442bbc652cb> with the following commit message:Thank you for applied!> I think that could be done entirely as Unicorn-specific Rack middleware > and not hook into Unicorn internal methods like process_client. ?Also, > the pid checks are unnecessary since that code never runs in the master.Thank you for advice. I rewrite my code. - https://gist.github.com/1258681 ------------------------------------------ Yuichi Tateno https://github.com/hotchpotch
This solution is awesome. I applied the patch on our staging server and I''ve confirmed a significant performance improve. Some pages(especially complicated ones) responds like 4 times faster than before. However it seems that we hit a same issue Yuichi had.?When we had an access on particular page, I can see memory size of worker increases 5-10MB...> Also, my rails application have memory leaks ;( > I wrote process killing code because GC.disable create more memory leak. > - https://gist.github.com/1258681Beside I want to avoid to use this solution if possible... I don''t actually understand is why GC.disable solution could introduce more memory leak. If I simplify the problem, the code is something like bellow: --------------- GC.disable (do something) GC.enable GC.start --------------- When the code block finishes, I expect that memory size should be (almost) equal with the case GC is enabled at begging. But it doesn''t seems so from our experience. Do anyone know why there could be significant difference on memory usage because of timing of GC? It might be a question on Ruby rather than Unicorn, though, I thought even just sharing my experience could be worth to someone here. Tatsuya Ono
Tatsuya Ono <ononoma at gmail.com> wrote:> I don''t actually understand is why GC.disable solution could introduce > more memory leak. If I simplify the problem, the code is something > like bellow: > > --------------- > GC.disable > (do something) > GC.enable > GC.start > --------------- > > When the code block finishes, I expect that memory size should be > (almost) equal with the case GC is enabled at begging. But it doesn''t > seems so from our experience. > > Do anyone know why there could be significant difference on memory > usage because of timing of GC? It might be a question on Ruby rather > than Unicorn, though, I thought even just sharing my experience could > be worth to someone here.Basically, the free(3) function in the C standard library does not guarantee memory is released back to the kernel (speed vs memory usage tradeoff). There was discussion of this on the usp.ruby mailing list starting at Message-ID: 20110914234917.GA2480 at dcvr.yhbt.net usp.ruby archives are at http://bogomips.org/usp.ruby/archives/2011.mbox.gz
Thanks Eric for the feedback. I actually had read that email and I think I understand it. But what I am experiencing seems a different story. Our rails app uses around 250MB memory usually. After applying this patch and calling GC.disabled on after_fork, the usage of memory increases on every request and goes up to 1GB easily. However, yes, I must say that I need to test more carefully. Let me come back later. I am going to have some stress test and monitor if Unicorn introduces swapping on VM with this solution. Hopefully I can do it tomorrow or later this week. Tatsuya On 10 October 2011 22:53, Eric Wong <normalperson at yhbt.net> wrote:> Tatsuya Ono <ononoma at gmail.com> wrote: >> I don''t actually understand is why GC.disable solution could introduce >> more memory leak. If I simplify the problem, the code is something >> like bellow: >> >> --------------- >> GC.disable >> (do something) >> GC.enable >> GC.start >> --------------- >> >> When the code block finishes, I expect that memory size should be >> (almost) equal with the case GC is enabled at begging. But it doesn''t >> seems so from our experience. >> >> Do anyone know why there could be significant difference on memory >> usage because of timing of GC? It might be a question on Ruby rather >> than Unicorn, though, I thought even just sharing my experience could >> be worth to someone here. > > Basically, the free(3) function in the C standard library does not > guarantee memory is released back to the kernel (speed vs memory usage > tradeoff). > > There was discussion of this on the usp.ruby mailing list starting at > Message-ID: 20110914234917.GA2480 at dcvr.yhbt.net > > usp.ruby archives are at http://bogomips.org/usp.ruby/archives/2011.mbox.gz > _______________________________________________ > Unicorn mailing list - mongrel-unicorn at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-unicorn > Do not quote signatures (like this one) or top post when replying >
Yes, you are right, Eric. The usage of memory stops increasing at a certain point. Besides I do not see any significant page I/O with it. I give the patch go for our live service without the UnicornKiller. I will report if we experience any issues occurred in the wild. Thanks Yuichi again for submitting the patch and sharing your knowledge. By the way, I tested this with Rails 2.3/Ruby 1.8.7/FreeBSD 8.2 Tatsuya On 11 October 2011 00:03, Tatsuya Ono <ononoma at gmail.com> wrote:> Thanks Eric for the feedback. > > I actually had read that email and I think I understand it. But what I > am experiencing seems a different story. Our rails app uses around > 250MB memory usually. After applying this patch and calling > GC.disabled on after_fork, the usage of memory increases on every > request and goes up to 1GB easily. > > However, yes, I must say that I need to test more carefully. Let me > come back later. I am going to have some stress test and monitor if > Unicorn introduces swapping on VM with this solution. Hopefully I can > do it tomorrow or later this week. > > > Tatsuya > > On 10 October 2011 22:53, Eric Wong <normalperson at yhbt.net> wrote: >> Tatsuya Ono <ononoma at gmail.com> wrote: >>> I don''t actually understand is why GC.disable solution could introduce >>> more memory leak. If I simplify the problem, the code is something >>> like bellow: >>> >>> --------------- >>> GC.disable >>> (do something) >>> GC.enable >>> GC.start >>> --------------- >>> >>> When the code block finishes, I expect that memory size should be >>> (almost) equal with the case GC is enabled at begging. But it doesn''t >>> seems so from our experience. >>> >>> Do anyone know why there could be significant difference on memory >>> usage because of timing of GC? It might be a question on Ruby rather >>> than Unicorn, though, I thought even just sharing my experience could >>> be worth to someone here. >> >> Basically, the free(3) function in the C standard library does not >> guarantee memory is released back to the kernel (speed vs memory usage >> tradeoff). >> >> There was discussion of this on the usp.ruby mailing list starting at >> Message-ID: 20110914234917.GA2480 at dcvr.yhbt.net >> >> usp.ruby archives are at http://bogomips.org/usp.ruby/archives/2011.mbox.gz >> _______________________________________________ >> Unicorn mailing list - mongrel-unicorn at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-unicorn >> Do not quote signatures (like this one) or top post when replying >> >