Hi all, I have an ROR application thats been running on the production server for almost a couple of months now. Everything seemed fine but the past few days have caused worry. The whole server slows down drastically and comes to a halt suddenly and it requires a reboot for the apps i have on the server to start being responsive again. Memory leak? I installed binarylogic''s memory logic http://github.com/binarylogic/memorylogic/tree/master which shows me memory usage on every line of my log along with the PID. I can see that the memory usage keeps going higher with every request on my local dev machine. Does that mean ive got a memory leak? How do i diagnose this? I have no experience with this and im not even sure what a bad number is when i see one in terms of memory usage. For information''s sake, when i start the app up locally, mem starts at around 46700 and keeps going up by about 400 with every request on average. How can i interpret this data? Anyone used memory logic before?
Frederick Cheung
2009-Jun-12 08:21 UTC
Re: How can I identify a memory leak? using memorylogic..
On Jun 12, 7:35 am, Ram <yourstruly.vi...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> Hi all, > > I have an ROR application thats been running on the production server > for almost a couple of months now. Everything seemed fine but the past > few days have caused worry. The whole server slows down drastically > and comes to a halt suddenly and it requires a reboot for the apps i > have on the server to start being responsive again. > > Memory leak? > > I installed binarylogic''s memory logichttp://github.com/binarylogic/memorylogic/tree/master > which shows me memory usage on every line of my log along with the > PID. I can see that the memory usage keeps going higher with every > request on my local dev machine. > Does that mean ive got a memory leak? How do i diagnose this? I have > no experience with this and im not even sure what a bad number is when > i see one in terms of memory usage. > > For information''s sake, when i start the app up locally, mem starts at > around 46700 and keeps going up by about 400 with every request on > average. How can i interpret this data? Anyone used memory logic > before?That''s probably a bad thing. You need to start digging deeper: every request or only some requests ? Run some tests that hit a certain action 1000 times running: can you narrow it down to a single action (or set of actions) ? Can you remove the leak by disabling bits of functionality or commenting out code ? Keep narrowing it down until you have your culprit. You may find a tool like bleak house (http:// blog.evanweaver.com/articles/2007/05/06/leak-proof-direct-heap- instrumentation-for-bleak_house/) is usefull too: these things can tell you what objects are being leaked which may help you find the problem too. Lastly, watch out for things which naturally survive between requests: class variables etc. Fred
Hi Fred, I have a few basic questions that if you could answer, would help me get a better grasp of things. 1. This number, the memory usage, has to go down after fractionally increasing when a request is made right? So if my app is consuming 40 MB of memory on startup, after going up a little for every request, ideally, it should come back down to 40MB.. right? For the record, thats not happening.. mem usage just keeps going higher and higher.. 2. If you have ever used New Relic RPM, you would know the Physical Mem usage graph in there.. its mem usage against time.. Im guessing it shold be close to constant? Mine is all over the place.. it spikes, it breaks, it falls and it levels.. i really dont understand why it breaks altogether! or any of the others.. your thoughts? 3. The way to go about diagnosing this is overloading every action one by one? I used Apache Benchmark once to do this to our homepage. Is that good enough? What should i be looking for when im looking for the leak? 4. As for class variables, the only ones I use are @@per_page for the will_paginate gem used for pagination. Besides this, there are a few in the plugins. my sessions hash carries way too little to be causing this. Besides, i clear it regularly. Any thoughts? 5. Should i be worried about the number of instances of my app that ive configured mod_rails to serve? I dont quite remember how many that was and dunno where to change it. Just vaguely seemed like something that could be related to the problem at hand. Hope you can guide me here. I have little to no idea about these things and could really use an experienced one guiding me.. thanks! On Jun 12, 1:21 pm, Frederick Cheung <frederick.che...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> On Jun 12, 7:35 am, Ram <yourstruly.vi...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > > > Hi all, > > > I have an ROR application thats been running on the production server > > for almost a couple of months now. Everything seemed fine but the past > > few days have caused worry. The whole server slows down drastically > > and comes to a halt suddenly and it requires a reboot for the apps i > > have on the server to start being responsive again. > > > Memory leak? > > > I installed binarylogic''s memory logichttp://github.com/binarylogic/memorylogic/tree/master > > which shows me memory usage on every line of my log along with the > > PID. I can see that the memory usage keeps going higher with every > > request on my local dev machine. > > Does that mean ive got a memory leak? How do i diagnose this? I have > > no experience with this and im not even sure what a bad number is when > > i see one in terms of memory usage. > > > For information''s sake, when i start the app up locally, mem starts at > > around 46700 and keeps going up by about 400 with every request on > > average. How can i interpret this data? Anyone used memory logic > > before? > > That''s probably a bad thing. You need to start digging deeper: every > request or only some requests ? Run some tests that hit a certain > action 1000 times running: can you narrow it down to a single action > (or set of actions) ? Can you remove the leak by disabling bits of > functionality or commenting out code ? Keep narrowing it down until > you have your culprit. You may find a tool like bleak house (http:// > blog.evanweaver.com/articles/2007/05/06/leak-proof-direct-heap- > instrumentation-for-bleak_house/) is usefull too: these things can > tell you what objects are being leaked which may help you find the > problem too. > > Lastly, watch out for things which naturally survive between requests: > class variables etc. > > Fred
As for bleakhouse, i went through it earlier and it seemed like quite a bit of setup so i put it off for later. Will do it now.. On Jun 12, 1:42 pm, Ram <yourstruly.vi...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> Hi Fred, > > I have a few basic questions that if you could answer, would help me > get a better grasp of things. > > 1. This number, the memory usage, has to go down after fractionally > increasing when a request is made right? So if my app is consuming 40 > MB of memory on startup, after going up a little for every request, > ideally, it should come back down to 40MB.. right? For the record, > thats not happening.. mem usage just keeps going higher and higher.. > > 2. If you have ever used New Relic RPM, you would know the Physical > Mem usage graph in there.. its mem usage against time.. Im guessing it > shold be close to constant? Mine is all over the place.. it spikes, it > breaks, it falls and it levels.. i really dont understand why it > breaks altogether! or any of the others.. your thoughts? > > 3. The way to go about diagnosing this is overloading every action one > by one? I used Apache Benchmark once to do this to our homepage. Is > that good enough? What should i be looking for when im looking for the > leak? > > 4. As for class variables, the only ones I use are @@per_page for the > will_paginate gem used for pagination. Besides this, there are a few > in the plugins. my sessions hash carries way too little to be causing > this. Besides, i clear it regularly. Any thoughts? > > 5. Should i be worried about the number of instances of my app that > ive configured mod_rails to serve? I dont quite remember how many that > was and dunno where to change it. Just vaguely seemed like something > that could be related to the problem at hand. > > Hope you can guide me here. I have little to no idea about these > things and could really use an experienced one guiding me.. thanks! > > On Jun 12, 1:21 pm, Frederick Cheung <frederick.che...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> > wrote: > > > On Jun 12, 7:35 am, Ram <yourstruly.vi...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > > Hi all, > > > > I have an ROR application thats been running on the production server > > > for almost a couple of months now. Everything seemed fine but the past > > > few days have caused worry. The whole server slows down drastically > > > and comes to a halt suddenly and it requires a reboot for the apps i > > > have on the server to start being responsive again. > > > > Memory leak? > > > > I installed binarylogic''s memory logichttp://github.com/binarylogic/memorylogic/tree/master > > > which shows me memory usage on every line of my log along with the > > > PID. I can see that the memory usage keeps going higher with every > > > request on my local dev machine. > > > Does that mean ive got a memory leak? How do i diagnose this? I have > > > no experience with this and im not even sure what a bad number is when > > > i see one in terms of memory usage. > > > > For information''s sake, when i start the app up locally, mem starts at > > > around 46700 and keeps going up by about 400 with every request on > > > average. How can i interpret this data? Anyone used memory logic > > > before? > > > That''s probably a bad thing. You need to start digging deeper: every > > request or only some requests ? Run some tests that hit a certain > > action 1000 times running: can you narrow it down to a single action > > (or set of actions) ? Can you remove the leak by disabling bits of > > functionality or commenting out code ? Keep narrowing it down until > > you have your culprit. You may find a tool like bleak house (http:// > > blog.evanweaver.com/articles/2007/05/06/leak-proof-direct-heap- > > instrumentation-for-bleak_house/) is usefull too: these things can > > tell you what objects are being leaked which may help you find the > > problem too. > > > Lastly, watch out for things which naturally survive between requests: > > class variables etc. > > > Fred
Frederick Cheung
2009-Jun-12 09:08 UTC
Re: How can I identify a memory leak? using memorylogic..
On Jun 12, 9:42 am, Ram <yourstruly.vi...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> Hi Fred, > > I have a few basic questions that if you could answer, would help me > get a better grasp of things. > > 1. This number, the memory usage, has to go down after fractionally > increasing when a request is made right? So if my app is consuming 40 > MB of memory on startup, after going up a little for every request, > ideally, it should come back down to 40MB.. right? For the record, > thats not happening.. mem usage just keeps going higher and higher..Ideally yes - this may not happen every request, but that should be the trend> > 2. If you have ever used New Relic RPM, you would know the Physical > Mem usage graph in there.. its mem usage against time.. Im guessing it > shold be close to constant? Mine is all over the place.. it spikes, it > breaks, it falls and it levels.. i really dont understand why it > breaks altogether! or any of the others.. your thoughts? >pretty constant on my apps (barring instances restarting and stuff like that)> 3. The way to go about diagnosing this is overloading every action one > by one? I used Apache Benchmark once to do this to our homepage. Is > that good enough? What should i be looking for when im looking for the > leak?Probably good enough. I''d look at the data from memory logic to see if you have actions you suspect more than others. Hammer them and if memory usage rockets then you''ve got a step closer to identifying your leak, if not you haven;t> > 4. As for class variables, the only ones I use are @@per_page for the > will_paginate gem used for pagination. Besides this, there are a few > in the plugins. my sessions hash carries way too little to be causing > this. Besides, i clear it regularly. Any thoughts? > > 5. Should i be worried about the number of instances of my app that > ive configured mod_rails to serve? I dont quite remember how many that > was and dunno where to change it. Just vaguely seemed like something > that could be related to the problem at hand. >wouldn''t think so. There have been the odd memory leak in rails (the ones I remember were in rails 2.2.2), so not a bad idea to scan through the changelogs etc. to see if there have been any fixed memory leaks and whether any of them sounds like they might affect you. Fred> Hope you can guide me here. I have little to no idea about these > things and could really use an experienced one guiding me.. thanks! > > On Jun 12, 1:21 pm, Frederick Cheung <frederick.che...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> > wrote: > > > > > On Jun 12, 7:35 am, Ram <yourstruly.vi...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > > Hi all, > > > > I have an ROR application thats been running on the production server > > > for almost a couple of months now. Everything seemed fine but the past > > > few days have caused worry. The whole server slows down drastically > > > and comes to a halt suddenly and it requires a reboot for the apps i > > > have on the server to start being responsive again. > > > > Memory leak? > > > > I installed binarylogic''s memory logichttp://github.com/binarylogic/memorylogic/tree/master > > > which shows me memory usage on every line of my log along with the > > > PID. I can see that the memory usage keeps going higher with every > > > request on my local dev machine. > > > Does that mean ive got a memory leak? How do i diagnose this? I have > > > no experience with this and im not even sure what a bad number is when > > > i see one in terms of memory usage. > > > > For information''s sake, when i start the app up locally, mem starts at > > > around 46700 and keeps going up by about 400 with every request on > > > average. How can i interpret this data? Anyone used memory logic > > > before? > > > That''s probably a bad thing. You need to start digging deeper: every > > request or only some requests ? Run some tests that hit a certain > > action 1000 times running: can you narrow it down to a single action > > (or set of actions) ? Can you remove the leak by disabling bits of > > functionality or commenting out code ? Keep narrowing it down until > > you have your culprit. You may find a tool like bleak house (http:// > > blog.evanweaver.com/articles/2007/05/06/leak-proof-direct-heap- > > instrumentation-for-bleak_house/) is usefull too: these things can > > tell you what objects are being leaked which may help you find the > > problem too. > > > Lastly, watch out for things which naturally survive between requests: > > class variables etc. > > > Fred
Fernando Perez
2009-Jul-30 00:09 UTC
Re: How can I identify a memory leak? using memorylogic..
Hi, Did you find the root of your problem? I am experiencing the same start / hang / reboot cycle for my rails app. -- Posted via http://www.ruby-forum.com/.
Hi Fernando, you should check your daemons/ crons/ background tasks if you have any running. try and stop them and see if things are ok. my problem was with a couple of daemons i was running. i stopped them and im running a system cron instead. took care of the hanging.. good luck! On Jul 30, 5:09 am, Fernando Perez <rails-mailing-l...-ARtvInVfO7ksV2N9l4h3zg@public.gmane.org> wrote:> Hi, > > Did you find the root of your problem? I am experiencing the same start > / hang / reboot cycle for my rails app. > -- > Posted viahttp://www.ruby-forum.com/.