[Backgroundrb-devel] Memory leak?

Ezra Zygmuntowicz
Mon, 07 Aug 2006 23:11:05 -0000

On Aug 7, 2006, at 3:51 PM, Joel Hayhurst wrote:

> I am using BackgrounDRb to resize images after they are accepted into
> our system.  However, it has become clear that this creates a memory
> leak.  I''m not sure exactly where the leak exists, but I
don''t think
> it''s in my own code as it is presently being used in production
and
> has resized thousands of images without a leak occurring, thanks to
> calling GC.start after every RMagick image read.
>
> I have it working like this.  First of all, I changed the start
> script to include environment.rb and therefore the ENTIRE Rails
> environment, because my Image model is an engine plugin.  Then, I
> have it perform this upon acceptance:
>
>          MiddleMan.new_worker :class => :resize_worker, :args =>
> self.id
>
> And here are the contents of ResizeWorker:
>
> class ResizeWorker < BackgrounDRb::Rails
>    def do_work(args)
>      image = Image.find(args)
>      image.resize_all
>      terminate
>    end
> end
>
> If I run this on a few dozen images simultaneously I watch as top
> starts slowly requiring more memory for backgroundrb.  Dropping into
> script/console tells me two things.
>
> For one, MiddleMan.jobs still has every job, despite my terminate()
> call.  Second, calling destroy_worker on any of these calls, or gc!
> (Time.now), does not free up any of the memory, even though the jobs
> disappear.  GC.start also does not free up any memory.
>
> If you have any advice on this, that would be great.
>
> Thanks,
> Joel


Hey Joel-

	Calling terminate does not immediately terminate the worker class.
All it does is flag the thread as ok to terminate when a
delete_worker call is made. So you will still need to call
delete_worker or gc! in order to clear your worker out of @jobs. As
far as actually getting the process memory back, you need to realize
that even if you call GC.start, that doesn''t mean that ruby will
automatically clean up all garbage. What platform are you on? And can
you confirm whether the real memory or the virtual memory decrease
after a delete_worker and a GC.start ?

	Where did you call GC.start from in your experiment? If you called
it from the command line then it will not effect the drb server. Try
this, add GC.start to the gc! method in the backgroundrb.rb file in
the plugin root and restarting your drb server. like this:

     def gc!(age)
       @timestamps.each do |job_key, timestamp|
         if timestamp[timestamp[:expire_type]] < age
           delete_worker(job_key)
         end
       end
       GC.start
     end

	And now run a bunch of workers until memory usage grows. Then run
MiddleMan.gc! Time.now from script/console and wait a little bit to
see if your memory usage goes down.  Please report back if this does
solve your problem and I will add it to the distro.

	Also RMagick is know to leak sometimes so I have been using
MiniMagick myself when I have needed to do image resizing in a worker.

	Let me know if none of these things solve your issue and I will try
to get to the bottom of this.

THanks
-Ezra