Backgroundrb devel - Jun 2006 - Rather ominous resource consumption figures

Hello,

Yesterday I posted about how to email from a worker. I decided to be a bit
piggy and just inhale all of rails. Sorted. I am noticing some rather
ominous behavior though.

The setup: My worker takes an uploaded image as a param and passes it (via
system call) to a C++ image processing engine. Take a look at these memory
consumption figures for successive runs uploading the identical image each
time:

All figures in megabytes

343 C++ image processing idle
365 max memory consumption during 3rd C++ image processing run

318 C++ image processing idle
340 max memory consumption during 2nd C++ image processing run

290 C++ image processing idle
310 max memory consumption during 1st C++ image processing run

256 fire up rails + lighttpd
225 fire up backgroundrb

207 base (Linux box idle)


As Sly Stone used to say ".. I want to take you hiya ..". Any thoughts
on
how to tame this beast?

Cheers,
Doug Turner
skype: dduuggllaa
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060630/fc2acaf7/attachment.html

Ah,

GC is your friend. I use RMagick to prep my image before processing by my
C++ engine. It was not releasing mems between successive calls to the
worker. I now call GC.start after I nolonger need the image object created
by RMagick.

However, t still does appear that  I have incurred a memory hit from the
initial call to my worker that is never released even when the worker dies.

-Doug


On 6/30/06, Douglass Turner <douglass.turner at gmail.com>
wrote:
>
> Hello,
>
> Yesterday I posted about how to email from a worker. I decided to be a bit
> piggy and just inhale all of rails. Sorted. I am noticing some rather
> ominous behavior though.
>
> The setup: My worker takes an uploaded image as a param and passes it (via
> system call) to a C++ image processing engine. Take a look at these memory
> consumption figures for successive runs uploading the identical image each
> time:
>
> All figures in megabytes
>
> 343 C++ image processing idle
> 365 max memory consumption during 3rd C++ image processing run
>
> 318 C++ image processing idle
> 340 max memory consumption during 2nd C++ image processing run
>
> 290 C++ image processing idle
> 310 max memory consumption during 1st C++ image processing run
>
> 256 fire up rails + lighttpd
> 225 fire up backgroundrb
>
> 207 base (Linux box idle)
>
>
> As Sly Stone used to say ".. I want to take you hiya ..". Any
thoughts on
> how to tame this beast?
>
> Cheers,
> Doug Turner
> skype: dduuggllaa
>


-- 
Doug Turner
mobile: 781 775 3708
skype: dduuggllaa
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060630/6e4f9308/attachment.html

Hey

On Jun 30, 2006, at 7:05 AM, Douglass Turner wrote:
> Ah,
>
> GC is your friend. I use RMagick to prep my image before processing  
> by my C++ engine. It was not releasing mems between successive  
> calls to the worker. I now call GC.start after I nolonger need the  
> image object created by RMagick.
>
> However, t still does appear that  I have incurred a memory hit  
> from the initial call to my worker that is never released even when  
> the worker dies.
>
> -Doug
>
>
> On 6/30/06, Douglass Turner <douglass.turner at gmail.com> wrote:
> Hello,
>
> Yesterday I posted about how to email from a worker. I decided to  
> be a bit piggy and just inhale all of rails. Sorted. I am noticing  
> some rather ominous behavior though.
>
> The setup: My worker takes an uploaded image as a param and passes  
> it (via system call) to a C++ image processing engine. Take a look  
> at these memory consumption figures for successive runs uploading  
> the identical image each time:
>
> All figures in megabytes
>
> 343 C++ image processing idle
> 365 max memory consumption during 3rd C++ image processing run
>
> 318 C++ image processing idle
> 340 max memory consumption during 2nd C++ image processing run
>
> 290 C++ image processing idle
> 310 max memory consumption during 1st C++ image processing run
>
> 256 fire up rails + lighttpd
> 225 fire up backgroundrb
>
> 207 base (Linux box idle)
>
>
> As Sly Stone used to say ".. I want to take you hiya ..". Any  
> thoughts on how to tame this beast?
>
> Cheers,
> Doug Turner
> skype: dduuggllaa
>
	In the current implementation you are responsible for calling  
MiddleMan.gc! with a timestamp. This will delete all workers older  
then the timestamp you pass to gc!. So this could acount for a  
perceived memory leak. So you either need to set up a cron script to  
expire things older then a certain age. Or you need to use  
MiddleMan.delete_worker(key). The reason for this is that the workers  
are held in the @jobs hash. So even after the worker is done working,  
it is not allowed to be GC''ed by ruby since the jobs hash still holds  
a ref to it. The next release will add a time to live param and a  
timer built in for expiring jobs and caches on a timed basis. For now  
you need to delete_worker when your done or use a cron script like this:

Here is a ruby script you could run from cron that will delete all  
workers and
cached items older then 30 minutes.

#!/usr/bin/env ruby
require "drb"
DRb.start_service
MiddleMan = DRbObject.new(nil, "druby://localhost:22222")
MiddleMan.gc!(Time.now - 60*30)

	Let me know if this helps you or if you still have problems. Then we  
can figure out what the issue is and correct it for you.

Cheers-
-Ezra


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060630/9e744c77/attachment.html