Rails - Nov 2006 - Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)

On Sat, 26 Aug 2006, Zed Shaw wrote:
> Howdy Folks,
>
> This release is after painstaking analysis of a memory leak that was
> reported by Bradley Taylor, reduced by myself, and then fixed after much
> work.  You should all thank Bradley for finding the bizarre fix.
>
> It turns out the Ruby has a memory leak when you use pretty much any
> thread locking primitive other than Sync (Mutex, Monitor, etc.):
>
> http://pastie.caboo.se/10194
>
> The fix (for whatever reason) is to use Sync and put it in a block:
>
> http://pastie.caboo.se/10317
>
> Those two scripts are mini versions of how Mongrel manages threads so
> that I could figure out a solution or get some input.  The graph is
> reported ram usage samples 1/second.  As you can see the first leaking
> graph goes up and doesn''t go down, the second (fixed) graph cycles
> properly.
>
> ** This is a Ruby issue, so if you have software using Mutex or Monitor,
> change to Sync now. **
>
> Tests of this latest pre-release show that the RAM is properly cycled by
> the GC and that it''s actually finally solved.  If you run your app
using
> this release and you still have a leak then use the memory debugging
> tools mongrel has to rule out your code (see below).
hi zed-

if you are really serious about fixing your leak i suggest you re-work your
tests.  as i mentioned before they have several race conditions, not least of
which that they both start a random number of threads, not 1000 as the code
suggests (you can easily confirm by printing out the number of times the
thread init loop executes).  further, sync.rb is the single ruby lib
i''ve had
memory issues with on production systems.  i have never managed to figure out
why that is...

in any case a careful script which allocated memory in a thread, waits for all
threads to finish allocation, checks memory, and then kills all threads before
checking again shows some suprising results which you should read carefully:


using mutex shows a nice cycle of memory freed:

     harp:~ > cat a.rb.mutex
     using: Mutex
     n: 420
     iter: 0
     with 420 threads holding memory : 44.0%
     with 0 threads holding memory : 13.0%
     iter: 1
     with 420 threads holding memory : 43.9%
     with 0 threads holding memory : 13.0%
     iter: 2
     with 420 threads holding memory : 44.1%
     with 0 threads holding memory : 13.3%
     iter: 3
     with 420 threads holding memory : 44.1%
     with 0 threads holding memory : 13.2%
     iter: 4
     with 420 threads holding memory : 44.0%
     with 0 threads holding memory : 13.5%
     iter: 5
     with 420 threads holding memory : 44.1%
     with 0 threads holding memory : 13.2%
     iter: 6
     with 420 threads holding memory : 43.9%
     with 0 threads holding memory : 13.2%
     iter: 7
     with 420 threads holding memory : 44.2%
     with 0 threads holding memory : 13.2%
     iter: 8
     with 420 threads holding memory : 44.1%
     with 0 threads holding memory : 13.5%
     iter: 9
     with 420 threads holding memory : 44.1%
     with 0 threads holding memory : 13.9%

using sync, on the other hand, looks leaky, though i''m not saying it
is.

     harp:~ > cat a.rb.sync
     using: Sync
     n: 420
     iter: 0
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 1.0%
     iter: 1
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 2.0%
     iter: 2
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 2.7%
     iter: 3
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 3.5%
     iter: 4
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 3.8%
     iter: 5
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 4.6%
     iter: 6
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 5.4%
     iter: 7
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 6.4%
     iter: 8
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 7.2%
     iter: 9
     with 420 threads holding memory : 43.7%
     with 0 threads holding memory : 8.1%

here is the code, note that it''s quite careful to only create a fixed
number
of threads, to wait for them to each init a mb of memory, and only then to
check memory usage.  likewise for checking after killing all threads -
it''s
done immediately after killing threads and running gc.  here is the code:

     harp:~ > cat a.rb
     require ''thread''
     require ''sync''

     class TestThreads
       def initialize which, n
         c = case which
           when /mutex/io
             Mutex
           when /sync/io
             Sync
         end
         @guard = c.new
         @n = Integer n
         puts "using: #{ c.name }"
         puts "n: #{ @n }"
       end

       def pct_mem # linux specific field pos i''m sure
         stdout = `ps v #{ Process.pid }`
         stdout.split(%r/\n/).last.strip.split(%r/\s+/)[8] +
''%''
       end

       def tq
         q = Queue.new
         t = Thread.new{
           mb = @guard.synchronize{ 0.chr * (2 ** 20) }
           q.push :ready
           Thread.stop
         }
         [t, q]
       end

       def run
         list = []
         10.times do |i|
           puts "iter: #{ i }"

           # load 1000 threads up
           @n.times{ list << tq }

           # wait for all threads to init memory with mb of data
           list.each{|t,q| q.pop}

           # show memory usage
           GC.start
           puts "with #{ list.size } threads holding memory : #{ pct_mem
}"

           # kill all threads - clean up
           list.each{|t,q| t.kill}
           list.clear
           sleep 1 until Thread.list.size == 1

           # show memory usage
           GC.start
           puts "with 0 threads holding memory : #{ pct_mem }"
         end
       end
     end

     $VERBOSE = nil
     STDOUT.sync = true
     Thread.abort_on_exception = true
     trap(''INT''){ exit }

     which, n, ignored = ARGV
     TestThreads.new(which, n).run


in any case, i''d carefully examine your tests (or the rails code if
that is
indeed what it''s modeled after) to make sure that they test
Mutex/Sync/Thread/Ruby and not your os virtual memory system and look closely
at the results again - like i said, i have had issues with sync.rb.

the point here is that it is probably the code in question and not Mutex per
se that was causing your process to grow in vmsize.

regards.

-a
-- 
to foster inner awareness, introspection, and reasoning is more efficient than
meditation and prayer.
- h.h. the 14th dalai lama
_______________________________________________
Rails mailing list
Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org
http://lists.rubyonrails.org/mailman/listinfo/rails

On Aug 26, 2006, at 5:12 AM, ara.t.howard-32lpuo7BZBA@public.gmane.org wrote:
> in any case, i''d carefully examine your tests (or the rails code
if
> that is
> indeed what it''s modeled after) to make sure that they test
> Mutex/Sync/Thread/Ruby and not your os virtual memory system and  
> look closely
> at the results again - like i said, i have had issues with sync.rb.
>
> the point here is that it is probably the code in question and not  
> Mutex per
> se that was causing your process to grow in vmsize.
>

I ran your test on OS/X looking at VSZ and RSS. And, like you,  
initially got Sync with no leak visible, and mutex with what looks  
like a bad leak. However, I notice that you only called GC once. I  
have a years old habit of always running GC at least three times when  
I really wanted GC to run (and in Java I had a loop that ran GC until  
it stopped freeing stuff which in some cases was eight or nine  
times). Superstition? Apparently not. On OS X, when I run GC three  
times neither sync nor mutex show a memory leak.

Zed, just for fun, try running GC a few times in a row (like  
GC.start; GC.start; GC.start) .

Cheers,
Bob

----
Bob Hutchison                  -- blogs at <http://www.recursive.ca/ 
hutch/>
Recursive Design Inc.          -- <http://www.recursive.ca/>
Raconteur                      -- <http://www.raconteur.info/>
xampl for Ruby                 -- <http://rubyforge.org/projects/xampl/>



_______________________________________________
Rails mailing list
Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org
http://lists.rubyonrails.org/mailman/listinfo/rails

On Aug 26, 2006, at 10:22 AM, Bob Hutchison wrote:
>
> On Aug 26, 2006, at 5:12 AM, ara.t.howard-32lpuo7BZBA@public.gmane.org
wrote:
>
>> in any case, i''d carefully examine your tests (or the rails
code
>> if that is
>> indeed what it''s modeled after) to make sure that they test
>> Mutex/Sync/Thread/Ruby and not your os virtual memory system and  
>> look closely
>> at the results again - like i said, i have had issues with sync.rb.
>>
>> the point here is that it is probably the code in question and not  
>> Mutex per
>> se that was causing your process to grow in vmsize.
>>
>
>
> I ran your test on OS/X looking at VSZ and RSS. And, like you,  
> initially got Sync with no leak visible, and mutex with what looks  
> like a bad leak. However, I notice that you only called GC once. I  
> have a years old habit of always running GC at least three times  
> when I really wanted GC to run (and in Java I had a loop that ran  
> GC until it stopped freeing stuff which in some cases was eight or  
> nine times). Superstition? Apparently not. On OS X, when I run GC  
> three times neither sync nor mutex show a memory leak.
>
> Zed, just for fun, try running GC a few times in a row (like  
> GC.start; GC.start; GC.start)
Well I tried your test on OS X. The Sync had no problem, the mutex  
showed the memory growth (though it eventually (fifth iteration I  
think) cleaned itself up). I modified your test to create exactly  
1000 threads and call GC three times at the end, things were better,  
i.e. it released its memory more quickly than without, but still not  
good. I ended up with:

       GC.start
       `sync; sync; sync`
       sleep 1
       GC.start
       `sync; sync; sync`
       sleep 1
       GC.start
       `sync; sync; sync`
       sleep 1
       GC.start
       `sync; sync; sync`
       sleep 1

and this made a bigger difference. The memory usage was much more  
tightly bound.

(And yes, the three calls to sync are also on purpose... in the late  
70s through the 80s, calling sync once didn''t guarantee anything, you  
had to call it a few times, three generally worked... I don''t know  
the current situation because it is easy enough to type  
sync;sync;sync (well, in truth, I usually alias sync to the three  
calls))

But of course, the point is that despite appearances there is likely  
no memory leak at all on OS X, just some kind of long term cycle of  
process resource utilisation -- this is a complex situation, Ruby GC,  
process resource utilisation/optimisation, and system optimisation  
all interacting. Who knows what''s actually going on.

So.

Cheers,
Bob

----
Bob Hutchison                  -- blogs at <http://www.recursive.ca/ 
hutch/>
Recursive Design Inc.          -- <http://www.recursive.ca/>
Raconteur                      -- <http://www.raconteur.info/>
xampl for Ruby                 -- <http://rubyforge.org/projects/xampl/>



_______________________________________________
Rails mailing list
Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org
http://lists.rubyonrails.org/mailman/listinfo/rails

Perhaps it''s just me, but I don''t like cross-posting. Please
post to
each mailing list separately, if you don''t mind. :)

On 8/25/06, Zed Shaw <zedshaw at zedshaw.com>
wrote:
> Howdy Folks,
>
> This release is after painstaking analysis of a memory leak that was
> reported by Bradley Taylor, reduced by myself, and then fixed after much
> work.  You should all thank Bradley for finding the bizarre fix.
>
> It turns out the Ruby has a memory leak when you use pretty much any
> thread locking primitive other than Sync (Mutex, Monitor, etc.):
>
> http://pastie.caboo.se/10194
>
> The fix (for whatever reason) is to use Sync and put it in a block:
>
> http://pastie.caboo.se/10317
>
> Those two scripts are mini versions of how Mongrel manages threads so
> that I could figure out a solution or get some input.  The graph is
> reported ram usage samples 1/second.  As you can see the first leaking
> graph goes up and doesn''t go down, the second (fixed) graph cycles
> properly.
>
> ** This is a Ruby issue, so if you have software using Mutex or Monitor,
> change to Sync now. **
>
> Tests of this latest pre-release show that the RAM is properly cycled by
> the GC and that it''s actually finally solved.  If you run your app
using
> this release and you still have a leak then use the memory debugging
> tools mongrel has to rule out your code (see below).
>
>
> CHANGES
>
> * No more allow_concurrency.  Until Ruby''s fixed I can''t
let people do
> this anymore.
> * USR1 debugging.  If you''re wondering about how
Mongrel''s locking of
> Rails impacts your application, or what is causing BAD CLIENT then just
> hit your mongrel_rails with USR1 and Mongrel will tell you.
> * More extensive and accurate memory debugging.  Use -B and look at the
> log/mongrel_log/objects.log to get a good idea of counts of objects,
> delta changes in counts, and mean+standard deviation lengths of objects
> with length methods.
> * Fixes a few places where sockets are closed and left in CLOSE_WAIT.
>
>
> INSTALLING
>
> As per usual:
>
> sudo gem install mongrel --source=http://mongrel.rubyforge.org/releases/
>
> Initial tests show it works on 1.8.5 and is actually faster, but this is
> unsupported for now.
>
>
> TESTING THIS RELEASE
>
> If you want to test the memory leak, here''s the process:
>
> 1) Start your application in *production* mode:
>         mongrel_rails start -e production
>
> 2) Hit it with USR1:
>         killall -USR1 mongrel_rails
>
> 3) Start running something that prints out the ram (here''s my fish
> code):
>         while sleep 1
>           ps aux | grep mongrel_rails | grep -v grep | grep -v gvim | ruby
-aln
> -e "puts split[4 .. 5].join('','')"
>         end
>
> 4) Thrash a simple rails controller with httperf:
>         httperf --server 127.0.0.1 --port 3000 --num-conns 1000 --rate 120
> --uri /testuri
>
> What you want to do is adjust num-conns and rate until Mongrel reports
> "X threads waiting for /testuri..."
>
> The bug only manifests itself when threads pile up behind the guard
> around Rails dispatching.  This is also how you''d find out which
Rails
> actions are too slow.
>
>
> Please report any bugs you find in this release, and a Win32 release
> will come out after I''m sure it works for everyone else.
>
>
> --
> Zed A. Shaw
> http://www.zedshaw.com/
> http://mongrel.rubyforge.org/
> http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

-- 
Cheers,

Kevin

"Any sufficiently advanced technology is indistinguishable from
Magic." - Arthur C. Clarke

Zed,

Thanks for the hard work, when I get home I''ll try running this one  
on my server and see how it holds!

-carl

On Aug 25, 2006, at 11:31 PM, Zed Shaw wrote:
> Howdy Folks,
>
> This release is after painstaking analysis of a memory leak that was
> reported by Bradley Taylor, reduced by myself, and then fixed after  
> much
> work.  You should all thank Bradley for finding the bizarre fix.
>
> It turns out the Ruby has a memory leak when you use pretty much any
> thread locking primitive other than Sync (Mutex, Monitor, etc.):
>
> http://pastie.caboo.se/10194
>
> The fix (for whatever reason) is to use Sync and put it in a block:
>
> http://pastie.caboo.se/10317
>
> Those two scripts are mini versions of how Mongrel manages threads so
> that I could figure out a solution or get some input.  The graph is
> reported ram usage samples 1/second.  As you can see the first leaking
> graph goes up and doesn''t go down, the second (fixed) graph cycles
> properly.
>
> ** This is a Ruby issue, so if you have software using Mutex or  
> Monitor,
> change to Sync now. **
>
> Tests of this latest pre-release show that the RAM is properly  
> cycled by
> the GC and that it''s actually finally solved.  If you run your app
> using
> this release and you still have a leak then use the memory debugging
> tools mongrel has to rule out your code (see below).
>
>
> CHANGES
>
> * No more allow_concurrency.  Until Ruby''s fixed I can''t
let people do
> this anymore.
> * USR1 debugging.  If you''re wondering about how
Mongrel''s locking of
> Rails impacts your application, or what is causing BAD CLIENT then  
> just
> hit your mongrel_rails with USR1 and Mongrel will tell you.
> * More extensive and accurate memory debugging.  Use -B and look at  
> the
> log/mongrel_log/objects.log to get a good idea of counts of objects,
> delta changes in counts, and mean+standard deviation lengths of  
> objects
> with length methods.
> * Fixes a few places where sockets are closed and left in CLOSE_WAIT.
>
>
> INSTALLING
>
> As per usual:
>
> sudo gem install mongrel --source=http://mongrel.rubyforge.org/ 
> releases/
>
> Initial tests show it works on 1.8.5 and is actually faster, but  
> this is
> unsupported for now.
>
>
> TESTING THIS RELEASE
>
> If you want to test the memory leak, here''s the process:
>
> 1) Start your application in *production* mode:
> 	mongrel_rails start -e production
>
> 2) Hit it with USR1:
> 	killall -USR1 mongrel_rails
>
> 3) Start running something that prints out the ram (here''s my fish
> code):
> 	while sleep 1
> 	  ps aux | grep mongrel_rails | grep -v grep | grep -v gvim | ruby  
> -aln
> -e "puts split[4 .. 5].join('','')"
> 	end
>
> 4) Thrash a simple rails controller with httperf:
> 	httperf --server 127.0.0.1 --port 3000 --num-conns 1000 --rate 120
> --uri /testuri
>
> What you want to do is adjust num-conns and rate until Mongrel reports
> "X threads waiting for /testuri..."
>
> The bug only manifests itself when threads pile up behind the guard
> around Rails dispatching.  This is also how you''d find out which
Rails
> actions are too slow.
>
>
> Please report any bugs you find in this release, and a Win32 release
> will come out after I''m sure it works for everyone else.
>
>
> -- 
> Zed A. Shaw
> http://www.zedshaw.com/
> http://mongrel.rubyforge.org/
> http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

On Sat, 2006-08-26 at 11:20 -0400, Bob Hutchison wrote:
> On Aug 26, 2006, at 10:22 AM, Bob Hutchison wrote:
> 
> Well I tried your test on OS X. The Sync had no problem, the mutex  
> showed the memory growth (though it eventually (fifth iteration I  
> think) cleaned itself up). I modified your test to create exactly  
> 1000 threads and call GC three times at the end, things were better,  
> i.e. it released its memory more quickly than without, but still not  
> good. I ended up with:
Thanks Bob, but I''ve gotta say this one more time, this test is not
about 1000 threads.  The test is about how *Mongrel* processes threads,
a specific bug when many threads are put into a ThreadGroup and wait
behind a Mutex, and how to stop that from leaking.

If you change the way the test is written so that it creates exactly
1000 threads, then this isn''t simulating Mongrel.  You''re most
likely
using additional synchronization primitives not used in Mongrel so your
test is wrong.  I mean, Mongrel doesn''t wait for 1000 threads, it just
cranks on them and sometimes it''s too slow so you build a log jam.

In this situation, we were seeing memory leaks.  Other people also
report the memory leaks and even reported this fixed it in other systems
unrelated to Mongrel.  Yes, you can write something else to not have
memory leaks, but then you''re not testing our leaking situation.  The
point is that the script with Mutex leaks, the one with Sync doesn''t.

The number 1000 was just a number picked at random so that I knew when
to stop simulating requests and start calling GC.  And, if you notice
there''s a loop that calls GC.start about 3-4 times.  There''s
no way I''m
going to put into Mongrel''s loop three calls to GC.

-- 
Zed A. Shaw
http://www.zedshaw.com/
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

On Sat, 2006-08-26 at 11:12 -0600, Kevin Williams wrote:
> Perhaps it''s just me, but I don''t like cross-posting.
Please post to
> each mailing list separately, if you don''t mind. :)
Kevin, how is it different if I put the mailing lists into one e-mail or
three if the message is the same.  Considering I don''t have tons of
time
and needed to get the word out so I can make the next release final, it
seems stupid to spend my time on some kind of "vanity send".

Is there something about your mail client that can''t handle this?


-- 
Zed A. Shaw
http://www.zedshaw.com/
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

On 8/26/06, Zed Shaw <zedshaw at zedshaw.com> wrote:
> Thanks Bob, but I''ve gotta say this one more time, this test is
not
> about 1000 threads.  The test is about how *Mongrel* processes threads,
> a specific bug when many threads are put into a ThreadGroup and wait
> behind a Mutex, and how to stop that from leaking.
>
> If you change the way the test is written so that it creates exactly
> 1000 threads, then this isn''t simulating Mongrel.  You''re
most likely
> using additional synchronization primitives not used in Mongrel so your
> test is wrong.  I mean, Mongrel doesn''t wait for 1000 threads, it
just
> cranks on them and sometimes it''s too slow so you build a log jam.
>
> In this situation, we were seeing memory leaks.  Other people also
> report the memory leaks and even reported this fixed it in other systems
> unrelated to Mongrel.  Yes, you can write something else to not have
> memory leaks, but then you''re not testing our leaking situation. 
The
> point is that the script with Mutex leaks, the one with Sync
doesn''t.
I''ve been testing with your pasted scripts and variations all day, and
I can not reproduce any results that indicate that Mutex leaks.

In your pasted script, the primary difference in behavior between
Mutex and Sync (which, under the covers, use an identical algorithm
for locking, though they differ in unlock semanticsi; mutex pops a
single thread off the waiting list and wakes it while Sync wakes them
all, letting one grab the other lock and the others go back to
waiting) is that Sync is slower.

All that I have to do in order to eliminate the phantom memory leaking
by Mutex is to insert a very small delay at the end of each
synchronized block.  On my test system, select(nil,nil,nil,0.025) does
the trick.

In your test scripts, this causes the Mutex variant to launch fewer
total threads, similar to the Sync variant (on my box, an iteration
with the Mutex variant as you pasted it actually ends up creating
about 2100 threads, while the Sync variant is around 1650, because
Sync is slower so it takes longer for threads to fall out of the
threadgroup as you are adding new ones in).

On variations that launch exactly 1000 threads (which is easily done
without using any other locking primitives), the difference boils down
to how fast objects can be created and how long the GC has to clean
them up.  Change the test() method to do something that creates some
strings and other objects, and it becomes clear very quickly that if
there is a burst of activity, a bunch of threads locking with a Mutex
outrun the GC''s ability to clean it up.  Memory consumption rises.

This also seems consistent on my tests so far comparing 1.8.4 to
1.8.5, which you mentioned seemed to exhibit worse RAM use
characteristics.  I still need to dig into the differences in the GC
subsystem code between the two versions, but the experimental evidence
that I have suggests that in 1.8.5 it is taking longer to get around
to cleaning up objects.  It seems to be faster when it does, as my
overall throughput is about 10% faster on 1.8.5, but I don''t think
I''m
liking the tradeoff that I am seeing with memory consumption when it
is pounded with objects.  Something looks wrong, there, but it''s not
related to Mutex.


Kirk Haines

I have done some testings of the code on Win32 with ruby 1.8.4, please
check the included pictures [1].

I have compared both test scripts, using Process Explorer to monitor
their memory use and CPU use.

So far, Win32 behave way different than *nix in this matter. If you
see the pngs, using mutex actually releases memory (GC) and finish
faster than sync.

Sync, on the contrary, never released memory (like GC wasn''t called at
all) and took longer to finish (I stopped it because took more than 6
minutes).

I could conclude that ruby implementations are too different to
compare between installations and platforms...

A solution for the memory leaking should be tested on any platform
before taking final decisions.

On 8/26/06, Kirk Haines <wyhaines at gmail.com> wrote:
[snip]
>
> On variations that launch exactly 1000 threads (which is easily done
> without using any other locking primitives), the difference boils down
> to how fast objects can be created and how long the GC has to clean
> them up.  Change the test() method to do something that creates some
> strings and other objects, and it becomes clear very quickly that if
> there is a burst of activity, a bunch of threads locking with a Mutex
> outrun the GC''s ability to clean it up.  Memory consumption rises.
Please provide a sample script (pastie?) and I will test it too on Win32.
> This also seems consistent on my tests so far comparing 1.8.4 to
> 1.8.5, which you mentioned seemed to exhibit worse RAM use
> characteristics.  I still need to dig into the differences in the GC
> subsystem code between the two versions, but the experimental evidence
> that I have suggests that in 1.8.5 it is taking longer to get around
> to cleaning up objects.  It seems to be faster when it does, as my
> overall throughput is about 10% faster on 1.8.5, but I don''t think
I''m
> liking the tradeoff that I am seeing with memory consumption when it
> is pounded with objects.  Something looks wrong, there, but it''s
not
> related to Mutex.
I agree with you that something is wrong, not with mutex, but with ruby itself.

People will start yelling me about this, but is true.

Not just with mutex, but with threads (oh lovely select()) and
sockets. On every platform exceptions are thrown differently or in
some cases, not raised at all!

We must investigate this to find a workaround that works "almost good"
on every platform.

-- 
Luis Lavena
Multimedia systems
-
Leaders are made, they are not born. They are made by hard effort,
which is the price which all of us must pay to achieve any goal that
is worthwhile.
Vince Lombardi

[1] Cannot add pngs to this mail (40K limit) so uploaded them to my server:
http://www.mmediasys.com/mongrel/mutex.png
http://www.mmediasys.com/mongrel/sync-1.png
http://www.mmediasys.com/mongrel/sync-2.png

On 8/26/06, Zed Shaw <zedshaw at zedshaw.com>
wrote:
> On Sat, 2006-08-26 at 11:12 -0600, Kevin Williams wrote:
> > Perhaps it''s just me, but I don''t like
cross-posting. Please post to
> > each mailing list separately, if you don''t mind. :)
>
> Kevin, how is it different if I put the mailing lists into one e-mail or
> three if the message is the same.  Considering I don''t have tons
of time
> and needed to get the word out so I can make the next release final, it
> seems stupid to spend my time on some kind of "vanity send".
>
> Is there something about your mail client that can''t handle this?
>
GMail gets the labels all crossed up.

Sorry to bother you.


-- 
Cheers,

Kevin

"Any sufficiently advanced technology is indistinguishable from
Magic." - Arthur C. Clarke

On Sat, 2006-08-26 at 20:26 -0300, Luis Lavena wrote:
> I have done some testings of the code on Win32 with ruby 1.8.4, please
> check the included pictures [1].
> 
Thank you Luis, *this* is what I need.  A test with Mongrel code and
evidence that it''s not fixed on a particular platform.  Thank you for
not guessing at random theories about why it works this way, and instead
doing the test. 

It''s also incredibly sad since it means that one way works on POSIX and
the other works on Win32.

I''ll take a look at this tomorrow and see what happens.

-- 
Zed A. Shaw
http://www.zedshaw.com/
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

On 8/27/06, Zed Shaw <zedshaw at zedshaw.com>
wrote:
> On Sat, 2006-08-26 at 20:26 -0300, Luis Lavena wrote:
> > I have done some testings of the code on Win32 with ruby 1.8.4, please
> > check the included pictures [1].
> >
>
> Thank you Luis, *this* is what I need.  A test with Mongrel code and
> evidence that it''s not fixed on a particular platform.  Thank you
for
> not guessing at random theories about why it works this way, and instead
> doing the test.
>
> It''s also incredibly sad since it means that one way works on
POSIX and
> the other works on Win32.
>
Yeah, I was shocked when I saw that.
> I''ll take a look at this tomorrow and see what happens.
What about a new approach to RailsHandler? I mean:

If just 1 thread can access Rails dispatcher, what if we
serialize/enqueue the requests?

Maybe will be too complicated, but MonitorMixin maybe could help:

http://www.ruby-doc.org/stdlib/libdoc/monitor/rdoc/classes/MonitorMixin.html

Anyway, talk this sunday about mongrel_service

Regards,

-- 
Luis Lavena
Multimedia systems
-
Leaders are made, they are not born. They are made by hard effort,
which is the price which all of us must pay to achieve any goal that
is worthwhile.
Vince Lombardi

On Aug 26, 2006, at 6:02 PM, Zed Shaw wrote:
> On Sat, 2006-08-26 at 11:20 -0400, Bob Hutchison wrote:
>> On Aug 26, 2006, at 10:22 AM, Bob Hutchison wrote:
>>
>
>> Well I tried your test on OS X. The Sync had no problem, the mutex
>> showed the memory growth (though it eventually (fifth iteration I
>> think) cleaned itself up). I modified your test to create exactly
>> 1000 threads and call GC three times at the end, things were better,
>> i.e. it released its memory more quickly than without, but still not
>> good. I ended up with:
>
> Thanks Bob, but I''ve gotta say this one more time, this test is
not
> about 1000 threads.  The test is about how *Mongrel* processes  
> threads,
> a specific bug when many threads are put into a ThreadGroup and wait
> behind a Mutex, and how to stop that from leaking.
I know it isn''t about 1000 threads, but when testing it *really*  
helps sometimes to lock as much down as possible. I was getting  
anywhere from 1005 through 1040 threads and that variation was too  
much. I like having exactly *one* thing vary between tests.

No matter. I posted on the ruby-talk list the result of some mucking  
about I was doing this morning. It seems that on OS X at least, the  
Mutex is holding threads in memory. If you delete the reference to  
the Mutex and then garbage collect you get a nice stable memory  
utilisation.

I have no idea if you can make use of this information. Is there some  
point at which you know you can clobber the Mutex in Mongrel?

Cheers,
Bob

----
Bob Hutchison                  -- blogs at <http://www.recursive.ca/ 
hutch/>
Recursive Design Inc.          -- <http://www.recursive.ca/>
Raconteur                      -- <http://www.raconteur.info/>
xampl for Ruby                 -- <http://rubyforge.org/projects/xampl/>

On 8/27/06, Kevin Williams <kevwil at gmail.com>
wrote:
> GMail gets the labels all crossed up.
I use gmail and the labels worked fine. The message was labeled with
rails, mongrel and ruby-talk labels. It shows only one message
although the mail server must have received three copies. One from
each list.

Pedro.

My mongrel filter is this:

Matches: to:(mongrel-users at rubyforge.org)
Do this: Skip Inbox, Apply label "mongrel"

I''m have unsubscribed from the rails list anyway, so it won''t
be an
issue anymore. I''ll just use ruby-forum to keep track of Rails stuff.


On 8/27/06, Pedro C?rte-Real <pedro at pedrocr.net>
wrote:
> On 8/27/06, Kevin Williams <kevwil at gmail.com> wrote:
> > GMail gets the labels all crossed up.
>
> I use gmail and the labels worked fine. The message was labeled with
> rails, mongrel and ruby-talk labels. It shows only one message
> although the mail server must have received three copies. One from
> each list.
>
> Pedro.
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

-- 
Cheers,

Kevin

"Any sufficiently advanced technology is indistinguishable from
Magic." - Arthur C. Clarke

Howdy Folks,

This release is after painstaking analysis of a memory leak that was
reported by Bradley Taylor, reduced by myself, and then fixed after much
work.  You should all thank Bradley for finding the bizarre fix.

It turns out the Ruby has a memory leak when you use pretty much any
thread locking primitive other than Sync (Mutex, Monitor, etc.):

http://pastie.caboo.se/10194

The fix (for whatever reason) is to use Sync and put it in a block:

http://pastie.caboo.se/10317

Those two scripts are mini versions of how Mongrel manages threads so
that I could figure out a solution or get some input.  The graph is
reported ram usage samples 1/second.  As you can see the first leaking
graph goes up and doesn''t go down, the second (fixed) graph cycles
properly.

** This is a Ruby issue, so if you have software using Mutex or Monitor,
change to Sync now. **

Tests of this latest pre-release show that the RAM is properly cycled by
the GC and that it''s actually finally solved.  If you run your app
using
this release and you still have a leak then use the memory debugging
tools mongrel has to rule out your code (see below).


CHANGES

* No more allow_concurrency.  Until Ruby''s fixed I can''t let
people do
this anymore.
* USR1 debugging.  If you''re wondering about how Mongrel''s
locking of
Rails impacts your application, or what is causing BAD CLIENT then just
hit your mongrel_rails with USR1 and Mongrel will tell you.
* More extensive and accurate memory debugging.  Use -B and look at the
log/mongrel_log/objects.log to get a good idea of counts of objects,
delta changes in counts, and mean+standard deviation lengths of objects
with length methods.
* Fixes a few places where sockets are closed and left in CLOSE_WAIT.


INSTALLING

As per usual:

sudo gem install mongrel --source=http://mongrel.rubyforge.org/releases/

Initial tests show it works on 1.8.5 and is actually faster, but this is
unsupported for now.


TESTING THIS RELEASE

If you want to test the memory leak, here''s the process:

1) Start your application in *production* mode: 
	mongrel_rails start -e production

2) Hit it with USR1:  
	killall -USR1 mongrel_rails

3) Start running something that prints out the ram (here''s my fish
code):
	while sleep 1
	  ps aux | grep mongrel_rails | grep -v grep | grep -v gvim | ruby -aln
-e "puts split[4 .. 5].join('','')"
	end

4) Thrash a simple rails controller with httperf:
	httperf --server 127.0.0.1 --port 3000 --num-conns 1000 --rate 120
--uri /testuri

What you want to do is adjust num-conns and rate until Mongrel reports
"X threads waiting for /testuri..."

The bug only manifests itself when threads pile up behind the guard
around Rails dispatching.  This is also how you''d find out which Rails
actions are too slow.


Please report any bugs you find in this release, and a Win32 release
will come out after I''m sure it works for everyone else.


-- 
Zed A. Shaw
http://www.zedshaw.com/
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

On Sat, 26 Aug 2006, Zed Shaw wrote:
> Howdy Folks,
>
> This release is after painstaking analysis of a memory leak that was
> reported by Bradley Taylor, reduced by myself, and then fixed after much
> work.  You should all thank Bradley for finding the bizarre fix.
>
> It turns out the Ruby has a memory leak when you use pretty much any
> thread locking primitive other than Sync (Mutex, Monitor, etc.):
>
> http://pastie.caboo.se/10194
>
> The fix (for whatever reason) is to use Sync and put it in a block:
>
> http://pastie.caboo.se/10317
>
> Those two scripts are mini versions of how Mongrel manages threads so
> that I could figure out a solution or get some input.  The graph is
> reported ram usage samples 1/second.  As you can see the first leaking
> graph goes up and doesn''t go down, the second (fixed) graph cycles
> properly.
>
> ** This is a Ruby issue, so if you have software using Mutex or Monitor,
> change to Sync now. **
>
> Tests of this latest pre-release show that the RAM is properly cycled by
> the GC and that it''s actually finally solved.  If you run your app
using
> this release and you still have a leak then use the memory debugging
> tools mongrel has to rule out your code (see below).
hi zed-

if you are really serious about fixing your leak i suggest you re-work your
tests.  as i mentioned before they have several race conditions, not least of
which that they both start a random number of threads, not 1000 as the code
suggests (you can easily confirm by printing out the number of times the
thread init loop executes).  further, sync.rb is the single ruby lib
i''ve had
memory issues with on production systems.  i have never managed to figure out
why that is...

in any case a careful script which allocated memory in a thread, waits for all
threads to finish allocation, checks memory, and then kills all threads before
checking again shows some suprising results which you should read carefully:


using mutex shows a nice cycle of memory freed:

     harp:~ > cat a.rb.mutex
     using: Mutex
     n: 420
     iter: 0
     with 420 threads holding memory : 44.0%
     with 0 threads holding memory : 13.0%
     iter: 1
     with 420 threads holding memory : 43.9%
     with 0 threads holding memory : 13.0%
     iter: 2
     with 420 threads holding memory : 44.1%
     with 0 threads holding memory : 13.3%
     iter: 3
     with 420 threads holding memory : 44.1%
     with 0 threads holding memory : 13.2%
     iter: 4
     with 420 threads holding memory : 44.0%
     with 0 threads holding memory : 13.5%
     iter: 5
     with 420 threads holding memory : 44.1%
     with 0 threads holding memory : 13.2%
     iter: 6
     with 420 threads holding memory : 43.9%
     with 0 threads holding memory : 13.2%
     iter: 7
     with 420 threads holding memory : 44.2%
     with 0 threads holding memory : 13.2%
     iter: 8
     with 420 threads holding memory : 44.1%
     with 0 threads holding memory : 13.5%
     iter: 9
     with 420 threads holding memory : 44.1%
     with 0 threads holding memory : 13.9%

using sync, on the other hand, looks leaky, though i''m not saying it
is.

     harp:~ > cat a.rb.sync
     using: Sync
     n: 420
     iter: 0
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 1.0%
     iter: 1
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 2.0%
     iter: 2
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 2.7%
     iter: 3
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 3.5%
     iter: 4
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 3.8%
     iter: 5
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 4.6%
     iter: 6
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 5.4%
     iter: 7
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 6.4%
     iter: 8
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 7.2%
     iter: 9
     with 420 threads holding memory : 43.7%
     with 0 threads holding memory : 8.1%

here is the code, note that it''s quite careful to only create a fixed
number
of threads, to wait for them to each init a mb of memory, and only then to
check memory usage.  likewise for checking after killing all threads -
it''s
done immediately after killing threads and running gc.  here is the code:

     harp:~ > cat a.rb
     require ''thread''
     require ''sync''

     class TestThreads
       def initialize which, n
         c = case which
           when /mutex/io
             Mutex
           when /sync/io
             Sync
         end
         @guard = c.new
         @n = Integer n
         puts "using: #{ c.name }"
         puts "n: #{ @n }"
       end

       def pct_mem # linux specific field pos i''m sure
         stdout = `ps v #{ Process.pid }`
         stdout.split(%r/\n/).last.strip.split(%r/\s+/)[8] +
''%''
       end

       def tq
         q = Queue.new
         t = Thread.new{
           mb = @guard.synchronize{ 0.chr * (2 ** 20) }
           q.push :ready
           Thread.stop
         }
         [t, q]
       end

       def run
         list = []
         10.times do |i|
           puts "iter: #{ i }"

           # load 1000 threads up
           @n.times{ list << tq }

           # wait for all threads to init memory with mb of data
           list.each{|t,q| q.pop}

           # show memory usage
           GC.start
           puts "with #{ list.size } threads holding memory : #{ pct_mem
}"

           # kill all threads - clean up
           list.each{|t,q| t.kill}
           list.clear
           sleep 1 until Thread.list.size == 1

           # show memory usage
           GC.start
           puts "with 0 threads holding memory : #{ pct_mem }"
         end
       end
     end

     $VERBOSE = nil
     STDOUT.sync = true
     Thread.abort_on_exception = true
     trap(''INT''){ exit }

     which, n, ignored = ARGV
     TestThreads.new(which, n).run


in any case, i''d carefully examine your tests (or the rails code if
that is
indeed what it''s modeled after) to make sure that they test
Mutex/Sync/Thread/Ruby and not your os virtual memory system and look closely
at the results again - like i said, i have had issues with sync.rb.

the point here is that it is probably the code in question and not Mutex per
se that was causing your process to grow in vmsize.

regards.

-a
-- 
to foster inner awareness, introspection, and reasoning is more efficient than
meditation and prayer.
- h.h. the 14th dalai lama

On Sat, 26 Aug 2006, Zed Shaw wrote:
> Howdy Folks,
>
> This release is after painstaking analysis of a memory leak that was
> reported by Bradley Taylor, reduced by myself, and then fixed after much
> work.  You should all thank Bradley for finding the bizarre fix.
>
> It turns out the Ruby has a memory leak when you use pretty much any
> thread locking primitive other than Sync (Mutex, Monitor, etc.):
>
> http://pastie.caboo.se/10194
>
> The fix (for whatever reason) is to use Sync and put it in a block:
>
> http://pastie.caboo.se/10317
>
> Those two scripts are mini versions of how Mongrel manages threads so
> that I could figure out a solution or get some input.  The graph is
> reported ram usage samples 1/second.  As you can see the first leaking
> graph goes up and doesn''t go down, the second (fixed) graph cycles
> properly.
>
> ** This is a Ruby issue, so if you have software using Mutex or Monitor,
> change to Sync now. **
>
> Tests of this latest pre-release show that the RAM is properly cycled by
> the GC and that it''s actually finally solved.  If you run your app
using
> this release and you still have a leak then use the memory debugging
> tools mongrel has to rule out your code (see below).
hi zed-

if you are really serious about fixing your leak i suggest you re-work your
tests.  as i mentioned before they have several race conditions, not least of
which that they both start a random number of threads, not 1000 as the code
suggests (you can easily confirm by printing out the number of times the
thread init loop executes).  further, sync.rb is the single ruby lib
i''ve had
memory issues with on production systems.  i have never managed to figure out
why that is...

in any case a careful script which allocated memory in a thread, waits for all
threads to finish allocation, checks memory, and then kills all threads before
checking again shows some suprising results which you should read carefully:


using mutex shows a nice cycle of memory freed:

     harp:~ > cat a.rb.mutex
     using: Mutex
     n: 420
     iter: 0
     with 420 threads holding memory : 44.0%
     with 0 threads holding memory : 13.0%
     iter: 1
     with 420 threads holding memory : 43.9%
     with 0 threads holding memory : 13.0%
     iter: 2
     with 420 threads holding memory : 44.1%
     with 0 threads holding memory : 13.3%
     iter: 3
     with 420 threads holding memory : 44.1%
     with 0 threads holding memory : 13.2%
     iter: 4
     with 420 threads holding memory : 44.0%
     with 0 threads holding memory : 13.5%
     iter: 5
     with 420 threads holding memory : 44.1%
     with 0 threads holding memory : 13.2%
     iter: 6
     with 420 threads holding memory : 43.9%
     with 0 threads holding memory : 13.2%
     iter: 7
     with 420 threads holding memory : 44.2%
     with 0 threads holding memory : 13.2%
     iter: 8
     with 420 threads holding memory : 44.1%
     with 0 threads holding memory : 13.5%
     iter: 9
     with 420 threads holding memory : 44.1%
     with 0 threads holding memory : 13.9%

using sync, on the other hand, looks leaky, though i''m not saying it
is.

     harp:~ > cat a.rb.sync
     using: Sync
     n: 420
     iter: 0
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 1.0%
     iter: 1
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 2.0%
     iter: 2
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 2.7%
     iter: 3
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 3.5%
     iter: 4
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 3.8%
     iter: 5
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 4.6%
     iter: 6
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 5.4%
     iter: 7
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 6.4%
     iter: 8
     with 420 threads holding memory : 43.8%
     with 0 threads holding memory : 7.2%
     iter: 9
     with 420 threads holding memory : 43.7%
     with 0 threads holding memory : 8.1%

here is the code, note that it''s quite careful to only create a fixed
number
of threads, to wait for them to each init a mb of memory, and only then to
check memory usage.  likewise for checking after killing all threads -
it''s
done immediately after killing threads and running gc.  here is the code:

     harp:~ > cat a.rb
     require ''thread''
     require ''sync''

     class TestThreads
       def initialize which, n
         c = case which
           when /mutex/io
             Mutex
           when /sync/io
             Sync
         end
         @guard = c.new
         @n = Integer n
         puts "using: #{ c.name }"
         puts "n: #{ @n }"
       end

       def pct_mem # linux specific field pos i''m sure
         stdout = `ps v #{ Process.pid }`
         stdout.split(%r/\n/).last.strip.split(%r/\s+/)[8] +
''%''
       end

       def tq
         q = Queue.new
         t = Thread.new{
           mb = @guard.synchronize{ 0.chr * (2 ** 20) }
           q.push :ready
           Thread.stop
         }
         [t, q]
       end

       def run
         list = []
         10.times do |i|
           puts "iter: #{ i }"

           # load 1000 threads up
           @n.times{ list << tq }

           # wait for all threads to init memory with mb of data
           list.each{|t,q| q.pop}

           # show memory usage
           GC.start
           puts "with #{ list.size } threads holding memory : #{ pct_mem
}"

           # kill all threads - clean up
           list.each{|t,q| t.kill}
           list.clear
           sleep 1 until Thread.list.size == 1

           # show memory usage
           GC.start
           puts "with 0 threads holding memory : #{ pct_mem }"
         end
       end
     end

     $VERBOSE = nil
     STDOUT.sync = true
     Thread.abort_on_exception = true
     trap(''INT''){ exit }

     which, n, ignored = ARGV
     TestThreads.new(which, n).run


in any case, i''d carefully examine your tests (or the rails code if
that is
indeed what it''s modeled after) to make sure that they test
Mutex/Sync/Thread/Ruby and not your os virtual memory system and look closely
at the results again - like i said, i have had issues with sync.rb.

the point here is that it is probably the code in question and not Mutex per
se that was causing your process to grow in vmsize.

regards.

-a
-- 
to foster inner awareness, introspection, and reasoning is more efficient than
meditation and prayer.
- h.h. the 14th dalai lama

On Aug 26, 2006, at 5:12 AM, ara.t.howard@noaa.gov wrote:
> in any case, i''d carefully examine your tests (or the rails code
if
> that is
> indeed what it''s modeled after) to make sure that they test
> Mutex/Sync/Thread/Ruby and not your os virtual memory system and  
> look closely
> at the results again - like i said, i have had issues with sync.rb.
>
> the point here is that it is probably the code in question and not  
> Mutex per
> se that was causing your process to grow in vmsize.
>

I ran your test on OS/X looking at VSZ and RSS. And, like you,  
initially got Sync with no leak visible, and mutex with what looks  
like a bad leak. However, I notice that you only called GC once. I  
have a years old habit of always running GC at least three times when  
I really wanted GC to run (and in Java I had a loop that ran GC until  
it stopped freeing stuff which in some cases was eight or nine  
times). Superstition? Apparently not. On OS X, when I run GC three  
times neither sync nor mutex show a memory leak.

Zed, just for fun, try running GC a few times in a row (like  
GC.start; GC.start; GC.start) .

Cheers,
Bob

----
Bob Hutchison                  -- blogs at <http://www.recursive.ca/ 
hutch/>
Recursive Design Inc.          -- <http://www.recursive.ca/>
Raconteur                      -- <http://www.raconteur.info/>
xampl for Ruby                 -- <http://rubyforge.org/projects/xampl/>

On Aug 26, 2006, at 10:22 AM, Bob Hutchison wrote:
>
> On Aug 26, 2006, at 5:12 AM, ara.t.howard@noaa.gov wrote:
>
>> in any case, i''d carefully examine your tests (or the rails
code
>> if that is
>> indeed what it''s modeled after) to make sure that they test
>> Mutex/Sync/Thread/Ruby and not your os virtual memory system and  
>> look closely
>> at the results again - like i said, i have had issues with sync.rb.
>>
>> the point here is that it is probably the code in question and not  
>> Mutex per
>> se that was causing your process to grow in vmsize.
>>
>
>
> I ran your test on OS/X looking at VSZ and RSS. And, like you,  
> initially got Sync with no leak visible, and mutex with what looks  
> like a bad leak. However, I notice that you only called GC once. I  
> have a years old habit of always running GC at least three times  
> when I really wanted GC to run (and in Java I had a loop that ran  
> GC until it stopped freeing stuff which in some cases was eight or  
> nine times). Superstition? Apparently not. On OS X, when I run GC  
> three times neither sync nor mutex show a memory leak.
>
> Zed, just for fun, try running GC a few times in a row (like  
> GC.start; GC.start; GC.start)
Well I tried your test on OS X. The Sync had no problem, the mutex  
showed the memory growth (though it eventually (fifth iteration I  
think) cleaned itself up). I modified your test to create exactly  
1000 threads and call GC three times at the end, things were better,  
i.e. it released its memory more quickly than without, but still not  
good. I ended up with:

       GC.start
       `sync; sync; sync`
       sleep 1
       GC.start
       `sync; sync; sync`
       sleep 1
       GC.start
       `sync; sync; sync`
       sleep 1
       GC.start
       `sync; sync; sync`
       sleep 1

and this made a bigger difference. The memory usage was much more  
tightly bound.

(And yes, the three calls to sync are also on purpose... in the late  
70s through the 80s, calling sync once didn''t guarantee anything, you  
had to call it a few times, three generally worked... I don''t know  
the current situation because it is easy enough to type  
sync;sync;sync (well, in truth, I usually alias sync to the three  
calls))

But of course, the point is that despite appearances there is likely  
no memory leak at all on OS X, just some kind of long term cycle of  
process resource utilisation -- this is a complex situation, Ruby GC,  
process resource utilisation/optimisation, and system optimisation  
all interacting. Who knows what''s actually going on.

So.

Cheers,
Bob

----
Bob Hutchison                  -- blogs at <http://www.recursive.ca/ 
hutch/>
Recursive Design Inc.          -- <http://www.recursive.ca/>
Raconteur                      -- <http://www.raconteur.info/>
xampl for Ruby                 -- <http://rubyforge.org/projects/xampl/>