ara.t.howard-32lpuo7BZBA@public.gmane.org
2006-Aug-26 09:12 UTC
Re: [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
On Sat, 26 Aug 2006, Zed Shaw wrote:> Howdy Folks, > > This release is after painstaking analysis of a memory leak that was > reported by Bradley Taylor, reduced by myself, and then fixed after much > work. You should all thank Bradley for finding the bizarre fix. > > It turns out the Ruby has a memory leak when you use pretty much any > thread locking primitive other than Sync (Mutex, Monitor, etc.): > > http://pastie.caboo.se/10194 > > The fix (for whatever reason) is to use Sync and put it in a block: > > http://pastie.caboo.se/10317 > > Those two scripts are mini versions of how Mongrel manages threads so > that I could figure out a solution or get some input. The graph is > reported ram usage samples 1/second. As you can see the first leaking > graph goes up and doesn''t go down, the second (fixed) graph cycles > properly. > > ** This is a Ruby issue, so if you have software using Mutex or Monitor, > change to Sync now. ** > > Tests of this latest pre-release show that the RAM is properly cycled by > the GC and that it''s actually finally solved. If you run your app using > this release and you still have a leak then use the memory debugging > tools mongrel has to rule out your code (see below).hi zed- if you are really serious about fixing your leak i suggest you re-work your tests. as i mentioned before they have several race conditions, not least of which that they both start a random number of threads, not 1000 as the code suggests (you can easily confirm by printing out the number of times the thread init loop executes). further, sync.rb is the single ruby lib i''ve had memory issues with on production systems. i have never managed to figure out why that is... in any case a careful script which allocated memory in a thread, waits for all threads to finish allocation, checks memory, and then kills all threads before checking again shows some suprising results which you should read carefully: using mutex shows a nice cycle of memory freed: harp:~ > cat a.rb.mutex using: Mutex n: 420 iter: 0 with 420 threads holding memory : 44.0% with 0 threads holding memory : 13.0% iter: 1 with 420 threads holding memory : 43.9% with 0 threads holding memory : 13.0% iter: 2 with 420 threads holding memory : 44.1% with 0 threads holding memory : 13.3% iter: 3 with 420 threads holding memory : 44.1% with 0 threads holding memory : 13.2% iter: 4 with 420 threads holding memory : 44.0% with 0 threads holding memory : 13.5% iter: 5 with 420 threads holding memory : 44.1% with 0 threads holding memory : 13.2% iter: 6 with 420 threads holding memory : 43.9% with 0 threads holding memory : 13.2% iter: 7 with 420 threads holding memory : 44.2% with 0 threads holding memory : 13.2% iter: 8 with 420 threads holding memory : 44.1% with 0 threads holding memory : 13.5% iter: 9 with 420 threads holding memory : 44.1% with 0 threads holding memory : 13.9% using sync, on the other hand, looks leaky, though i''m not saying it is. harp:~ > cat a.rb.sync using: Sync n: 420 iter: 0 with 420 threads holding memory : 43.8% with 0 threads holding memory : 1.0% iter: 1 with 420 threads holding memory : 43.8% with 0 threads holding memory : 2.0% iter: 2 with 420 threads holding memory : 43.8% with 0 threads holding memory : 2.7% iter: 3 with 420 threads holding memory : 43.8% with 0 threads holding memory : 3.5% iter: 4 with 420 threads holding memory : 43.8% with 0 threads holding memory : 3.8% iter: 5 with 420 threads holding memory : 43.8% with 0 threads holding memory : 4.6% iter: 6 with 420 threads holding memory : 43.8% with 0 threads holding memory : 5.4% iter: 7 with 420 threads holding memory : 43.8% with 0 threads holding memory : 6.4% iter: 8 with 420 threads holding memory : 43.8% with 0 threads holding memory : 7.2% iter: 9 with 420 threads holding memory : 43.7% with 0 threads holding memory : 8.1% here is the code, note that it''s quite careful to only create a fixed number of threads, to wait for them to each init a mb of memory, and only then to check memory usage. likewise for checking after killing all threads - it''s done immediately after killing threads and running gc. here is the code: harp:~ > cat a.rb require ''thread'' require ''sync'' class TestThreads def initialize which, n c = case which when /mutex/io Mutex when /sync/io Sync end @guard = c.new @n = Integer n puts "using: #{ c.name }" puts "n: #{ @n }" end def pct_mem # linux specific field pos i''m sure stdout = `ps v #{ Process.pid }` stdout.split(%r/\n/).last.strip.split(%r/\s+/)[8] + ''%'' end def tq q = Queue.new t = Thread.new{ mb = @guard.synchronize{ 0.chr * (2 ** 20) } q.push :ready Thread.stop } [t, q] end def run list = [] 10.times do |i| puts "iter: #{ i }" # load 1000 threads up @n.times{ list << tq } # wait for all threads to init memory with mb of data list.each{|t,q| q.pop} # show memory usage GC.start puts "with #{ list.size } threads holding memory : #{ pct_mem }" # kill all threads - clean up list.each{|t,q| t.kill} list.clear sleep 1 until Thread.list.size == 1 # show memory usage GC.start puts "with 0 threads holding memory : #{ pct_mem }" end end end $VERBOSE = nil STDOUT.sync = true Thread.abort_on_exception = true trap(''INT''){ exit } which, n, ignored = ARGV TestThreads.new(which, n).run in any case, i''d carefully examine your tests (or the rails code if that is indeed what it''s modeled after) to make sure that they test Mutex/Sync/Thread/Ruby and not your os virtual memory system and look closely at the results again - like i said, i have had issues with sync.rb. the point here is that it is probably the code in question and not Mutex per se that was causing your process to grow in vmsize. regards. -a -- to foster inner awareness, introspection, and reasoning is more efficient than meditation and prayer. - h.h. the 14th dalai lama _______________________________________________ Rails mailing list Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org http://lists.rubyonrails.org/mailman/listinfo/rails
Bob Hutchison
2006-Aug-26 14:22 UTC
Re: [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
On Aug 26, 2006, at 5:12 AM, ara.t.howard-32lpuo7BZBA@public.gmane.org wrote:> in any case, i''d carefully examine your tests (or the rails code if > that is > indeed what it''s modeled after) to make sure that they test > Mutex/Sync/Thread/Ruby and not your os virtual memory system and > look closely > at the results again - like i said, i have had issues with sync.rb. > > the point here is that it is probably the code in question and not > Mutex per > se that was causing your process to grow in vmsize. >I ran your test on OS/X looking at VSZ and RSS. And, like you, initially got Sync with no leak visible, and mutex with what looks like a bad leak. However, I notice that you only called GC once. I have a years old habit of always running GC at least three times when I really wanted GC to run (and in Java I had a loop that ran GC until it stopped freeing stuff which in some cases was eight or nine times). Superstition? Apparently not. On OS X, when I run GC three times neither sync nor mutex show a memory leak. Zed, just for fun, try running GC a few times in a row (like GC.start; GC.start; GC.start) . Cheers, Bob ---- Bob Hutchison -- blogs at <http://www.recursive.ca/ hutch/> Recursive Design Inc. -- <http://www.recursive.ca/> Raconteur -- <http://www.raconteur.info/> xampl for Ruby -- <http://rubyforge.org/projects/xampl/> _______________________________________________ Rails mailing list Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org http://lists.rubyonrails.org/mailman/listinfo/rails
Bob Hutchison
2006-Aug-26 15:20 UTC
Re: [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
On Aug 26, 2006, at 10:22 AM, Bob Hutchison wrote:> > On Aug 26, 2006, at 5:12 AM, ara.t.howard-32lpuo7BZBA@public.gmane.org wrote: > >> in any case, i''d carefully examine your tests (or the rails code >> if that is >> indeed what it''s modeled after) to make sure that they test >> Mutex/Sync/Thread/Ruby and not your os virtual memory system and >> look closely >> at the results again - like i said, i have had issues with sync.rb. >> >> the point here is that it is probably the code in question and not >> Mutex per >> se that was causing your process to grow in vmsize. >> > > > I ran your test on OS/X looking at VSZ and RSS. And, like you, > initially got Sync with no leak visible, and mutex with what looks > like a bad leak. However, I notice that you only called GC once. I > have a years old habit of always running GC at least three times > when I really wanted GC to run (and in Java I had a loop that ran > GC until it stopped freeing stuff which in some cases was eight or > nine times). Superstition? Apparently not. On OS X, when I run GC > three times neither sync nor mutex show a memory leak. > > Zed, just for fun, try running GC a few times in a row (like > GC.start; GC.start; GC.start)Well I tried your test on OS X. The Sync had no problem, the mutex showed the memory growth (though it eventually (fifth iteration I think) cleaned itself up). I modified your test to create exactly 1000 threads and call GC three times at the end, things were better, i.e. it released its memory more quickly than without, but still not good. I ended up with: GC.start `sync; sync; sync` sleep 1 GC.start `sync; sync; sync` sleep 1 GC.start `sync; sync; sync` sleep 1 GC.start `sync; sync; sync` sleep 1 and this made a bigger difference. The memory usage was much more tightly bound. (And yes, the three calls to sync are also on purpose... in the late 70s through the 80s, calling sync once didn''t guarantee anything, you had to call it a few times, three generally worked... I don''t know the current situation because it is easy enough to type sync;sync;sync (well, in truth, I usually alias sync to the three calls)) But of course, the point is that despite appearances there is likely no memory leak at all on OS X, just some kind of long term cycle of process resource utilisation -- this is a complex situation, Ruby GC, process resource utilisation/optimisation, and system optimisation all interacting. Who knows what''s actually going on. So. Cheers, Bob ---- Bob Hutchison -- blogs at <http://www.recursive.ca/ hutch/> Recursive Design Inc. -- <http://www.recursive.ca/> Raconteur -- <http://www.raconteur.info/> xampl for Ruby -- <http://rubyforge.org/projects/xampl/> _______________________________________________ Rails mailing list Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org http://lists.rubyonrails.org/mailman/listinfo/rails
Kevin Williams
2006-Aug-26 17:12 UTC
[Mongrel] [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
Perhaps it''s just me, but I don''t like cross-posting. Please post to each mailing list separately, if you don''t mind. :) On 8/25/06, Zed Shaw <zedshaw at zedshaw.com> wrote:> Howdy Folks, > > This release is after painstaking analysis of a memory leak that was > reported by Bradley Taylor, reduced by myself, and then fixed after much > work. You should all thank Bradley for finding the bizarre fix. > > It turns out the Ruby has a memory leak when you use pretty much any > thread locking primitive other than Sync (Mutex, Monitor, etc.): > > http://pastie.caboo.se/10194 > > The fix (for whatever reason) is to use Sync and put it in a block: > > http://pastie.caboo.se/10317 > > Those two scripts are mini versions of how Mongrel manages threads so > that I could figure out a solution or get some input. The graph is > reported ram usage samples 1/second. As you can see the first leaking > graph goes up and doesn''t go down, the second (fixed) graph cycles > properly. > > ** This is a Ruby issue, so if you have software using Mutex or Monitor, > change to Sync now. ** > > Tests of this latest pre-release show that the RAM is properly cycled by > the GC and that it''s actually finally solved. If you run your app using > this release and you still have a leak then use the memory debugging > tools mongrel has to rule out your code (see below). > > > CHANGES > > * No more allow_concurrency. Until Ruby''s fixed I can''t let people do > this anymore. > * USR1 debugging. If you''re wondering about how Mongrel''s locking of > Rails impacts your application, or what is causing BAD CLIENT then just > hit your mongrel_rails with USR1 and Mongrel will tell you. > * More extensive and accurate memory debugging. Use -B and look at the > log/mongrel_log/objects.log to get a good idea of counts of objects, > delta changes in counts, and mean+standard deviation lengths of objects > with length methods. > * Fixes a few places where sockets are closed and left in CLOSE_WAIT. > > > INSTALLING > > As per usual: > > sudo gem install mongrel --source=http://mongrel.rubyforge.org/releases/ > > Initial tests show it works on 1.8.5 and is actually faster, but this is > unsupported for now. > > > TESTING THIS RELEASE > > If you want to test the memory leak, here''s the process: > > 1) Start your application in *production* mode: > mongrel_rails start -e production > > 2) Hit it with USR1: > killall -USR1 mongrel_rails > > 3) Start running something that prints out the ram (here''s my fish > code): > while sleep 1 > ps aux | grep mongrel_rails | grep -v grep | grep -v gvim | ruby -aln > -e "puts split[4 .. 5].join('','')" > end > > 4) Thrash a simple rails controller with httperf: > httperf --server 127.0.0.1 --port 3000 --num-conns 1000 --rate 120 > --uri /testuri > > What you want to do is adjust num-conns and rate until Mongrel reports > "X threads waiting for /testuri..." > > The bug only manifests itself when threads pile up behind the guard > around Rails dispatching. This is also how you''d find out which Rails > actions are too slow. > > > Please report any bugs you find in this release, and a Win32 release > will come out after I''m sure it works for everyone else. > > > -- > Zed A. Shaw > http://www.zedshaw.com/ > http://mongrel.rubyforge.org/ > http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help. > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-- Cheers, Kevin "Any sufficiently advanced technology is indistinguishable from Magic." - Arthur C. Clarke
Carl Lerche
2006-Aug-26 20:29 UTC
[Mongrel] [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
Zed, Thanks for the hard work, when I get home I''ll try running this one on my server and see how it holds! -carl On Aug 25, 2006, at 11:31 PM, Zed Shaw wrote:> Howdy Folks, > > This release is after painstaking analysis of a memory leak that was > reported by Bradley Taylor, reduced by myself, and then fixed after > much > work. You should all thank Bradley for finding the bizarre fix. > > It turns out the Ruby has a memory leak when you use pretty much any > thread locking primitive other than Sync (Mutex, Monitor, etc.): > > http://pastie.caboo.se/10194 > > The fix (for whatever reason) is to use Sync and put it in a block: > > http://pastie.caboo.se/10317 > > Those two scripts are mini versions of how Mongrel manages threads so > that I could figure out a solution or get some input. The graph is > reported ram usage samples 1/second. As you can see the first leaking > graph goes up and doesn''t go down, the second (fixed) graph cycles > properly. > > ** This is a Ruby issue, so if you have software using Mutex or > Monitor, > change to Sync now. ** > > Tests of this latest pre-release show that the RAM is properly > cycled by > the GC and that it''s actually finally solved. If you run your app > using > this release and you still have a leak then use the memory debugging > tools mongrel has to rule out your code (see below). > > > CHANGES > > * No more allow_concurrency. Until Ruby''s fixed I can''t let people do > this anymore. > * USR1 debugging. If you''re wondering about how Mongrel''s locking of > Rails impacts your application, or what is causing BAD CLIENT then > just > hit your mongrel_rails with USR1 and Mongrel will tell you. > * More extensive and accurate memory debugging. Use -B and look at > the > log/mongrel_log/objects.log to get a good idea of counts of objects, > delta changes in counts, and mean+standard deviation lengths of > objects > with length methods. > * Fixes a few places where sockets are closed and left in CLOSE_WAIT. > > > INSTALLING > > As per usual: > > sudo gem install mongrel --source=http://mongrel.rubyforge.org/ > releases/ > > Initial tests show it works on 1.8.5 and is actually faster, but > this is > unsupported for now. > > > TESTING THIS RELEASE > > If you want to test the memory leak, here''s the process: > > 1) Start your application in *production* mode: > mongrel_rails start -e production > > 2) Hit it with USR1: > killall -USR1 mongrel_rails > > 3) Start running something that prints out the ram (here''s my fish > code): > while sleep 1 > ps aux | grep mongrel_rails | grep -v grep | grep -v gvim | ruby > -aln > -e "puts split[4 .. 5].join('','')" > end > > 4) Thrash a simple rails controller with httperf: > httperf --server 127.0.0.1 --port 3000 --num-conns 1000 --rate 120 > --uri /testuri > > What you want to do is adjust num-conns and rate until Mongrel reports > "X threads waiting for /testuri..." > > The bug only manifests itself when threads pile up behind the guard > around Rails dispatching. This is also how you''d find out which Rails > actions are too slow. > > > Please report any bugs you find in this release, and a Win32 release > will come out after I''m sure it works for everyone else. > > > -- > Zed A. Shaw > http://www.zedshaw.com/ > http://mongrel.rubyforge.org/ > http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help. > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
Zed Shaw
2006-Aug-26 22:02 UTC
[Mongrel] [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
On Sat, 2006-08-26 at 11:20 -0400, Bob Hutchison wrote:> On Aug 26, 2006, at 10:22 AM, Bob Hutchison wrote: >> Well I tried your test on OS X. The Sync had no problem, the mutex > showed the memory growth (though it eventually (fifth iteration I > think) cleaned itself up). I modified your test to create exactly > 1000 threads and call GC three times at the end, things were better, > i.e. it released its memory more quickly than without, but still not > good. I ended up with:Thanks Bob, but I''ve gotta say this one more time, this test is not about 1000 threads. The test is about how *Mongrel* processes threads, a specific bug when many threads are put into a ThreadGroup and wait behind a Mutex, and how to stop that from leaking. If you change the way the test is written so that it creates exactly 1000 threads, then this isn''t simulating Mongrel. You''re most likely using additional synchronization primitives not used in Mongrel so your test is wrong. I mean, Mongrel doesn''t wait for 1000 threads, it just cranks on them and sometimes it''s too slow so you build a log jam. In this situation, we were seeing memory leaks. Other people also report the memory leaks and even reported this fixed it in other systems unrelated to Mongrel. Yes, you can write something else to not have memory leaks, but then you''re not testing our leaking situation. The point is that the script with Mutex leaks, the one with Sync doesn''t. The number 1000 was just a number picked at random so that I knew when to stop simulating requests and start calling GC. And, if you notice there''s a loop that calls GC.start about 3-4 times. There''s no way I''m going to put into Mongrel''s loop three calls to GC. -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
Zed Shaw
2006-Aug-26 22:04 UTC
[Mongrel] [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
On Sat, 2006-08-26 at 11:12 -0600, Kevin Williams wrote:> Perhaps it''s just me, but I don''t like cross-posting. Please post to > each mailing list separately, if you don''t mind. :)Kevin, how is it different if I put the mailing lists into one e-mail or three if the message is the same. Considering I don''t have tons of time and needed to get the word out so I can make the next release final, it seems stupid to spend my time on some kind of "vanity send". Is there something about your mail client that can''t handle this? -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
Kirk Haines
2006-Aug-26 22:36 UTC
[Mongrel] [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
On 8/26/06, Zed Shaw <zedshaw at zedshaw.com> wrote:> Thanks Bob, but I''ve gotta say this one more time, this test is not > about 1000 threads. The test is about how *Mongrel* processes threads, > a specific bug when many threads are put into a ThreadGroup and wait > behind a Mutex, and how to stop that from leaking. > > If you change the way the test is written so that it creates exactly > 1000 threads, then this isn''t simulating Mongrel. You''re most likely > using additional synchronization primitives not used in Mongrel so your > test is wrong. I mean, Mongrel doesn''t wait for 1000 threads, it just > cranks on them and sometimes it''s too slow so you build a log jam. > > In this situation, we were seeing memory leaks. Other people also > report the memory leaks and even reported this fixed it in other systems > unrelated to Mongrel. Yes, you can write something else to not have > memory leaks, but then you''re not testing our leaking situation. The > point is that the script with Mutex leaks, the one with Sync doesn''t.I''ve been testing with your pasted scripts and variations all day, and I can not reproduce any results that indicate that Mutex leaks. In your pasted script, the primary difference in behavior between Mutex and Sync (which, under the covers, use an identical algorithm for locking, though they differ in unlock semanticsi; mutex pops a single thread off the waiting list and wakes it while Sync wakes them all, letting one grab the other lock and the others go back to waiting) is that Sync is slower. All that I have to do in order to eliminate the phantom memory leaking by Mutex is to insert a very small delay at the end of each synchronized block. On my test system, select(nil,nil,nil,0.025) does the trick. In your test scripts, this causes the Mutex variant to launch fewer total threads, similar to the Sync variant (on my box, an iteration with the Mutex variant as you pasted it actually ends up creating about 2100 threads, while the Sync variant is around 1650, because Sync is slower so it takes longer for threads to fall out of the threadgroup as you are adding new ones in). On variations that launch exactly 1000 threads (which is easily done without using any other locking primitives), the difference boils down to how fast objects can be created and how long the GC has to clean them up. Change the test() method to do something that creates some strings and other objects, and it becomes clear very quickly that if there is a burst of activity, a bunch of threads locking with a Mutex outrun the GC''s ability to clean it up. Memory consumption rises. This also seems consistent on my tests so far comparing 1.8.4 to 1.8.5, which you mentioned seemed to exhibit worse RAM use characteristics. I still need to dig into the differences in the GC subsystem code between the two versions, but the experimental evidence that I have suggests that in 1.8.5 it is taking longer to get around to cleaning up objects. It seems to be faster when it does, as my overall throughput is about 10% faster on 1.8.5, but I don''t think I''m liking the tradeoff that I am seeing with memory consumption when it is pounded with objects. Something looks wrong, there, but it''s not related to Mutex. Kirk Haines
Luis Lavena
2006-Aug-26 23:26 UTC
[Mongrel] [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
I have done some testings of the code on Win32 with ruby 1.8.4, please check the included pictures [1]. I have compared both test scripts, using Process Explorer to monitor their memory use and CPU use. So far, Win32 behave way different than *nix in this matter. If you see the pngs, using mutex actually releases memory (GC) and finish faster than sync. Sync, on the contrary, never released memory (like GC wasn''t called at all) and took longer to finish (I stopped it because took more than 6 minutes). I could conclude that ruby implementations are too different to compare between installations and platforms... A solution for the memory leaking should be tested on any platform before taking final decisions. On 8/26/06, Kirk Haines <wyhaines at gmail.com> wrote: [snip]> > On variations that launch exactly 1000 threads (which is easily done > without using any other locking primitives), the difference boils down > to how fast objects can be created and how long the GC has to clean > them up. Change the test() method to do something that creates some > strings and other objects, and it becomes clear very quickly that if > there is a burst of activity, a bunch of threads locking with a Mutex > outrun the GC''s ability to clean it up. Memory consumption rises.Please provide a sample script (pastie?) and I will test it too on Win32.> This also seems consistent on my tests so far comparing 1.8.4 to > 1.8.5, which you mentioned seemed to exhibit worse RAM use > characteristics. I still need to dig into the differences in the GC > subsystem code between the two versions, but the experimental evidence > that I have suggests that in 1.8.5 it is taking longer to get around > to cleaning up objects. It seems to be faster when it does, as my > overall throughput is about 10% faster on 1.8.5, but I don''t think I''m > liking the tradeoff that I am seeing with memory consumption when it > is pounded with objects. Something looks wrong, there, but it''s not > related to Mutex.I agree with you that something is wrong, not with mutex, but with ruby itself. People will start yelling me about this, but is true. Not just with mutex, but with threads (oh lovely select()) and sockets. On every platform exceptions are thrown differently or in some cases, not raised at all! We must investigate this to find a workaround that works "almost good" on every platform. -- Luis Lavena Multimedia systems - Leaders are made, they are not born. They are made by hard effort, which is the price which all of us must pay to achieve any goal that is worthwhile. Vince Lombardi [1] Cannot add pngs to this mail (40K limit) so uploaded them to my server: http://www.mmediasys.com/mongrel/mutex.png http://www.mmediasys.com/mongrel/sync-1.png http://www.mmediasys.com/mongrel/sync-2.png
Kevin Williams
2006-Aug-26 23:38 UTC
[Mongrel] [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
On 8/26/06, Zed Shaw <zedshaw at zedshaw.com> wrote:> On Sat, 2006-08-26 at 11:12 -0600, Kevin Williams wrote: > > Perhaps it''s just me, but I don''t like cross-posting. Please post to > > each mailing list separately, if you don''t mind. :) > > Kevin, how is it different if I put the mailing lists into one e-mail or > three if the message is the same. Considering I don''t have tons of time > and needed to get the word out so I can make the next release final, it > seems stupid to spend my time on some kind of "vanity send". > > Is there something about your mail client that can''t handle this? >GMail gets the labels all crossed up. Sorry to bother you. -- Cheers, Kevin "Any sufficiently advanced technology is indistinguishable from Magic." - Arthur C. Clarke
Zed Shaw
2006-Aug-27 05:20 UTC
[Mongrel] [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
On Sat, 2006-08-26 at 20:26 -0300, Luis Lavena wrote:> I have done some testings of the code on Win32 with ruby 1.8.4, please > check the included pictures [1]. >Thank you Luis, *this* is what I need. A test with Mongrel code and evidence that it''s not fixed on a particular platform. Thank you for not guessing at random theories about why it works this way, and instead doing the test. It''s also incredibly sad since it means that one way works on POSIX and the other works on Win32. I''ll take a look at this tomorrow and see what happens. -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
Luis Lavena
2006-Aug-27 09:56 UTC
[Mongrel] [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
On 8/27/06, Zed Shaw <zedshaw at zedshaw.com> wrote:> On Sat, 2006-08-26 at 20:26 -0300, Luis Lavena wrote: > > I have done some testings of the code on Win32 with ruby 1.8.4, please > > check the included pictures [1]. > > > > Thank you Luis, *this* is what I need. A test with Mongrel code and > evidence that it''s not fixed on a particular platform. Thank you for > not guessing at random theories about why it works this way, and instead > doing the test. > > It''s also incredibly sad since it means that one way works on POSIX and > the other works on Win32. >Yeah, I was shocked when I saw that.> I''ll take a look at this tomorrow and see what happens.What about a new approach to RailsHandler? I mean: If just 1 thread can access Rails dispatcher, what if we serialize/enqueue the requests? Maybe will be too complicated, but MonitorMixin maybe could help: http://www.ruby-doc.org/stdlib/libdoc/monitor/rdoc/classes/MonitorMixin.html Anyway, talk this sunday about mongrel_service Regards, -- Luis Lavena Multimedia systems - Leaders are made, they are not born. They are made by hard effort, which is the price which all of us must pay to achieve any goal that is worthwhile. Vince Lombardi
Bob Hutchison
2006-Aug-27 14:16 UTC
[Mongrel] [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
On Aug 26, 2006, at 6:02 PM, Zed Shaw wrote:> On Sat, 2006-08-26 at 11:20 -0400, Bob Hutchison wrote: >> On Aug 26, 2006, at 10:22 AM, Bob Hutchison wrote: >> > >> Well I tried your test on OS X. The Sync had no problem, the mutex >> showed the memory growth (though it eventually (fifth iteration I >> think) cleaned itself up). I modified your test to create exactly >> 1000 threads and call GC three times at the end, things were better, >> i.e. it released its memory more quickly than without, but still not >> good. I ended up with: > > Thanks Bob, but I''ve gotta say this one more time, this test is not > about 1000 threads. The test is about how *Mongrel* processes > threads, > a specific bug when many threads are put into a ThreadGroup and wait > behind a Mutex, and how to stop that from leaking.I know it isn''t about 1000 threads, but when testing it *really* helps sometimes to lock as much down as possible. I was getting anywhere from 1005 through 1040 threads and that variation was too much. I like having exactly *one* thing vary between tests. No matter. I posted on the ruby-talk list the result of some mucking about I was doing this morning. It seems that on OS X at least, the Mutex is holding threads in memory. If you delete the reference to the Mutex and then garbage collect you get a nice stable memory utilisation. I have no idea if you can make use of this information. Is there some point at which you know you can clobber the Mutex in Mongrel? Cheers, Bob ---- Bob Hutchison -- blogs at <http://www.recursive.ca/ hutch/> Recursive Design Inc. -- <http://www.recursive.ca/> Raconteur -- <http://www.raconteur.info/> xampl for Ruby -- <http://rubyforge.org/projects/xampl/>
Pedro CĂ´rte-Real
2006-Aug-27 14:45 UTC
[Mongrel] [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
On 8/27/06, Kevin Williams <kevwil at gmail.com> wrote:> GMail gets the labels all crossed up.I use gmail and the labels worked fine. The message was labeled with rails, mongrel and ruby-talk labels. It shows only one message although the mail server must have received three copies. One from each list. Pedro.
Kevin Williams
2006-Aug-27 21:15 UTC
[Mongrel] [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
My mongrel filter is this: Matches: to:(mongrel-users at rubyforge.org) Do this: Skip Inbox, Apply label "mongrel" I''m have unsubscribed from the rails list anyway, so it won''t be an issue anymore. I''ll just use ruby-forum to keep track of Rails stuff. On 8/27/06, Pedro C?rte-Real <pedro at pedrocr.net> wrote:> On 8/27/06, Kevin Williams <kevwil at gmail.com> wrote: > > GMail gets the labels all crossed up. > > I use gmail and the labels worked fine. The message was labeled with > rails, mongrel and ruby-talk labels. It shows only one message > although the mail server must have received three copies. One from > each list. > > Pedro. > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-- Cheers, Kevin "Any sufficiently advanced technology is indistinguishable from Magic." - Arthur C. Clarke
Zed Shaw
2006-Nov-04 01:56 UTC
[Rails] [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
Howdy Folks, This release is after painstaking analysis of a memory leak that was reported by Bradley Taylor, reduced by myself, and then fixed after much work. You should all thank Bradley for finding the bizarre fix. It turns out the Ruby has a memory leak when you use pretty much any thread locking primitive other than Sync (Mutex, Monitor, etc.): http://pastie.caboo.se/10194 The fix (for whatever reason) is to use Sync and put it in a block: http://pastie.caboo.se/10317 Those two scripts are mini versions of how Mongrel manages threads so that I could figure out a solution or get some input. The graph is reported ram usage samples 1/second. As you can see the first leaking graph goes up and doesn''t go down, the second (fixed) graph cycles properly. ** This is a Ruby issue, so if you have software using Mutex or Monitor, change to Sync now. ** Tests of this latest pre-release show that the RAM is properly cycled by the GC and that it''s actually finally solved. If you run your app using this release and you still have a leak then use the memory debugging tools mongrel has to rule out your code (see below). CHANGES * No more allow_concurrency. Until Ruby''s fixed I can''t let people do this anymore. * USR1 debugging. If you''re wondering about how Mongrel''s locking of Rails impacts your application, or what is causing BAD CLIENT then just hit your mongrel_rails with USR1 and Mongrel will tell you. * More extensive and accurate memory debugging. Use -B and look at the log/mongrel_log/objects.log to get a good idea of counts of objects, delta changes in counts, and mean+standard deviation lengths of objects with length methods. * Fixes a few places where sockets are closed and left in CLOSE_WAIT. INSTALLING As per usual: sudo gem install mongrel --source=http://mongrel.rubyforge.org/releases/ Initial tests show it works on 1.8.5 and is actually faster, but this is unsupported for now. TESTING THIS RELEASE If you want to test the memory leak, here''s the process: 1) Start your application in *production* mode: mongrel_rails start -e production 2) Hit it with USR1: killall -USR1 mongrel_rails 3) Start running something that prints out the ram (here''s my fish code): while sleep 1 ps aux | grep mongrel_rails | grep -v grep | grep -v gvim | ruby -aln -e "puts split[4 .. 5].join('','')" end 4) Thrash a simple rails controller with httperf: httperf --server 127.0.0.1 --port 3000 --num-conns 1000 --rate 120 --uri /testuri What you want to do is adjust num-conns and rate until Mongrel reports "X threads waiting for /testuri..." The bug only manifests itself when threads pile up behind the guard around Rails dispatching. This is also how you''d find out which Rails actions are too slow. Please report any bugs you find in this release, and a Win32 release will come out after I''m sure it works for everyone else. -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
ara.t.howard@noaa.gov
2006-Nov-04 01:57 UTC
[Rails] Re: [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
On Sat, 26 Aug 2006, Zed Shaw wrote:> Howdy Folks, > > This release is after painstaking analysis of a memory leak that was > reported by Bradley Taylor, reduced by myself, and then fixed after much > work. You should all thank Bradley for finding the bizarre fix. > > It turns out the Ruby has a memory leak when you use pretty much any > thread locking primitive other than Sync (Mutex, Monitor, etc.): > > http://pastie.caboo.se/10194 > > The fix (for whatever reason) is to use Sync and put it in a block: > > http://pastie.caboo.se/10317 > > Those two scripts are mini versions of how Mongrel manages threads so > that I could figure out a solution or get some input. The graph is > reported ram usage samples 1/second. As you can see the first leaking > graph goes up and doesn''t go down, the second (fixed) graph cycles > properly. > > ** This is a Ruby issue, so if you have software using Mutex or Monitor, > change to Sync now. ** > > Tests of this latest pre-release show that the RAM is properly cycled by > the GC and that it''s actually finally solved. If you run your app using > this release and you still have a leak then use the memory debugging > tools mongrel has to rule out your code (see below).hi zed- if you are really serious about fixing your leak i suggest you re-work your tests. as i mentioned before they have several race conditions, not least of which that they both start a random number of threads, not 1000 as the code suggests (you can easily confirm by printing out the number of times the thread init loop executes). further, sync.rb is the single ruby lib i''ve had memory issues with on production systems. i have never managed to figure out why that is... in any case a careful script which allocated memory in a thread, waits for all threads to finish allocation, checks memory, and then kills all threads before checking again shows some suprising results which you should read carefully: using mutex shows a nice cycle of memory freed: harp:~ > cat a.rb.mutex using: Mutex n: 420 iter: 0 with 420 threads holding memory : 44.0% with 0 threads holding memory : 13.0% iter: 1 with 420 threads holding memory : 43.9% with 0 threads holding memory : 13.0% iter: 2 with 420 threads holding memory : 44.1% with 0 threads holding memory : 13.3% iter: 3 with 420 threads holding memory : 44.1% with 0 threads holding memory : 13.2% iter: 4 with 420 threads holding memory : 44.0% with 0 threads holding memory : 13.5% iter: 5 with 420 threads holding memory : 44.1% with 0 threads holding memory : 13.2% iter: 6 with 420 threads holding memory : 43.9% with 0 threads holding memory : 13.2% iter: 7 with 420 threads holding memory : 44.2% with 0 threads holding memory : 13.2% iter: 8 with 420 threads holding memory : 44.1% with 0 threads holding memory : 13.5% iter: 9 with 420 threads holding memory : 44.1% with 0 threads holding memory : 13.9% using sync, on the other hand, looks leaky, though i''m not saying it is. harp:~ > cat a.rb.sync using: Sync n: 420 iter: 0 with 420 threads holding memory : 43.8% with 0 threads holding memory : 1.0% iter: 1 with 420 threads holding memory : 43.8% with 0 threads holding memory : 2.0% iter: 2 with 420 threads holding memory : 43.8% with 0 threads holding memory : 2.7% iter: 3 with 420 threads holding memory : 43.8% with 0 threads holding memory : 3.5% iter: 4 with 420 threads holding memory : 43.8% with 0 threads holding memory : 3.8% iter: 5 with 420 threads holding memory : 43.8% with 0 threads holding memory : 4.6% iter: 6 with 420 threads holding memory : 43.8% with 0 threads holding memory : 5.4% iter: 7 with 420 threads holding memory : 43.8% with 0 threads holding memory : 6.4% iter: 8 with 420 threads holding memory : 43.8% with 0 threads holding memory : 7.2% iter: 9 with 420 threads holding memory : 43.7% with 0 threads holding memory : 8.1% here is the code, note that it''s quite careful to only create a fixed number of threads, to wait for them to each init a mb of memory, and only then to check memory usage. likewise for checking after killing all threads - it''s done immediately after killing threads and running gc. here is the code: harp:~ > cat a.rb require ''thread'' require ''sync'' class TestThreads def initialize which, n c = case which when /mutex/io Mutex when /sync/io Sync end @guard = c.new @n = Integer n puts "using: #{ c.name }" puts "n: #{ @n }" end def pct_mem # linux specific field pos i''m sure stdout = `ps v #{ Process.pid }` stdout.split(%r/\n/).last.strip.split(%r/\s+/)[8] + ''%'' end def tq q = Queue.new t = Thread.new{ mb = @guard.synchronize{ 0.chr * (2 ** 20) } q.push :ready Thread.stop } [t, q] end def run list = [] 10.times do |i| puts "iter: #{ i }" # load 1000 threads up @n.times{ list << tq } # wait for all threads to init memory with mb of data list.each{|t,q| q.pop} # show memory usage GC.start puts "with #{ list.size } threads holding memory : #{ pct_mem }" # kill all threads - clean up list.each{|t,q| t.kill} list.clear sleep 1 until Thread.list.size == 1 # show memory usage GC.start puts "with 0 threads holding memory : #{ pct_mem }" end end end $VERBOSE = nil STDOUT.sync = true Thread.abort_on_exception = true trap(''INT''){ exit } which, n, ignored = ARGV TestThreads.new(which, n).run in any case, i''d carefully examine your tests (or the rails code if that is indeed what it''s modeled after) to make sure that they test Mutex/Sync/Thread/Ruby and not your os virtual memory system and look closely at the results again - like i said, i have had issues with sync.rb. the point here is that it is probably the code in question and not Mutex per se that was causing your process to grow in vmsize. regards. -a -- to foster inner awareness, introspection, and reasoning is more efficient than meditation and prayer. - h.h. the 14th dalai lama
Ara.T.Howard
2006-Nov-04 01:57 UTC
[Rails] Re: [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
On Sat, 26 Aug 2006, Zed Shaw wrote:> Howdy Folks, > > This release is after painstaking analysis of a memory leak that was > reported by Bradley Taylor, reduced by myself, and then fixed after much > work. You should all thank Bradley for finding the bizarre fix. > > It turns out the Ruby has a memory leak when you use pretty much any > thread locking primitive other than Sync (Mutex, Monitor, etc.): > > http://pastie.caboo.se/10194 > > The fix (for whatever reason) is to use Sync and put it in a block: > > http://pastie.caboo.se/10317 > > Those two scripts are mini versions of how Mongrel manages threads so > that I could figure out a solution or get some input. The graph is > reported ram usage samples 1/second. As you can see the first leaking > graph goes up and doesn''t go down, the second (fixed) graph cycles > properly. > > ** This is a Ruby issue, so if you have software using Mutex or Monitor, > change to Sync now. ** > > Tests of this latest pre-release show that the RAM is properly cycled by > the GC and that it''s actually finally solved. If you run your app using > this release and you still have a leak then use the memory debugging > tools mongrel has to rule out your code (see below).hi zed- if you are really serious about fixing your leak i suggest you re-work your tests. as i mentioned before they have several race conditions, not least of which that they both start a random number of threads, not 1000 as the code suggests (you can easily confirm by printing out the number of times the thread init loop executes). further, sync.rb is the single ruby lib i''ve had memory issues with on production systems. i have never managed to figure out why that is... in any case a careful script which allocated memory in a thread, waits for all threads to finish allocation, checks memory, and then kills all threads before checking again shows some suprising results which you should read carefully: using mutex shows a nice cycle of memory freed: harp:~ > cat a.rb.mutex using: Mutex n: 420 iter: 0 with 420 threads holding memory : 44.0% with 0 threads holding memory : 13.0% iter: 1 with 420 threads holding memory : 43.9% with 0 threads holding memory : 13.0% iter: 2 with 420 threads holding memory : 44.1% with 0 threads holding memory : 13.3% iter: 3 with 420 threads holding memory : 44.1% with 0 threads holding memory : 13.2% iter: 4 with 420 threads holding memory : 44.0% with 0 threads holding memory : 13.5% iter: 5 with 420 threads holding memory : 44.1% with 0 threads holding memory : 13.2% iter: 6 with 420 threads holding memory : 43.9% with 0 threads holding memory : 13.2% iter: 7 with 420 threads holding memory : 44.2% with 0 threads holding memory : 13.2% iter: 8 with 420 threads holding memory : 44.1% with 0 threads holding memory : 13.5% iter: 9 with 420 threads holding memory : 44.1% with 0 threads holding memory : 13.9% using sync, on the other hand, looks leaky, though i''m not saying it is. harp:~ > cat a.rb.sync using: Sync n: 420 iter: 0 with 420 threads holding memory : 43.8% with 0 threads holding memory : 1.0% iter: 1 with 420 threads holding memory : 43.8% with 0 threads holding memory : 2.0% iter: 2 with 420 threads holding memory : 43.8% with 0 threads holding memory : 2.7% iter: 3 with 420 threads holding memory : 43.8% with 0 threads holding memory : 3.5% iter: 4 with 420 threads holding memory : 43.8% with 0 threads holding memory : 3.8% iter: 5 with 420 threads holding memory : 43.8% with 0 threads holding memory : 4.6% iter: 6 with 420 threads holding memory : 43.8% with 0 threads holding memory : 5.4% iter: 7 with 420 threads holding memory : 43.8% with 0 threads holding memory : 6.4% iter: 8 with 420 threads holding memory : 43.8% with 0 threads holding memory : 7.2% iter: 9 with 420 threads holding memory : 43.7% with 0 threads holding memory : 8.1% here is the code, note that it''s quite careful to only create a fixed number of threads, to wait for them to each init a mb of memory, and only then to check memory usage. likewise for checking after killing all threads - it''s done immediately after killing threads and running gc. here is the code: harp:~ > cat a.rb require ''thread'' require ''sync'' class TestThreads def initialize which, n c = case which when /mutex/io Mutex when /sync/io Sync end @guard = c.new @n = Integer n puts "using: #{ c.name }" puts "n: #{ @n }" end def pct_mem # linux specific field pos i''m sure stdout = `ps v #{ Process.pid }` stdout.split(%r/\n/).last.strip.split(%r/\s+/)[8] + ''%'' end def tq q = Queue.new t = Thread.new{ mb = @guard.synchronize{ 0.chr * (2 ** 20) } q.push :ready Thread.stop } [t, q] end def run list = [] 10.times do |i| puts "iter: #{ i }" # load 1000 threads up @n.times{ list << tq } # wait for all threads to init memory with mb of data list.each{|t,q| q.pop} # show memory usage GC.start puts "with #{ list.size } threads holding memory : #{ pct_mem }" # kill all threads - clean up list.each{|t,q| t.kill} list.clear sleep 1 until Thread.list.size == 1 # show memory usage GC.start puts "with 0 threads holding memory : #{ pct_mem }" end end end $VERBOSE = nil STDOUT.sync = true Thread.abort_on_exception = true trap(''INT''){ exit } which, n, ignored = ARGV TestThreads.new(which, n).run in any case, i''d carefully examine your tests (or the rails code if that is indeed what it''s modeled after) to make sure that they test Mutex/Sync/Thread/Ruby and not your os virtual memory system and look closely at the results again - like i said, i have had issues with sync.rb. the point here is that it is probably the code in question and not Mutex per se that was causing your process to grow in vmsize. regards. -a -- to foster inner awareness, introspection, and reasoning is more efficient than meditation and prayer. - h.h. the 14th dalai lama
Bob Hutchison
2006-Nov-04 01:58 UTC
[Rails] Re: [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
On Aug 26, 2006, at 5:12 AM, ara.t.howard@noaa.gov wrote:> in any case, i''d carefully examine your tests (or the rails code if > that is > indeed what it''s modeled after) to make sure that they test > Mutex/Sync/Thread/Ruby and not your os virtual memory system and > look closely > at the results again - like i said, i have had issues with sync.rb. > > the point here is that it is probably the code in question and not > Mutex per > se that was causing your process to grow in vmsize. >I ran your test on OS/X looking at VSZ and RSS. And, like you, initially got Sync with no leak visible, and mutex with what looks like a bad leak. However, I notice that you only called GC once. I have a years old habit of always running GC at least three times when I really wanted GC to run (and in Java I had a loop that ran GC until it stopped freeing stuff which in some cases was eight or nine times). Superstition? Apparently not. On OS X, when I run GC three times neither sync nor mutex show a memory leak. Zed, just for fun, try running GC a few times in a row (like GC.start; GC.start; GC.start) . Cheers, Bob ---- Bob Hutchison -- blogs at <http://www.recursive.ca/ hutch/> Recursive Design Inc. -- <http://www.recursive.ca/> Raconteur -- <http://www.raconteur.info/> xampl for Ruby -- <http://rubyforge.org/projects/xampl/>
Bob Hutchison
2006-Nov-04 01:59 UTC
[Rails] Re: [ANN] Mongrel 0.3.13.4 Pre-Release -- Ruby''s LEAK Fixed (Death To Mutex!)
On Aug 26, 2006, at 10:22 AM, Bob Hutchison wrote:> > On Aug 26, 2006, at 5:12 AM, ara.t.howard@noaa.gov wrote: > >> in any case, i''d carefully examine your tests (or the rails code >> if that is >> indeed what it''s modeled after) to make sure that they test >> Mutex/Sync/Thread/Ruby and not your os virtual memory system and >> look closely >> at the results again - like i said, i have had issues with sync.rb. >> >> the point here is that it is probably the code in question and not >> Mutex per >> se that was causing your process to grow in vmsize. >> > > > I ran your test on OS/X looking at VSZ and RSS. And, like you, > initially got Sync with no leak visible, and mutex with what looks > like a bad leak. However, I notice that you only called GC once. I > have a years old habit of always running GC at least three times > when I really wanted GC to run (and in Java I had a loop that ran > GC until it stopped freeing stuff which in some cases was eight or > nine times). Superstition? Apparently not. On OS X, when I run GC > three times neither sync nor mutex show a memory leak. > > Zed, just for fun, try running GC a few times in a row (like > GC.start; GC.start; GC.start)Well I tried your test on OS X. The Sync had no problem, the mutex showed the memory growth (though it eventually (fifth iteration I think) cleaned itself up). I modified your test to create exactly 1000 threads and call GC three times at the end, things were better, i.e. it released its memory more quickly than without, but still not good. I ended up with: GC.start `sync; sync; sync` sleep 1 GC.start `sync; sync; sync` sleep 1 GC.start `sync; sync; sync` sleep 1 GC.start `sync; sync; sync` sleep 1 and this made a bigger difference. The memory usage was much more tightly bound. (And yes, the three calls to sync are also on purpose... in the late 70s through the 80s, calling sync once didn''t guarantee anything, you had to call it a few times, three generally worked... I don''t know the current situation because it is easy enough to type sync;sync;sync (well, in truth, I usually alias sync to the three calls)) But of course, the point is that despite appearances there is likely no memory leak at all on OS X, just some kind of long term cycle of process resource utilisation -- this is a complex situation, Ruby GC, process resource utilisation/optimisation, and system optimisation all interacting. Who knows what''s actually going on. So. Cheers, Bob ---- Bob Hutchison -- blogs at <http://www.recursive.ca/ hutch/> Recursive Design Inc. -- <http://www.recursive.ca/> Raconteur -- <http://www.raconteur.info/> xampl for Ruby -- <http://rubyforge.org/projects/xampl/>