Eventmachine talk - Oct 2006 - Memory use and GC

I know this really isn''t an EM question per say, although it is an EM
based app.  I spent some time last night with sending 10mb and 20mb
files through stompserver and ran into some garbage collection issues
I can''t figure out.  I put in an event time that calls GC.start every
5 seconds and started sending/consuming messages.  Most of my tests
were 10 10mb/20mb files at a time, sending then consuming them.  The
pattern I kept seeing is that for the first 2-3 cycles memory use
would go up to around 100mb, then when GC.start was called it would go
back down to 8mb.  But on the 4th or 5th cycle things changed.  The
baseline memory was then at around 110mb and would climb 50mb or so
each cycle.  This was with the file based queue and all file IO is
done in blocks.  The parser is instantiated as a local variable for
each frame that is processed.

So two questions.  How to find what objects are not being GC''d, and
does this pattern give any signs as to where to look?

Chris

On 10/25/06, snacktime <snacktime at gmail.com>
wrote:
>
> I know this really isn''t an EM question per say, although it is an
EM
> based app.  I spent some time last night with sending 10mb and 20mb
> files through stompserver and ran into some garbage collection issues
> I can''t figure out.  I put in an event time that calls GC.start
every
> 5 seconds and started sending/consuming messages.  Most of my tests
> were 10 10mb/20mb files at a time, sending then consuming them.  The
> pattern I kept seeing is that for the first 2-3 cycles memory use
> would go up to around 100mb, then when GC.start was called it would go
> back down to 8mb.  But on the 4th or 5th cycle things changed.  The
> baseline memory was then at around 110mb and would climb 50mb or so
> each cycle.  This was with the file based queue and all file IO is
> done in blocks.  The parser is instantiated as a local variable for
> each frame that is processed.
>
> So two questions.  How to find what objects are not being GC''d,
and
> does this pattern give any signs as to where to look?


You might start by looking at ObjectSpace and its related APIs. Ruby''s
memory manager has something in it that changes the allocation pattern after
a short time, just as you describe. I''ve read ruby-core posts about it
but I
can''t remember exactly what it is. Possibly it starts allocating memory
in
larger chunks when it senses pressure from a lot of object-creations.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20061025/d3a80866/attachment.html

> You might start by looking at ObjectSpace and its related APIs.
Ruby''s
> memory manager has something in it that changes the allocation pattern
after
> a short time, just as you describe. I''ve read ruby-core posts
about it but I
> can''t remember exactly what it is. Possibly it starts allocating
memory in
> larger chunks when it senses pressure from a lot of object-creations.
That was actually my first thought when I saw the pattern, but I
couldn''t find anything to back it up.