This an open plea for someone with knowledge of RubyGems to explain why RubyGems stays in memory (over 10MB) after a Rails application is loaded. I am unable to figure out why this would be necessary. I haven''t had any response from Jim or Chad from RubyGems about it so I thought I would throw it open to the list to see if anyone could educate me or share thier experiences with this. There is a more detailed write up on my blog ( http://www.stevelongdo.com/articles/2006/08/08/rubygems-dependency-management-and-memory-use). -- Thanks, -Steve http://www.stevelongdo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://wrath.rubyonrails.org/pipermail/rails/attachments/20060808/51551e0a/attachment-0001.html
Steve Longdo
2006-Aug-08 23:59 UTC
[Rails] Re: RubyGems runtime memory use in Rails apps...
Anyone have any experience with Rails and gemconfigure? http://www.rubygems.org/read/chapter/4#page73 On 8/8/06, Steve Longdo <steve.longdo@gmail.com> wrote:> > This an open plea for someone with knowledge of RubyGems to explain why > RubyGems stays in memory (over 10MB) after a Rails application is loaded. I > am unable to figure out why this would be necessary. I haven''t had any > response from Jim or Chad from RubyGems about it so I thought I would throw > it open to the list to see if anyone could educate me or share thier > experiences with this. > > There is a more detailed write up on my blog ( http://www.stevelongdo.com/articles/2006/08/08/rubygems-dependency-management-and-memory-use > ). > > -- > Thanks, > -Steve > http://www.stevelongdo.com >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://wrath.rubyonrails.org/pipermail/rails/attachments/20060808/1dbe5701/attachment-0001.html
Hello Steve,
Well, since you asked for help, can I suggest that you provide
pertinent details ? OS used, ruby version, and also the example code
that you used to profile would be nice ;) I must say, that rubygems
hooks into the main process from what I can (hence the require) so I
would -expect- it to stay resident in memory. Which reminds me, what
memory are you profiling here ? VSZ ? RSZ ? Then again, in the day and
age of 1GB in my desktop machine, its probably highly easy to dismiss a
RSZ of 10MB ;) Heck, almost all of the servers at work have >4GB. Thats
not the point, I know, but, there is no point in chasing rabbits down
holes if you don''t need to (also called the ''premature
optimisation''
problem).
Details details, the devil is in the details*
Regards
Stef
(* this should not be taken as an endorsement of satan nor that I have
-the- answer to your problems ;)
Steve Longdo wrote:> This an open plea for someone with knowledge of RubyGems to explain
> why RubyGems stays in memory (over 10MB) after a Rails application is
> loaded. I am unable to figure out why this would be necessary. I
> haven''t had any response from Jim or Chad from RubyGems about it
so I
> thought I would throw it open to the list to see if anyone could
> educate me or share thier experiences with this.
>
> There is a more detailed write up on my blog (
>
http://www.stevelongdo.com/articles/2006/08/08/rubygems-dependency-management-and-memory-use
>
<http://www.stevelongdo.com/articles/2006/08/08/rubygems-dependency-management-and-memory-use>
> ).
>
> --
> Thanks,
> -Steve
> http://www.stevelongdo.com
> ------------------------------------------------------------------------
>
> _______________________________________________
> Rails mailing list
> Rails@lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
>
Stef!!!! Thank you for answering at all! I am a leper on the list it feels like sometimes. I am profiling Typo, an app in need of some very "Mature Optimization". So no worries there. Locally I use OS X/PPC, Ruby 1.8.4. However I am doing the real profiling against my www.stevelongdo.com site on TextDrive. It is also Ruby 1.8.4 and I think a Linux flavor not Solaris 10 yet anyway. Typo and its external dependant version of Rails were trunk a couple days ago. TextDrive limits are based on RSZ going over 48MB. VSZ is almost always lower than that when Ruby decides to allocate some more space and send RSZ over the limit. I am not sure whether they are using Stefan Kaes suggested patched GC or if it is plain vanilla Ruby GC. I don''t for sure how to find that out on TextDrive. At any rate there is no good reason for my personal Typo weblog to cross into the 48MB zone. I''ll bet all of my content together is 512kB. Outside of all that though I am trying to learn more about how RubyGems actually does its business and whether it could do it in less than ~11MB of RAM. I am barreling into the code now since response to my "plea" for help went unanswered. I would expect that Shared Host providers would want to have somebody look at this because it should be an issue for all Rails apps not just Typo. Imagine if we figure out how to shave 10MB off of each Rails app out there? Shared Hosts become more viable, people send me presents and bow in my presence... I''ve gotten a bit further in my research tonight take a look at my latest post: http://www.stevelongdo.com/articles/2006/08/09/rubygems-single-multi-tons-and-memorysize All hail Stef our dark underlord...Oh not affiliated with Satan, I see. :-P On 8/8/06, Stef T <stef@ummon.com> wrote:> > > Hello Steve, > Well, since you asked for help, can I suggest that you provide > pertinent details ? OS used, ruby version, and also the example code > that you used to profile would be nice ;) I must say, that rubygems > hooks into the main process from what I can (hence the require) so I > would -expect- it to stay resident in memory. Which reminds me, what > memory are you profiling here ? VSZ ? RSZ ? Then again, in the day and > age of 1GB in my desktop machine, its probably highly easy to dismiss a > RSZ of 10MB ;) Heck, almost all of the servers at work have >4GB. Thats > not the point, I know, but, there is no point in chasing rabbits down > holes if you don''t need to (also called the ''premature optimisation'' > problem). > > Details details, the devil is in the details* > Regards > Stef > (* this should not be taken as an endorsement of satan nor that I have > -the- answer to your problems ;) > > Steve Longdo wrote: > > This an open plea for someone with knowledge of RubyGems to explain > > why RubyGems stays in memory (over 10MB) after a Rails application is > > loaded. I am unable to figure out why this would be necessary. I > > haven''t had any response from Jim or Chad from RubyGems about it so I > > thought I would throw it open to the list to see if anyone could > > educate me or share thier experiences with this. > > > > There is a more detailed write up on my blog ( > > > http://www.stevelongdo.com/articles/2006/08/08/rubygems-dependency-management-and-memory-use > > < > http://www.stevelongdo.com/articles/2006/08/08/rubygems-dependency-management-and-memory-use > > > > ). > > >- Thanks, -Steve http://www.stevelongdo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://wrath.rubyonrails.org/pipermail/rails/attachments/20060809/c6a836a6/attachment.html
Hello Steve,
No no, your definitely not a pariah, and your emails are definitely
making it onto the list ;) A crazy thought appears, perhaps woefully and
stupidly in my head, but, its 1AM EST here, so, forgive me for this,
but, why not simply use the design pattern of Singleton in
custom_require.rb if you believe this is the problem ?
There now follows a rather clunky ''human diff'' .. please
bear with me :)
at line 24 (in the require method)
@gempath_searcher ||= Gem::GemPathSearcher.new
should become
@gempath_searcher = Gem::GemPathSearcher.instance
And then you want to change the module Gem part below to look
something like;
module Gem
require ''singleton''
#
# GemPathSearcher has the capability to find loadable files inside
# gems. It generates data up front to speed up searches later.
#
class GemPathSearcher
include Singleton
If you then put a ''puts @gempath_searcher.object_id'' after
the line
24 change, and run (say) your ruby ./script/server, you will see a huge
spew of lines like ;
stef@stef-desktop:~/work/myTest$ ruby ./script/server
-606820690
-606820690
-606820690
If you ''undo'' those changes, and go back to the good
old-fashioned
code, you will see something like;
stef@stef-desktop:~/work/myTest$ ruby ./script/server
-606681356
-606869086
-606681356
-606818976
Different object_id''s means different object''s .. so .. by
using the
Singleton design pattern, your gemserver will now only be created
''once''. Now, of course, wether or not this breaks things
horrible OR if
this does indeed fix your memory leakage, you will have to let us all
know :)
Either way, have a nice night, I am shuffling off to bed, for some
well earned R&R. Let me know how it all works out though please,
irregardless ;)
Regards
Stef
(ps. I could never worship Satan, but only Cthulhu, mean to say, why
worship a lesser evil ;D
(pps. that -is- a joke for the humour impaired ;)
Steve Longdo wrote:> Stef!!!!
>
> Thank you for answering at all! I am a leper on the list it feels
> like sometimes. I am profiling Typo, an app in need of some very
> "Mature Optimization". So no worries there. Locally I use OS
X/PPC,
> Ruby 1.8.4. However I am doing the real profiling against my
> www.stevelongdo.com <http://www.stevelongdo.com> site on TextDrive.
> It is also Ruby 1.8.4 and I think a Linux flavor not Solaris 10 yet
> anyway.
>
> Typo and its external dependant version of Rails were trunk a couple
> days ago. TextDrive limits are based on RSZ going over 48MB. VSZ is
> almost always lower than that when Ruby decides to allocate some more
> space and send RSZ over the limit. I am not sure whether they are
> using Stefan Kaes suggested patched GC or if it is plain vanilla Ruby
> GC. I don''t for sure how to find that out on TextDrive.
>
> At any rate there is no good reason for my personal Typo weblog to
> cross into the 48MB zone. I''ll bet all of my content together is
512kB.
>
> Outside of all that though I am trying to learn more about how
> RubyGems actually does its business and whether it could do it in less
> than ~11MB of RAM. I am barreling into the code now since response to
> my "plea" for help went unanswered. I would expect that Shared
Host
> providers would want to have somebody look at this because it should
> be an issue for all Rails apps not just Typo. Imagine if we figure
> out how to shave 10MB off of each Rails app out there? Shared Hosts
> become more viable, people send me presents and bow in my presence...
>
> I''ve gotten a bit further in my research tonight take a look at my
> latest post:
>
http://www.stevelongdo.com/articles/2006/08/09/rubygems-single-multi-tons-and-memorysize
>
<http://www.stevelongdo.com/articles/2006/08/09/rubygems-single-multi-tons-and-memorysize>
>
>
> All hail Stef our dark underlord...Oh not affiliated with Satan, I
> see. :-P
>
> On 8/8/06, *Stef T* <stef@ummon.com <mailto:stef@ummon.com>>
wrote:
>
>
> Hello Steve,
> Well, since you asked for help, can I suggest that you provide
> pertinent details ? OS used, ruby version, and also the example code
> that you used to profile would be nice ;) I must say, that rubygems
> hooks into the main process from what I can (hence the require) so I
> would -expect- it to stay resident in memory. Which reminds me, what
> memory are you profiling here ? VSZ ? RSZ ? Then again, in the day and
> age of 1GB in my desktop machine, its probably highly easy to
> dismiss a
> RSZ of 10MB ;) Heck, almost all of the servers at work have >4GB.
> Thats
> not the point, I know, but, there is no point in chasing rabbits down
> holes if you don''t need to (also called the
''premature optimisation''
> problem).
>
> Details details, the devil is in the details*
> Regards
> Stef
> (* this should not be taken as an endorsement of satan nor that I have
> -the- answer to your problems ;)
>
> Steve Longdo wrote:
> > This an open plea for someone with knowledge of RubyGems to
explain
> > why RubyGems stays in memory (over 10MB) after a Rails
> application is
> > loaded. I am unable to figure out why this would be necessary. I
> > haven''t had any response from Jim or Chad from RubyGems
about it
> so I
> > thought I would throw it open to the list to see if anyone could
> > educate me or share thier experiences with this.
> >
> > There is a more detailed write up on my blog (
> >
>
http://www.stevelongdo.com/articles/2006/08/08/rubygems-dependency-management-and-memory-use
> >
>
<http://www.stevelongdo.com/articles/2006/08/08/rubygems-dependency-management-and-memory-use
>
<http://www.stevelongdo.com/articles/2006/08/08/rubygems-dependency-management-and-memory-use>>
> > ).
> >
>
> -
> Thanks,
> -Steve
> http://www.stevelongdo.com
I have posted an update on this at my blog: http://www.stevelongdo.com/articles/2006/08/12/rubygems-memory-reduction Basically RubyGems needs to be patched. Stef''s solution below works perfectly, once you figure out how to set it up on a shared host! I will be following up on exactly how to do that tomorrow. Looks like maybe Nicholas Seckar and the Rails core are going to be doing something different soon anyway. http://weblog.rubyonrails.org/2006/8/11/reloading-revamped On 8/9/06, Stef T <stef@ummon.com> wrote:> > > Hello Steve, > No no, your definitely not a pariah, and your emails are definitely > making it onto the list ;) A crazy thought appears, perhaps woefully and > stupidly in my head, but, its 1AM EST here, so, forgive me for this, > but, why not simply use the design pattern of Singleton in > custom_require.rb if you believe this is the problem ? > > There now follows a rather clunky ''human diff'' .. please bear with me > :) > > at line 24 (in the require method) > @gempath_searcher ||= Gem::GemPathSearcher.new > should become > @gempath_searcher = Gem::GemPathSearcher.instance > > And then you want to change the module Gem part below to look > something like; > > module Gem > require ''singleton'' > # > # GemPathSearcher has the capability to find loadable files inside > # gems. It generates data up front to speed up searches later. > # > class GemPathSearcher > include Singleton > > If you then put a ''puts @gempath_searcher.object_id'' after the line > 24 change, and run (say) your ruby ./script/server, you will see a huge > spew of lines like ; > > stef@stef-desktop:~/work/myTest$ ruby ./script/server > -606820690 > -606820690 > -606820690 > > If you ''undo'' those changes, and go back to the good old-fashioned > code, you will see something like; > > stef@stef-desktop:~/work/myTest$ ruby ./script/server > -606681356 > -606869086 > -606681356 > -606818976 > > Different object_id''s means different object''s .. so .. by using the > Singleton design pattern, your gemserver will now only be created > ''once''. Now, of course, wether or not this breaks things horrible OR if > this does indeed fix your memory leakage, you will have to let us all > know :) > > Either way, have a nice night, I am shuffling off to bed, for some > well earned R&R. Let me know how it all works out though please, > irregardless ;) > Regards > Stef > (ps. I could never worship Satan, but only Cthulhu, mean to say, why > worship a lesser evil ;D > (pps. that -is- a joke for the humour impaired ;) > Steve Longdo wrote: > > Stef!!!! > > > > Thank you for answering at all! I am a leper on the list it feels > > like sometimes. I am profiling Typo, an app in need of some very > > "Mature Optimization". So no worries there. Locally I use OS X/PPC, > > Ruby 1.8.4. However I am doing the real profiling against my > > www.stevelongdo.com <http://www.stevelongdo.com> site on TextDrive. > > It is also Ruby 1.8.4 and I think a Linux flavor not Solaris 10 yet > > anyway. > > > > Typo and its external dependant version of Rails were trunk a couple > > days ago. TextDrive limits are based on RSZ going over 48MB. VSZ is > > almost always lower than that when Ruby decides to allocate some more > > space and send RSZ over the limit. I am not sure whether they are > > using Stefan Kaes suggested patched GC or if it is plain vanilla Ruby > > GC. I don''t for sure how to find that out on TextDrive. > > > > At any rate there is no good reason for my personal Typo weblog to > > cross into the 48MB zone. I''ll bet all of my content together is 512kB. > > > > Outside of all that though I am trying to learn more about how > > RubyGems actually does its business and whether it could do it in less > > than ~11MB of RAM. I am barreling into the code now since response to > > my "plea" for help went unanswered. I would expect that Shared Host > > providers would want to have somebody look at this because it should > > be an issue for all Rails apps not just Typo. Imagine if we figure > > out how to shave 10MB off of each Rails app out there? Shared Hosts > > become more viable, people send me presents and bow in my presence... > > > > I''ve gotten a bit further in my research tonight take a look at my > > latest post: > > > http://www.stevelongdo.com/articles/2006/08/09/rubygems-single-multi-tons-and-memorysize > > < > http://www.stevelongdo.com/articles/2006/08/09/rubygems-single-multi-tons-and-memorysize > > > > > > > > All hail Stef our dark underlord...Oh not affiliated with Satan, I > > see. :-P > > > > On 8/8/06, *Stef T* <stef@ummon.com <mailto:stef@ummon.com>> wrote: > > > > > > Hello Steve, > > Well, since you asked for help, can I suggest that you provide > > pertinent details ? OS used, ruby version, and also the example code > > that you used to profile would be nice ;) I must say, that rubygems > > hooks into the main process from what I can (hence the require) so I > > would -expect- it to stay resident in memory. Which reminds me, what > > memory are you profiling here ? VSZ ? RSZ ? Then again, in the day > and > > age of 1GB in my desktop machine, its probably highly easy to > > dismiss a > > RSZ of 10MB ;) Heck, almost all of the servers at work have >4GB. > > Thats > > not the point, I know, but, there is no point in chasing rabbits > down > > holes if you don''t need to (also called the ''premature optimisation'' > > problem). > > > > Details details, the devil is in the details* > > Regards > > Stef > > (* this should not be taken as an endorsement of satan nor that I > have > > -the- answer to your problems ;) > > > > Steve Longdo wrote: > > > This an open plea for someone with knowledge of RubyGems to > explain > > > why RubyGems stays in memory (over 10MB) after a Rails > > application is > > > loaded. I am unable to figure out why this would be necessary. I > > > haven''t had any response from Jim or Chad from RubyGems about it > > so I > > > thought I would throw it open to the list to see if anyone could > > > educate me or share thier experiences with this. > > > > > > There is a more detailed write up on my blog ( > > > > > > http://www.stevelongdo.com/articles/2006/08/08/rubygems-dependency-management-and-memory-use > > > > > < > http://www.stevelongdo.com/articles/2006/08/08/rubygems-dependency-management-and-memory-use > > < > http://www.stevelongdo.com/articles/2006/08/08/rubygems-dependency-management-and-memory-use > >> > > > ). > > > > > > > - > > Thanks, > > -Steve > > http://www.stevelongdo.com > >-- Thanks, -Steve http://www.stevelongdo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://wrath.rubyonrails.org/pipermail/rails/attachments/20060812/c243a7d9/attachment-0001.html