I just recently upgraded a rails app of mine to run on edge (and the 1-2-pre-release branch) and I noticed my fcgis required roughly 6-8MBs more memory after just a couple requests. For example, each fcgi on edge would start around 40MB and rise to ~46MBs after a couple requests. I downgraded my app back to 1.1.6 and each fcgi would start at around 33MBs and rise to ~38MB. As a result of the increased memory requirements, my app can no longer be run on TextDrive using edge since they have a max resident memory cap of 48MBs which I would quickly hit (and they just kill your fcgi when you do). Is this increased memory requirement to be expected? I would be happy to give you more details about my setup if you think it may have an impact on why the memory usage is so high in my case. Thanks, Tom Davies http://atomgiant.com http://gifthat.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
> Is this increased memory requirement to be expected? > > I would be happy to give you more details about my setup if you think > it may have an impact on why the memory usage is so high in my case.I''ve had a sneaking suspicion about this for awhile. Even Beast, my 500 LOC forum starts off with about 40MB. First I compared Beast on edge vs Beast on rev 4818 (edge rails at the time of its first release). They both started at about 22MB, but Beast on edge jumped to about 37MB after making a single request to 4 controllers. I then incremented the revision number until I started seeing the same memory increases and found it started after changeset 5223, the one that introduced ActiveSupport::Multibyte. Anyone else seeing similar results? -- Rick Olson http://weblog.techno-weenie.net http://mephistoblog.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
I ran my app using mongrels -B flag to see if I was leaking memory and one thing I noticed was the second highest entry is related to Multibyte: ActiveSupport::Multibyte::Handlers::Codepoint, 17721 So, it would seem that multibyte may be the culprit. Tom On 11/8/06, Rick Olson <technoweenie@gmail.com> wrote:> > > Is this increased memory requirement to be expected? > > > > I would be happy to give you more details about my setup if you think > > it may have an impact on why the memory usage is so high in my case. > > I''ve had a sneaking suspicion about this for awhile. Even Beast, my > 500 LOC forum starts off with about 40MB. > > First I compared Beast on edge vs Beast on rev 4818 (edge rails at the > time of its first release). They both started at about 22MB, but > Beast on edge jumped to about 37MB after making a single request to 4 > controllers. I then incremented the revision number until I started > seeing the same memory increases and found it started after changeset > 5223, the one that introduced ActiveSupport::Multibyte. Anyone else > seeing similar results? > > -- > Rick Olson > http://weblog.techno-weenie.net > http://mephistoblog.com > > > >-- Tom Davies http://atomgiant.com http://gifthat.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
On 11/9/06, Tom Davies <atomgiant@gmail.com> wrote:> > I ran my app using mongrels -B flag to see if I was leaking memory and > one thing I noticed was the second highest entry is related to > Multibyte: > > ActiveSupport::Multibyte::Handlers::Codepoint, 17721 > > So, it would seem that multibyte may be the culprit.Does redefining KCODE, or making String#chars just return self make any difference? -- Cheers Koz --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
On Nov 8, 2006, at 7:51 PM, Michael Koziarski wrote:> > Does redefining KCODE, or making String#chars just return self make > any difference?Nope, the unicode tables are always loaded and frankly take up a lot of space. The reason they''re always loaded is that we don''t know in advance if someone is going to set KCODE to utf-8. Manfred --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
there isn''t some kind of ruby event that checks when a constant or a variable gets assigned? this way you could do your stuff at the moment KCODE gets set. but it would not help much for user like me who set it almost all the time. On 11/9/06, Manfred Stienstra <manfred@gmail.com> wrote:> > On Nov 8, 2006, at 7:51 PM, Michael Koziarski wrote: > > > > Does redefining KCODE, or making String#chars just return self make > > any difference? > > Nope, the unicode tables are always loaded and frankly take up a lot > of space. The reason they''re always loaded is that we don''t know in > advance if someone is going to set KCODE to utf-8. > > Manfred > > > >-- gcc -O0 -DRUBY_EXPORT -rdynamic -Wl,-export-dynamic -L. main.o -lruby-static -ldl -lcrypt -lm -o ruby Everyone is trying their hardest to do their job but management has set it up so that it''s impossible. Take the control over your money, track your expenses http://justbudget.com Mathieu --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
On 11/9/06, Manfred Stienstra <manfred@gmail.com> wrote:> > On Nov 8, 2006, at 7:51 PM, Michael Koziarski wrote: > > > > Does redefining KCODE, or making String#chars just return self make > > any difference? > > Nope, the unicode tables are always loaded and frankly take up a lot > of space. The reason they''re always loaded is that we don''t know in > advance if someone is going to set KCODE to utf-8.Couldn''t we lazy load the UnicodeDatabase instance? # UniCode Database UCD = UnicodeDatabase.new That way instead of chewing up that ram every time, we just chew it up when using the utf8 handler? Also, does the utf8_handler_proc implementation need UCD? -- Cheers Koz --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
On Nov 9, 2006, at 12:47 AM, Michael Koziarski wrote:> > Couldn''t we lazy load the UnicodeDatabase instance? > > # UniCode Database > UCD = UnicodeDatabase.new > > That way instead of chewing up that ram every time, we just chew it up > when using the utf8 handler? > > Also, does the utf8_handler_proc implementation need UCD?Yes, we can lazy load the database, I think that''s the best solution... The _proc implementation wouldn''t need the UCD if all the unicode operations we need were implemented in utf8_proc, but unfortunately downcase is broken in utf8_proc at the moment. All the alternative unicode handlers are descendants of utf8_handler so we always have to pure Ruby fallback. Manfred --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
> Yes, we can lazy load the database, I think that''s the best solution...Awesome, care to make a patch?> The _proc implementation wouldn''t need the UCD if all the unicode > operations we need were implemented in utf8_proc, but unfortunately > downcase is broken in utf8_proc at the moment. All the alternative > unicode handlers are descendants of utf8_handler so we always have to > pure Ruby fallback.Either way, if KCODE is set to something other than ''u'', then we should probably ensure that people don''t have to load the big codepoints table. That way people with memory constrained environments can still run. Then if someone really cares about memory usage and String#chars, they could improve one of the c-based handlers to completely replace the pure-ruby fallback position. -- Cheers Koz --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
On 9-nov-2006, at 9:48, Michael Koziarski wrote:> >> Yes, we can lazy load the database, I think that''s the best >> solution... > > Awesome, care to make a patch?Sure. I''ll get one done before the end of the weekend.>> The _proc implementation wouldn''t need the UCD if all the unicode >> operations we need were implemented in utf8_proc, but unfortunately >> downcase is broken in utf8_proc at the moment. All the alternative >> unicode handlers are descendants of utf8_handler so we always have to >> pure Ruby fallback. > > Either way, if KCODE is set to something other than ''u'', then we > should probably ensure that people don''t have to load the big > codepoints table. That way people with memory constrained > environments can still run.I agree. I didn''t notice it took that much memory. I never expected that a marshaled file of 600k could grow to more than 15 Mb in Ruby.> Then if someone really cares about memory usage and String#chars, they > could improve one of the c-based handlers to completely replace the > pure-ruby fallback position.That was also on the agenda, but I have to admit I don''t know if I have time for that anywhere this year (: Manfred --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
Hi all, I can''t take a look at the multibyte support right now but could this http://redhanded.hobix.com/inspect/theMarshWalker.html be used to further reduce the memory requirements by lazy loading the codepoints themselve ? even at the cost of increased load times for the first pages. for people needing a bit of international language on a shared hosting this could be a useful compromise (if at all possible). jean On 11/9/06, Manfred Stienstra <manfred@gmail.com> wrote:> > > On 9-nov-2006, at 9:48, Michael Koziarski wrote: > > > > >> Yes, we can lazy load the database, I think that''s the best > >> solution... > > > > Awesome, care to make a patch? > > Sure. I''ll get one done before the end of the weekend. > > >> The _proc implementation wouldn''t need the UCD if all the unicode > >> operations we need were implemented in utf8_proc, but unfortunately > >> downcase is broken in utf8_proc at the moment. All the alternative > >> unicode handlers are descendants of utf8_handler so we always have to > >> pure Ruby fallback. > > > > Either way, if KCODE is set to something other than ''u'', then we > > should probably ensure that people don''t have to load the big > > codepoints table. That way people with memory constrained > > environments can still run. > > I agree. I didn''t notice it took that much memory. I never expected > that a marshaled file of 600k could grow to more than 15 Mb in Ruby. > > > Then if someone really cares about memory usage and String#chars, they > > could improve one of the c-based handlers to completely replace the > > pure-ruby fallback position. > > That was also on the agenda, but I have to admit I don''t know if I > have time for that anywhere this year (: > > Manfred > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
On 9-nov-2006, at 11:44, Jean Helou wrote:> > I can''t take a look at the multibyte support right now but could this > http://redhanded.hobix.com/inspect/theMarshWalker.html be used to > further reduce the memory requirements by lazy loading the codepoints > themselve ? even at the cost of increased load times for the first > pages.Thanks for the interesting suggestion, but loading all the codepoints separately would force the unicode operations to a slow and grinding halt. We _can_ load the unicode database in parts. I will investigate some solutions. Manfred --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
> > Sure. I''ll get one done before the end of the weekend. >Excellent. Thanks for your help guys. I am sure everyone running on shared hosts will thank you as well. Tom Davies http://atomgiant.com http://gifthat.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
> Sure. I''ll get one done before the end of the weekend.http://dev.rubyonrails.org/changeset/5476 AS tests all pass. All good? I''ll merge this to stable if it is. -- Rick Olson http://weblog.techno-weenie.net http://mephistoblog.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
On Nov 9, 2006, at 8:42 PM, Rick Olson wrote:>> Sure. I''ll get one done before the end of the weekend. > > http://dev.rubyonrails.org/changeset/5476 > > AS tests all pass. All good? I''ll merge this to stable if it is.All the tests run for me too and I don''t see any problems with this solution. Thanks, you just freed up my weekend (: Manfred --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---