Rails core - Jan 2006 - spinner/spawner: is it really necessary to load the Rails environment?

Hi,

since the Rails env needs quite a lot of RAM and start-up time, is it 
really necessary to load it in the spinner/spawner processes?

Andreas

> since the Rails env needs quite a lot of RAM and start-up time, is it
> really necessary to load it in the spinner/spawner processes?
I''m not sure I understand the question. You can start FCGIs any way
you want. The spawner is just a shortcut for starting many FCGIs at
once and the spinner is a insurance against fallen FCGIs. How does
either have any effect on RAM or start-up time?
--
David Heinemeier Hansson
http://www.loudthinking.com -- Broadcasting Brain
http://www.basecamphq.com   -- Online project management
http://www.backpackit.com   -- Personal information manager
http://www.rubyonrails.com  -- Web-application framework

David Heinemeier Hansson wrote:
>>since the Rails env needs quite a lot of RAM and start-up time, is it
>>really necessary to load it in the spinner/spawner processes?
> 
> 
> I''m not sure I understand the question. You can start FCGIs any
way
> you want. The spawner is just a shortcut for starting many FCGIs at
> once and the spinner is a insurance against fallen FCGIs. How does
> either have any effect on RAM or start-up time?
The spinner process is running all the time, and because it has loaded 
and initialized the whole Rails environment (boot.rb etc.) it uses a lot 
of RAM. The same goes for the spawner process. It is started every 
#{interval} seconds, and because it loads the Rails environment every 
time - this can put a considerable load on a small server. IMO neither 
spinner nor spawner do really need the Rails environment, all they do is 
to call external scripts.

Andreas

> > I''m not sure I understand the question. You can start FCGIs
any way
> > you want. The spawner is just a shortcut for starting many FCGIs at
> > once and the spinner is a insurance against fallen FCGIs. How does
> > either have any effect on RAM or start-up time?
>
> The spinner process is running all the time, and because it has loaded
> and initialized the whole Rails environment (boot.rb etc.) it uses a lot
> of RAM. The same goes for the spawner process. It is started every
> #{interval} seconds, and because it loads the Rails environment every
> time - this can put a considerable load on a small server. IMO neither
> spinner nor spawner do really need the Rails environment, all they do is
> to call external scripts.
If you''re in an environment where the RAM usage of 1 spinner is a
concern, then you probably don''t need/want the functionality of the
spinner anyway. The spawner is triggered by the spinner, so if you
just use it on its own, then it only uses the RAM for a few seconds
until all the spinners are started.

But hey, if you can make a silky clean refactoring that makes the
scripts not require the environment startup, but still use stubs like
all the others, then I''m all game for applying!
--
David Heinemeier Hansson
http://www.loudthinking.com -- Broadcasting Brain
http://www.basecamphq.com   -- Online project management
http://www.backpackit.com   -- Personal information manager
http://www.rubyonrails.com  -- Web-application framework

> The spinner process is running all the time, and because it has loaded
> and initialized the whole Rails environment (boot.rb etc.) it uses a lot
> of RAM. The same goes for the spawner process. It is started every
> #{interval} seconds, and because it loads the Rails environment every
> time - this can put a considerable load on a small server. IMO neither
> spinner nor spawner do really need the Rails environment, all they do is
> to call external scripts.
I deploy apps on a shared hosting server without the spinner, and
everything is fine.  I just use the facilities provided by Lighttpd.

--
rick
http://techno-weenie.net

Rick Olson wrote:
>>The spinner process is running all the time, and because it has loaded
>>and initialized the whole Rails environment (boot.rb etc.) it uses a lot
>>of RAM. The same goes for the spawner process. It is started every
>>#{interval} seconds, and because it loads the Rails environment every
>>time - this can put a considerable load on a small server. IMO neither
>>spinner nor spawner do really need the Rails environment, all they do is
>>to call external scripts.
> 
> 
> I deploy apps on a shared hosting server without the spinner, and
> everything is fine.  I just use the facilities provided by Lighttpd.
I''m doing this at the moment, but I would prefer to be able to restart 
Lighttpd and the various FastCGI servers independently.

On Jan 9, 2006, at 8:41 PM, Rick Olson wrote:
>> The spinner process is running all the time, and because it has  
>> loaded
>> and initialized the whole Rails environment (boot.rb etc.) it uses  
>> a lot
>> of RAM. The same goes for the spawner process. It is started every
>> #{interval} seconds, and because it loads the Rails environment every
>> time - this can put a considerable load on a small server. IMO  
>> neither
>> spinner nor spawner do really need the Rails environment, all they  
>> do is
>> to call external scripts.
>
> I deploy apps on a shared hosting server without the spinner, and
> everything is fine.  I just use the facilities provided by Lighttpd.
Note, too, that boot.rb is not the "whole Rails
environment"--that''s
environment.rb. Boot.rb just sets up the initializer so that the path  
to the railties can be found. Thus, spinner/spawner/etc. load only a  
bare minimal subset of the rails environment, just enough to find the  
files they need in railties/lib/commands/process.

- Jamis

> Note, too, that boot.rb is not the "whole Rails
environment"--that''s
> environment.rb. Boot.rb just sets up the initializer so that the path
> to the railties can be found. Thus, spinner/spawner/etc. load only a
> bare minimal subset of the rails environment, just enough to find the
> files they need in railties/lib/commands/process.
That''s true. How much memory did you find the spinner to take,
Andreas? I believe the Ruby baseline is something like 3.7MB, if I
remember correctly.
--
David Heinemeier Hansson
http://www.loudthinking.com -- Broadcasting Brain
http://www.basecamphq.com   -- Online project management
http://www.backpackit.com   -- Personal information manager
http://www.rubyonrails.com  -- Web-application framework

David Heinemeier Hansson wrote:
>>>I''m not sure I understand the question. You can start FCGIs
any way
>>>you want. The spawner is just a shortcut for starting many FCGIs at
>>>once and the spinner is a insurance against fallen FCGIs. How does
>>>either have any effect on RAM or start-up time?
>>
>>The spinner process is running all the time, and because it has loaded
>>and initialized the whole Rails environment (boot.rb etc.) it uses a lot
>>of RAM. The same goes for the spawner process. It is started every
>>#{interval} seconds, and because it loads the Rails environment every
>>time - this can put a considerable load on a small server. IMO neither
>>spinner nor spawner do really need the Rails environment, all they do is
>>to call external scripts.
> 
> 
> If you''re in an environment where the RAM usage of 1 spinner is a
> concern, then you probably don''t need/want the functionality of
the
> spinner anyway. The spawner is triggered by the spinner, so if you
> just use it on its own, then it only uses the RAM for a few seconds
> until all the spinners are started.
It''s the other way round. The spinner creates a new spawner process 
every #{interval = 5} seconds. Starting the spawner is rather expensive 
if it''s done in a 5-second interval.
> But hey, if you can make a silky clean refactoring that makes the
> scripts not require the environment startup, but still use stubs like
> all the others, then I''m all game for applying!
I can''t think of an elegent way to do this right now, but attached is a
small patch that moves the spinner functionality into the spawner.


_______________________________________________
Rails-core mailing list
Rails-core@lists.rubyonrails.org
http://lists.rubyonrails.org/mailman/listinfo/rails-core

David Heinemeier Hansson wrote:
>>Note, too, that boot.rb is not the "whole Rails
environment"--that''s
>>environment.rb. Boot.rb just sets up the initializer so that the path
>>to the railties can be found. Thus, spinner/spawner/etc. load only a
>>bare minimal subset of the rails environment, just enough to find the
>>files they need in railties/lib/commands/process.
> 
> 
> That''s true. How much memory did you find the spinner to take,
> Andreas?
About 17 MB - that''s huge for my tiny server :)

> I''m doing this at the moment, but I would prefer to be able to
restart
> Lighttpd and the various FastCGI servers independently.
ps -ax | grep ruby
kill -s USR2 {PID}

That will restart a ruby process.  It should have the full path of
your dispatch.fcgi file in the process line.  Unfortunately, you can''t
restart Lighty without bringing everything else down.  Or if you can,
I''d sure like to know :)

I think kill -s USR2 {lighttpd_pid} will tell lighty to re-load its
config though.


--
rick
http://techno-weenie.net

I suggest peeking at the lighttpd-config package I have made to look at how
you can manage many unique instances of lighttpd in a virtual host
environment.

http://rubyforge.org/projects/lighttpd-config

I have the configuration options abstracted to the point I can deploy a new
lighttpd instance in about 60 seconds. I hope to shorten this after
gemifying the package and having a setup script to configure the variables
through CLI prompts. If you are game to do this, I would be happy to commit
and credit ;)

-Nb

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Nathaniel S. H. Brown                           http://nshb.net 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
> -----Original Message-----
> From: rails-core-bounces@lists.rubyonrails.org 
> [mailto:rails-core-bounces@lists.rubyonrails.org] On Behalf Of Andreas
> Sent: January 9, 2006 8:03 PM
> To: rails-core@lists.rubyonrails.org
> Subject: [Rails-core] Re: spinner/spawner: is it really 
> necessary to loadthe Rails environment?
> 
> Rick Olson wrote:
> >>The spinner process is running all the time, and because it 
> has loaded 
> >>and initialized the whole Rails environment (boot.rb etc.) 
> it uses a 
> >>lot of RAM. The same goes for the spawner process. It is 
> started every 
> >>#{interval} seconds, and because it loads the Rails 
> environment every 
> >>time - this can put a considerable load on a small server. 
> IMO neither 
> >>spinner nor spawner do really need the Rails environment, 
> all they do 
> >>is to call external scripts.
> > 
> > 
> > I deploy apps on a shared hosting server without the spinner, and 
> > everything is fine.  I just use the facilities provided by Lighttpd.
> 
> I''m doing this at the moment, but I would prefer to be able 
> to restart Lighttpd and the various FastCGI servers independently.
> 
> _______________________________________________
> Rails-core mailing list
> Rails-core@lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails-core
>

> I think kill -s USR2 {lighttpd_pid} will tell lighty to re-load its
> config though.
Remember too that script/process/repear will restart the FCGIs from
within themselves. So if you just need to deploy updated application
logic, you don''t need (and shouldn''t) restart the web server.
--
David Heinemeier Hansson
http://www.loudthinking.com -- Broadcasting Brain
http://www.basecamphq.com   -- Online project management
http://www.backpackit.com   -- Personal information manager
http://www.rubyonrails.com  -- Web-application framework

Andreas Schwarz schrieb:
> attached is a
> small patch that moves the spinner functionality into the spawner.
I created a ticket for this patch:
http://dev.rubyonrails.org/ticket/3461