I hacked a forking FCGI dispatcher for Rails. I have no idea if this is
the correct way to do it, but it seems to work fine. It improves memory
usage a bit; after it is started the processes share almost 10 MB (of 12
MB total), but unfortunately it drops to 2 MB after a few requests.
====================================#!/usr/local/bin/ruby
require File.dirname(__FILE__) + "/../config/environment"
require ''dispatcher''
require ''fcgi''
parent = true
num = 0
max = 3
while parent && num < max
pid = fork()
if pid == nil
# child
parent = false
else
# parent
num += 1
end
end
if parent
Process.wait
else
FCGI.each_cgi { |cgi| Dispatcher.dispatch(cgi) }
end
=====================================
On Thu, 10 Mar 2005, Andreas Schwarz wrote:> I hacked a forking FCGI dispatcher for Rails. I have no idea if this is the > correct way to do it, but it seems to work fine. It improves memory usage a > bit; after it is started the processes share almost 10 MB (of 12 MB total), > but unfortunately it drops to 2 MB after a few requests. > > ====================================> #!/usr/local/bin/ruby > > require File.dirname(__FILE__) + "/../config/environment" > require ''dispatcher'' > require ''fcgi'' > > parent = true > num = 0 > max = 3 > while parent && num < max > pid = fork() > if pid == nil > # child > parent = false > else > # parent > num += 1 > end > end > > if parent > Process.wait > else > FCGI.each_cgi { |cgi| Dispatcher.dispatch(cgi) } > end > ====================================why are you doing this? not using apache? this is essentially what the fastcgi process manager in apache already does and the number of processes is configurable in httpd.conf unless, of course, you aren''t using apache... also - it doesn''t seem like this would work: you will carry your ipc channels going between web server and original fcgi process from parent to child and then all children will share the same pipes back to web server. but maybe i am missing something here since you are saying it works... something tells me that multiple requests will make this fail... what happens when web server attempts requests number 4 and you are waiting for 3 processes which never exit? again, i might not be wrapping my mind around this but it seems not quite right... interesting though ;-) cheers. -a -- ==============================================================================| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov | PHONE :: 303.497.6469 | When you do something, you should burn yourself completely, like a good | bonfire, leaving no trace of yourself. --Shunryu Suzuki ===============================================================================
Isn''t that mod_fastcgi can do this for you?
BTW, probably you should use Process.waitall method like in:
#!/usr/local/bin/ruby
require File.dirname(__FILE__) + "/../config/environment"
require ''dispatcher''
require ''fcgi''
MAX_CHILDREN = 3
MAX_CHILDREN.times do
fork do
FCGI.each_cgi { |cgi| Dispatcher.dispatch(cgi) }
exit
end
end
Process.waitall
Cheers,
Kent
Andreas Schwarz <usenet-ARtvInVfO7ksV2N9l4h3zg@public.gmane.org> writes:
> I hacked a forking FCGI dispatcher for Rails. I have no idea if this
> is the correct way to do it, but it seems to work fine. It improves
> memory usage a bit; after it is started the processes share almost 10
> MB (of 12 MB total), but unfortunately it drops to 2 MB after a few
> requests.
>
> ====================================> #!/usr/local/bin/ruby
> require File.dirname(__FILE__) + "/../config/environment"
> require ''dispatcher''
> require ''fcgi''
> parent = true
> num = 0
> max = 3
> while parent && num < max
> pid = fork()
> if pid == nil
> # child
> parent = false
> else
> # parent
> num += 1
> end
> end
> if parent
> Process.wait
> else
> FCGI.each_cgi { |cgi| Dispatcher.dispatch(cgi) }
> end
> ====================================>
> _______________________________________________
> Rails mailing list
> Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
Ara.T.Howard wrote:> On Thu, 10 Mar 2005, Andreas Schwarz wrote: > >> I hacked a forking FCGI dispatcher for Rails. I have no idea if this >> is the >> correct way to do it, but it seems to work fine. It improves memory >> usage a >> bit; after it is started the processes share almost 10 MB (of 12 MB >> total), >> but unfortunately it drops to 2 MB after a few requests. >> >> ====================================>> #!/usr/local/bin/ruby >> >> require File.dirname(__FILE__) + "/../config/environment" >> require ''dispatcher'' >> require ''fcgi'' >> >> parent = true >> num = 0 >> max = 3 >> while parent && num < max >> pid = fork() >> if pid == nil >> # child >> parent = false >> else >> # parent >> num += 1 >> end >> end >> >> if parent >> Process.wait >> else >> FCGI.each_cgi { |cgi| Dispatcher.dispatch(cgi) } >> end >> ====================================> > > why are you doing this? not using apache? this is essentially what the > fastcgi process manager in apache already does and the number of > processes is > configurable in httpd.conf unless, of course, you aren''t using apache...The difference is that Apache starts independent processes, while the forked processes share their memory. A memory page is only copied when the process starts writing something in the memory ("copy on write"). Unfortunately the ruby processes seem to write stuff to a lot of different pages when a request is handled, so in the end most of the memory won''t be shared any more.> also - it doesn''t seem like this would work: you will carry your ipc > channels > going between web server and original fcgi process from parent to child and > then all children will share the same pipes back to web server. but > maybe i > am missing something here since you are saying it works... something > tells me > that multiple requests will make this fail...I have no idea - I tested it with a few thousand concurrent requests, and there weren''t any problems. I wouldn''t use it for production sites, though; it''s only a hack to see how much memory can be saved with fork. PHP does the same, one socket for multiple forked FCGI servers.
Kent Sibilev wrote:> Isn''t that mod_fastcgi can do this for you?The difference is the memory usage, see my reply to Ara''s post.> BTW, probably you should use Process.waitall method like in: > > #!/usr/local/bin/ruby > > require File.dirname(__FILE__) + "/../config/environment" > require ''dispatcher'' > require ''fcgi'' > > MAX_CHILDREN = 3 > > MAX_CHILDREN.times do > fork do > FCGI.each_cgi { |cgi| Dispatcher.dispatch(cgi) } > exit > end > end > Process.waitallThanks, this looks a lot better than my "C style" code.
On Thu, 10 Mar 2005, Andreas Schwarz wrote:> The difference is that Apache starts independent processes, while the forked > processes share their memory. A memory page is only copied when the process > starts writing something in the memory ("copy on write"). Unfortunately the > ruby processes seem to write stuff to a lot of different pages when a > request is handled, so in the end most of the memory won''t be shared any > more.right - so why do it if there is no real savings? i''m not so sure it''s so simple anyhow: your forked processes take memory in the kernel so your processes could look a bit smaller when actually the system is using just as much memory - this is tough to meter...>> also - it doesn''t seem like this would work: you will carry your ipc >> channels going between web server and original fcgi process from parent to >> child and then all children will share the same pipes back to web server. >> but maybe i am missing something here since you are saying it works... >> something tells me that multiple requests will make this fail... > > I have no idea - I tested it with a few thousand concurrent requests, and > there weren''t any problems. I wouldn''t use it for production sites, though; > it''s only a hack to see how much memory can be saved with fork. > > PHP does the same, one socket for multiple forked FCGI servers.huh. i''ll have to wrap my head around that... cheers. -a -- ==============================================================================| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov | PHONE :: 303.497.6469 | When you do something, you should burn yourself completely, like a good | bonfire, leaving no trace of yourself. --Shunryu Suzuki ===============================================================================
This is sliding off-topic a bit, but this might be interesting to anyone familiar with fastcgi internals to consider: http://www.mems-exchange.org/software/scgi/ "SCGI is a CGI replacement written by Neil Schemenauer, one of Quixote''s developers, and is similar to FastCGI but is designed to be easier to implement. mod_scgi simply forwards requests to an already-running SCGI server on a different TCP port, and doesn''t try to start or stop processes, leaving that up to the SCGI server." Quixote is pretty dang fast from many accounts, and these guys are good coders (python), so might be worth looking at for Rails usage if people have trouble with fastcgi - of course the ruby end would need to be written... "Easier to implement" sounds Rails-ish, though I guess this would be going against the "it already works" part of fastcgi. And lighttpd already supports fcgi. Just a thought... -- ______________________________ toddgrimason*todd-AT-slack.net
Ara.T.Howard wrote:> On Thu, 10 Mar 2005, Andreas Schwarz wrote: > >> The difference is that Apache starts independent processes, while the >> forked >> processes share their memory. A memory page is only copied when the >> process >> starts writing something in the memory ("copy on write"). >> Unfortunately the >> ruby processes seem to write stuff to a lot of different pages when a >> request is handled, so in the end most of the memory won''t be shared any >> more. > > > right - so why do it if there is no real savings?There are some savings, but unfortunately not as much as with PHP.> i''m not so sure it''s so > simple anyhow: your forked processes take memory in the kernel so your > processes could look a bit smaller when actually the system is using > just as > much memory - this is tough to meter...I looked at the "shared" values in top.