Rails - Dec 2005 - Difficult process of restarting SCGI, Lighttpd, Apache, sessions

I''ve managed to get Rails working using Apache->Lighttpd->SCGI.
Restarting everything is really
difficult. This appears to be the process:

- stop apache
- stop lighttpd
- stop scgi
- make sure scgi still isn''t running (it sometimes does) - if it is,
kill it
- delete all session files (they often cause permission errors)
- start scgi
- start apache
- start lighttpd
- cross fingers (since this process doesn''t always work and I have to
repeat it)

When the above doesn''t work, I get ''500'' errors on
Rails'' pages, ''Web server may possibly be
configured wrong.'' in scgi''s log, nothing in Rails''
logs, and only proxy connection errors in
apache''s logs if it can''t connect.

Is there an easier and less problematic way?

thanks
csn



	
		
__________________________________ 
Start your day with Yahoo! - Make it your home page! 
http://www.yahoo.com/r/hs

I''ve never used the exact setup you have, but generally when you need
to perform the same tasks over and over it''s nice to write a single
script that performs all the commands.  Then you can just call

# ./uber-restart-script

and everything is taken care of.

Pat


On 12/1/05, CSN
<cool_screen_name90001-/E1597aS9LQAvxtiuMwx3w@public.gmane.org>
wrote:
> I''ve managed to get Rails working using
Apache->Lighttpd->SCGI. Restarting everything is really
> difficult. This appears to be the process:
>
> - stop apache
> - stop lighttpd
> - stop scgi
> - make sure scgi still isn''t running (it sometimes does) - if it
is, kill it
> - delete all session files (they often cause permission errors)
> - start scgi
> - start apache
> - start lighttpd
> - cross fingers (since this process doesn''t always work and I have
to repeat it)
>
> When the above doesn''t work, I get ''500'' errors
on Rails'' pages, ''Web server may possibly be
> configured wrong.'' in scgi''s log, nothing in
Rails'' logs, and only proxy connection errors in
> apache''s logs if it can''t connect.
>
> Is there an easier and less problematic way?
>
> thanks
> csn
>
>
>
>
>
> __________________________________
> Start your day with Yahoo! - Make it your home page!
> http://www.yahoo.com/r/hs
> _______________________________________________
> Rails mailing list
> Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
>

On Dec 1, 2005, at 12:09 PM, CSN wrote:
> I''ve managed to get Rails working using
Apache->Lighttpd->SCGI.
> Restarting everything is really
> difficult. This appears to be the process:
>
> - stop apache
> - stop lighttpd
> - stop scgi
> - make sure scgi still isn''t running (it sometimes does) - if it  
> is, kill it
> - delete all session files (they often cause permission errors)
> - start scgi
> - start apache
> - start lighttpd
> - cross fingers (since this process doesn''t always work and I have
> to repeat it)
>
> When the above doesn''t work, I get ''500'' errors
on Rails'' pages,
> ''Web server may possibly be
> configured wrong.'' in scgi''s log, nothing in
Rails'' logs, and only
> proxy connection errors in
> apache''s logs if it can''t connect.
>
> Is there an easier and less problematic way?
>
> thanks
> csn
>
I don''t think you should need to restart apache or lighttpd. Since  
scgi lets you rails app run on its own outside the webserver you  
should just stop and restart your scgi processes. Thats one of the  
advantages to scgi.


Cheers-

-Ezra Zygmuntowicz
Yakima Herald-Republic
WebMaster
http://yakimaherald.com
509-577-7732
ezra-gdxLOakOTQ9oetBuM9ipNAC/G2K4zDHf@public.gmane.org

--- Ezra Zygmuntowicz <ezra-SD1UcFUIF7QjH4SCZfkgnw@public.gmane.org>
wrote:
> 
> On Dec 1, 2005, at 12:09 PM, CSN wrote:
> 
> > I''ve managed to get Rails working using
Apache->Lighttpd->SCGI.
> > Restarting everything is really
> > difficult. This appears to be the process:
> >
> > - stop apache
> > - stop lighttpd
> > - stop scgi
> > - make sure scgi still isn''t running (it sometimes does) - if
it
> > is, kill it
> > - delete all session files (they often cause permission errors)
> > - start scgi
> > - start apache
> > - start lighttpd
> > - cross fingers (since this process doesn''t always work and I
have
> > to repeat it)
> >
> > When the above doesn''t work, I get ''500''
errors on Rails'' pages,
> > ''Web server may possibly be
> > configured wrong.'' in scgi''s log, nothing in
Rails'' logs, and only
> > proxy connection errors in
> > apache''s logs if it can''t connect.
> >
> > Is there an easier and less problematic way?
> >
> > thanks
> > csn
> >
> 
> I don''t think you should need to restart apache or lighttpd. Since
> scgi lets you rails app run on its own outside the webserver you  
> should just stop and restart your scgi processes. Thats one of the  
> advantages to scgi.
That''s what I would have thought too, but after I do
''scgi_ctrl restart'' I get ''500 internal
error'' errors on my Rails'' pages. Restarting Lighttpd often
fixes it, but sometimes I have to go
through the complete aforementioned process.

csn



> 
> 
> Cheers-
> 
> -Ezra Zygmuntowicz
> Yakima Herald-Republic
> WebMaster
> http://yakimaherald.com
> 509-577-7732
> ezra-gdxLOakOTQ9oetBuM9ipNAC/G2K4zDHf@public.gmane.org
> 
> 
> 
> _______________________________________________
> Rails mailing list
> Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
> 


		
__________________________________________ 
Yahoo! DSL – Something to write home about. 
Just $16.99/mo. or less. 
dsl.yahoo.com

CSN wrote:
> --- Ezra Zygmuntowicz <ezra-SD1UcFUIF7QjH4SCZfkgnw@public.gmane.org>
wrote:
>> I don''t think you should need to restart apache or lighttpd.
Since
>> scgi lets you rails app run on its own outside the webserver you  
>> should just stop and restart your scgi processes. Thats one of the  
>> advantages to scgi.
> 
> That''s what I would have thought too, but after I do
''scgi_ctrl restart'' I get ''500 internal
> error'' errors on my Rails'' pages. Restarting Lighttpd
often fixes it, but sometimes I have to go
> through the complete aforementioned process.

Wow... maybe I won''t be looking into running SCGI, if that''s
the case. I run my
production apps under Lighttpd - FastCGI. There''s an Apache proxy in
front, but
I only need to restart it when I change the httpd.conf file.

For my Rails apps (0.14.3), all I need to do, to reload the code, is:

sudo ./script/process/reaper

And it just works. I suppose if SCGI gets to this stage, I might consider 
switching, but I''m not sure I really see the point. FastCGI under
Lighttpd is
fast and it works.

Incidentally, if a FastCGI guru out there knows how to run my FastCGI listeners 
as a user other than root, so I don''t need to sudo the reaper command,
that
would be a real help. Especially when I start hosting apps for multiple users on
my server. Or is the trick to run multiple Lighttpd instances, each as a 
separate user, since Lighttpd launches the FastCGI listeners...

-Brian

Try this:

1)  set the "disable-time" => 0 inside your SCGI lighttpd config
(same
place you put host and port).
2)  restart lighttpd.
3)  shutdown your scgi.
4)  Hit the web page.

Now, if you get a 500 error, that''s good.  Then do this:

1)  Start scgi, *right away hit the page*.

You should get the page back immediately.

Do this about 10 times until you''re confident that yes, lighttpd has
now
*stopped* putting a ridiculous 60 second death time on your scgi (and
fastcgi) processor.

Once you do this, you can restart any of the three servers independently. 
Extra geek points for making a special 500 page that tell people the
server is down temporarily.

The cause?  As mentioned before lighttpd will disable a processor (scgi or
fcgi) if it is down for a default time of 60 seconds.  Since this means
that a quick restart of the scgi process could knock your app out for 60
seconds, this turns out to be an incredibly bad design choice.

Another alternative I''ve toyed with is "scgi flopping". 
Haven''t
implemented it, but the idea is that you''d run a cluster like normal of
say 3 processors.  The "flop" command would do the following:

1)  Take down processors 1 and 2 and wait for them to exit.  Lighttpd sees
that those are down, so starts redirecting all traffic to #3.
2)  Lighttpd disables them for 60 seconds like normal, so you now have
about a minute to get them back up.  The flop command would restart #1,
and #2.
3)  Once #1 and #2 were up, it would shutdown #3.  Lighttpd disables #3
for 60 seconds and starts sending requests to #1 and #2.  Tricky part here
is figuring out when lighttpd has enabled #1 and #2.  Probably look at the
log files.
4)  Finally, flop would then restart #3 with the new code and you''d be
back in business.

Again, this is ultra freaking complicated, but would probably give you a
very very graceful restart to a new code base.

Anyway, try the trick above.

Zed A. Shaw
http://www.zedshaw.com/
> I''ve never used the exact setup you have, but generally when you
need
> to perform the same tasks over and over it''s nice to write a
single
> script that performs all the commands.  Then you can just call
>
> # ./uber-restart-script
>
> and everything is taken care of.
>
> Pat
>
>
> On 12/1/05, CSN
<cool_screen_name90001-/E1597aS9LQAvxtiuMwx3w@public.gmane.org> wrote:
>> I''ve managed to get Rails working using
Apache->Lighttpd->SCGI.
>> Restarting everything is really
>> difficult. This appears to be the process:
>>
>> - stop apache
>> - stop lighttpd
>> - stop scgi
>> - make sure scgi still isn''t running (it sometimes does) - if
it is,
>> kill it
>> - delete all session files (they often cause permission errors)
>> - start scgi
>> - start apache
>> - start lighttpd
>> - cross fingers (since this process doesn''t always work and I
have to
>> repeat it)
>>
>> When the above doesn''t work, I get ''500''
errors on Rails'' pages, ''Web
>> server may possibly be
>> configured wrong.'' in scgi''s log, nothing in
Rails'' logs, and only proxy
>> connection errors in
>> apache''s logs if it can''t connect.
>>
>> Is there an easier and less problematic way?
>>
>> thanks
>> csn
>>
>>
>>
>>
>>
>> __________________________________
>> Start your day with Yahoo! - Make it your home page!
>> http://www.yahoo.com/r/hs
>> _______________________________________________
>> Rails mailing list
>> Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org
>> http://lists.rubyonrails.org/mailman/listinfo/rails
>>
> _______________________________________________
> Rails mailing list
> Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
>

On Thu, 01 Dec 2005 17:08:45 -0500
"Brian V. Hughes"
<brianvh-ilmOVS5JQ6Xj7r8U7pfrKh2eb7JE58TQ@public.gmane.org> wrote:
> CSN wrote:
> 
> Wow... maybe I won''t be looking into running SCGI, if
that''s the
> case. I run my production apps under Lighttpd - FastCGI. There''s
an
> Apache proxy in front, but I only need to restart it when I change
> the httpd.conf file.
The problem is lighttpd *not* SCGI.  Read my message to this thread
explaining what to do.

Zed

--- zedshaw-dd7LMGGEL7NBDgjK7y7TUQ@public.gmane.org wrote:
> Try this:
> 
> 1)  set the "disable-time" => 0 inside your SCGI lighttpd
config (same
> place you put host and port).
> 2)  restart lighttpd.
> 3)  shutdown your scgi.
> 4)  Hit the web page.
> 
> Now, if you get a 500 error, that''s good.  Then do this:
> 
> 1)  Start scgi, *right away hit the page*.
> 
> You should get the page back immediately.
> 
> Do this about 10 times until you''re confident that yes, lighttpd
has now
> *stopped* putting a ridiculous 60 second death time on your scgi (and
> fastcgi) processor.
> 
> Once you do this, you can restart any of the three servers independently. 
> Extra geek points for making a special 500 page that tell people the
> server is down temporarily.
> 
> The cause?  As mentioned before lighttpd will disable a processor (scgi or
> fcgi) if it is down for a default time of 60 seconds.  Since this means
> that a quick restart of the scgi process could knock your app out for 60
> seconds, this turns out to be an incredibly bad design choice.

I was wondering what all the ''Immediate shutdown since nobody is
connected.'' messages meant in
scgi''s log.

Is this the correct syntax?

$HTTP["host"] =~ "www\.mydomain\.com:8000$" {
  server.document-root = "/var/www/html/rails/public/"
  disable-time = 0
}

I still get ''500'' errors after restarting scgi (Lighttpd was
restarted after changing its config),
but you''re right - it takes about a minute then pages load again.


thanks
csn


		
__________________________________________ 
Yahoo! DSL – Something to write home about. 
Just $16.99/mo. or less. 
dsl.yahoo.com

On 12/4/05, CSN
<cool_screen_name90001-/E1597aS9LQAvxtiuMwx3w@public.gmane.org>
wrote:
>
> Is this the correct syntax?
>
> $HTTP["host"] =~ "www\.mydomain\.com:8000$" {
>   server.document-root = "/var/www/html/rails/public/"
>   disable-time = 0
> }
I believe it should be

scgi.server = ( "dispatch.scgi" => ((
  "host" => "127.0.0.1",
  "port" => 9999,
  "check-local" => "disable",
  "disable-time" => 0
 )) )

(someone  please correct me if i''m wrong)

martin

On 12/1/05, Brian V. Hughes
<brianvh-ilmOVS5JQ6Xj7r8U7pfrKh2eb7JE58TQ@public.gmane.org>
wrote:
> CSN wrote:
> > --- Ezra Zygmuntowicz
<ezra-SD1UcFUIF7QjH4SCZfkgnw@public.gmane.org> wrote:
> >> I don''t think you should need to restart apache or
lighttpd. Since
> >> scgi lets you rails app run on its own outside the webserver you
> >> should just stop and restart your scgi processes. Thats one of the
> >> advantages to scgi.
> >
> > That''s what I would have thought too, but after I do
''scgi_ctrl restart'' I get ''500 internal
> > error'' errors on my Rails'' pages. Restarting
Lighttpd often fixes it, but sometimes I have to go
> > through the complete aforementioned process.
>
>
> Wow... maybe I won''t be looking into running SCGI, if
that''s the case. I run my
> production apps under Lighttpd - FastCGI. There''s an Apache proxy
in front, but
> I only need to restart it when I change the httpd.conf file.
>
> For my Rails apps (0.14.3), all I need to do, to reload the code, is:
>
> sudo ./script/process/reaper
>
> And it just works. I suppose if SCGI gets to this stage, I might consider
> switching, but I''m not sure I really see the point. FastCGI under
Lighttpd is
> fast and it works.
>
> Incidentally, if a FastCGI guru out there knows how to run my FastCGI
listeners
> as a user other than root, so I don''t need to sudo the reaper
command, that
> would be a real help. Especially when I start hosting apps for multiple
users on
> my server. Or is the trick to run multiple Lighttpd instances, each as a
> separate user, since Lighttpd launches the FastCGI listeners...
Or, can somone tell me how they update their production code on
TextDrive using lighttpd + fcgi?

I''m just killing the lighttpd server and restarting it, since I
didn''t
know about this script/process/reaper thing.

On Dec 7, 2005, at 3:42 PM, Joe Van Dyk wrote:
> On 12/1/05, Brian V. Hughes
<brianvh-ilmOVS5JQ6Xj7r8U7pfrKh2eb7JE58TQ@public.gmane.org> wrote:
>> CSN wrote:
>>> --- Ezra Zygmuntowicz
<ezra-SD1UcFUIF7QjH4SCZfkgnw@public.gmane.org> wrote:
>>>> I don''t think you should need to restart apache or
lighttpd. Since
>>>> scgi lets you rails app run on its own outside the webserver
you
>>>> should just stop and restart your scgi processes. Thats one of
the
>>>> advantages to scgi.
>>>
>>> That''s what I would have thought too, but after I do
''scgi_ctrl
>>> restart'' I get ''500 internal
>>> error'' errors on my Rails'' pages. Restarting
Lighttpd often fixes
>>> it, but sometimes I have to go
>>> through the complete aforementioned process.
>>
>>
>> Wow... maybe I won''t be looking into running SCGI, if
that''s the
>> case. I run my
>> production apps under Lighttpd - FastCGI. There''s an Apache
proxy
>> in front, but
>> I only need to restart it when I change the httpd.conf file.
>>
>> For my Rails apps (0.14.3), all I need to do, to reload the code, is:
>>
>> sudo ./script/process/reaper
>>
>> And it just works. I suppose if SCGI gets to this stage, I might  
>> consider
>> switching, but I''m not sure I really see the point. FastCGI
under
>> Lighttpd is
>> fast and it works.
>>
>> Incidentally, if a FastCGI guru out there knows how to run my  
>> FastCGI listeners
>> as a user other than root, so I don''t need to sudo the reaper
>> command, that
>> would be a real help. Especially when I start hosting apps for  
>> multiple users on
>> my server. Or is the trick to run multiple Lighttpd instances,  
>> each as a
>> separate user, since Lighttpd launches the FastCGI listeners...
>
> Or, can somone tell me how they update their production code on
> TextDrive using lighttpd + fcgi?
>
> I''m just killing the lighttpd server and restarting it, since I
didn''t
> know about this script/process/reaper thing.


#!/usr/local/bin/ruby


pstab = `ps axww`

def kill_proc(line)
   its = line.strip.split(/\s+/)
   pid = its[0]
   puts "KILLING #{line}"
   `kill -9 #{pid}`
   sleep(1)
end

pstab.scan(/lighttpd -f/) do |lighty|
   kill_proc lighty
end

pstab.scan(/^.*dispatch\.fcgi\s*$/) do |line|
   kill_proc line
end

if pstab =~ /dispatch\.fcgi/
   puts "Error, zombie dispatch.fcgi''s still lurking."
else
   puts "Rogue dispatch.fcgi''s cleared, starting lighty!"
   `/usr/local/sbin/lighttpd -f /home/USERNAME/lighttpd/lighttpd.conf`
end

	That will kill lighty, then kill the fcgi''s. It will then check to  
make sure the fcgi''s are dead and if they are it will restart lighty  
otherwise it will warn you of zombie fcgi''s and you should run it  
again. This seems to work good on txtdrv. You will need to edit it  
accordingly if you have more than one lighty instance running. Also  
script/reaper works on textdrive and is another way of restarting a  
production rails app.


Cheers-

-Ezra Zygmuntowicz
Yakima Herald-Republic
WebMaster
http://yakimaherald.com
509-577-7732
ezra-gdxLOakOTQ9oetBuM9ipNAC/G2K4zDHf@public.gmane.org

--- Martin DeMello <martindemello-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> On 12/4/05, CSN
<cool_screen_name90001-/E1597aS9LQAvxtiuMwx3w@public.gmane.org> wrote:
> >
> > Is this the correct syntax?
> >
> > $HTTP["host"] =~ "www\.mydomain\.com:8000$" {
> >   server.document-root = "/var/www/html/rails/public/"
> >   disable-time = 0
> > }
> 
> I believe it should be
> 
> scgi.server = ( "dispatch.scgi" => ((
>   "host" => "127.0.0.1",
>   "port" => 9999,
>   "check-local" => "disable",
>   "disable-time" => 0
>  )) )
> 
> (someone  please correct me if i''m wrong)
Ah... that appears to do the trick!

I''ve been running SCGI+Lighttpd+Apache in production for over a week
now. Aside from Lighty dying
a couple times (for unknown reasons - perhaps server load), the setup has worked
fine, and seems
to perform a lot better than my old setup (with no small thanks to
Rails'' caching, very likely).
Anybody know how to make Lighty automatically restart if it dies?

thanks
csn

> 
> martin

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com

On 12/7/05, Ezra Zygmuntowicz
<ezra-SD1UcFUIF7QjH4SCZfkgnw@public.gmane.org>
wrote:
> Also
> script/reaper works on textdrive and is another way of restarting a
> production rails app.
Can someone post their Switchtower task that uses the reaper script to
restart Rails after a code update?  I can''t seem to get it to work (I
have to rewrite it as I don''t have sudo on TxD), and just running
script/process/reaper doesn''t seem to do the trick.