mongrel unicorn - Nov 2012 - select(): Interrupted system call from curb when stopping unicorn

Hi,

We''ve just migrated one of our rails applications from nginx/passenger
running on REE 1.8.7 Ubuntu 8.04 to unicorn running on MRI 1.9.3 on
Ubuntu 10.04. The app makes a number of calls to internal services
using curb.

Our deployment script stops unicorn by sending SIGQUIT to the unicorn
master, sleeps for a few seconds to ensure that HAProxy has taken the
node out of service and then starts unicorn again.

However, since the migration, we''ve started seeing a few errors on
each deploy, when the workers receive the QUIT from the master:

 ERROR (  XXX)
{"exception":{"class":"RuntimeError","message":"select():
Interrupted system
call","backtrace":["/path/bundle/ruby/1.9.1/gems/curb-0.7.18/lib/curl/easy.rb:39:in
`perform''","/path/bundle/ruby/1.9.1/gems/curb-0.7.18/lib/curl/easy.rb:39:in
`perform''","/path/bundle/ruby/1.9.1/gems/songkick-transport-0.1.4/lib/songkick/transport/curb.rb:50:in
...

If we''re reading this correctly, curb is inside a select() call, gets
interrupted by the QUIT signal and doesn''t retry the select() and the
process terminates, causing an error for the unfortunate end user.

Our options for stopping our users seeing error pages seem to be:

1) Patch curb to retry the select() if errno = EINTR as per
https://github.com/taf2/curb/issues/117

We''re not confident enough in our ability to do this properly, but
it''s probably the most correct way to solve this.

2) Do something to make sure that requests have all completed before
sending the QUIT.

For example, we could change our deployment to make the health check
url that HAProxy polls start returning a non-200 return code, wait
until all connections have completed and then send the QUIT.

The main issue with this is that short of polling the HAProxy status
page, we''re not entirely sure how to verify that all connections have
completed.

Does anyone have any suggested solutions or comments?

Regards,
Graham

Graham Bleach <graham at darkskills.org.uk> wrote:
> Hi,
> 
> We''ve just migrated one of our rails applications from
nginx/passenger
> running on REE 1.8.7 Ubuntu 8.04 to unicorn running on MRI 1.9.3 on
> Ubuntu 10.04. The app makes a number of calls to internal services
> using curb.
> 
> Our deployment script stops unicorn by sending SIGQUIT to the unicorn
> master, sleeps for a few seconds to ensure that HAProxy has taken the
> node out of service and then starts unicorn again.
> 
> However, since the migration, we''ve started seeing a few errors on
> each deploy, when the workers receive the QUIT from the master:
> 
>  ERROR (  XXX)
{"exception":{"class":"RuntimeError","message":"select():
> Interrupted system
>
call","backtrace":["/path/bundle/ruby/1.9.1/gems/curb-0.7.18/lib/curl/easy.rb:39:in
>
`perform''","/path/bundle/ruby/1.9.1/gems/curb-0.7.18/lib/curl/easy.rb:39:in
>
`perform''","/path/bundle/ruby/1.9.1/gems/songkick-transport-0.1.4/lib/songkick/transport/curb.rb:50:in
> ...
> 
> If we''re reading this correctly, curb is inside a select() call,
gets
> interrupted by the QUIT signal and doesn''t retry the select() and
the
> process terminates, causing an error for the unfortunate end user.
> 
> Our options for stopping our users seeing error pages seem to be:
> 
> 1) Patch curb to retry the select() if errno = EINTR as per
> https://github.com/taf2/curb/issues/117
Taking a quick look at the curb source, I see no reason it implements
curb_select() itself instead of just using rb_thread_select() (or
rb_thread_fd_select(), the latter is recommended for new Rubies as
it handles high-numbered FDs better).

Any rb_thread*select() function implemented by Ruby is already aware of
multithreading and (for 1.9) calls rb_thread_blocking_region()
internally.

For now, you can probably just: #undef HAVE_RB_THREAD_BLOCKING_REGION

(Feel free to forward my comments to the curb folks, I don''t like
 logging into websites)
> We''re not confident enough in our ability to do this properly, but
> it''s probably the most correct way to solve this.
Practice :)