[Mongrel] Mongrel Timeout for System Calls

Justin Diana
2009-09-04T19:45:36Z

I''ve got a problem with timing out a C RFC call.

Essentially, I''ve got code that initializes and executes an RFC to a
remote server that, depending on the server, could take a significant
amount of time to return a response.

To minimize end-user impact, I wrapped the call with a timeout of 20
seconds.

When I executed a test, it didn''t appear to do anything as the process
still took 120-130 seconds to complete.

After digging around a bit, I discovered this writeup about timeout and
system calls
(http://adamblog.heroku.com/past/2008/6/17/battling_wedged_mongrels_with_a/).

I immediately installed SystemTimer
(http://ph7spot.com/articles/system_timer) and tried again:

begin
  require ''system_timer''
  MyTimer = SystemTimer
rescue LoadError
  require ''timeout''
  MyTimer = Timeout
end

def invoke(rfc)
    MyTimer.timeout_after(20.seconds) do
      rfc.invoke
    end
    rescue Timeout::Error
      raise Timeout::Error, "RFC Call timed out after 20 seconds"
end

This time, it executed and waited the full length of time (120-130
seconds).  20 seconds AFTER it completed, I then saw the "RFC Call timed
out after 20 seconds" error written to the logs.

After some more investigation, I read about Terminator and the use of
new threads to kill stuck/long-running threads.

I installed it and updated my above block of code to use the Terminator
syntax.

This time when I ran it, a view of the process list showed that the new
thread got created with a timer of 20 seconds.  20 seconds into
processing, the thread showed as <Defunct> and nothing happened to the
existing thread... that is until it completed.  Once the process
finished running, the mongrel log shows that it was issued a restart
command and restarted.

What am I doing wrong?  There must be a way to timeout a system level C
API/RFC call...

Running Ruby 1.8.6 p114 on CentOS 5.1 64-bit with mongrel 1.1.5

Thanks in advance for anyone who can help!
--
Posted via http://www.ruby-forum.com/.