Mongrel users - May 2006 - Rails, ActionController::Base.allow_concurrency and mongrel

Hello all,

Are you aware of the rails ActionController::Base.allow_concurrency option ?
If this option is true, rails is able to process simultaneous multiples
actions.

It appears that rails is thread-safe (if the user application is), and
that this option is mostly unknown.
Webrick take care of this option in rails-1.1.2/lib/webrick_server.rb at
line 80 :
    REQUEST_MUTEX.lock unless ActionController::Base.allow_concurrency

I think that even if using multi-threading in mongrel would not increase
performance, it could dramatically decreases memory requirements for
rails apps (usefull for shared hosting) and makes configuration easyer
(proxy becomes optionnal)

Thanks :)

-- 
 Fran?ois SIMOND

Here is a rudimentary patch against mongrel-0.3.12.4, that basically works !
I could not imagine that it would be so easy  :) 

-- 
 Fran?ois SIMOND

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: patch
Url:
http://rubyforge.org/pipermail/mongrel-users/attachments/20060528/f1446434/attachment.bat

On Sun, 2006-05-28 at 19:38 +0200, Fran?ois SIMOND
wrote:
> Here is a rudimentary patch against mongrel-0.3.12.4, that basically works
!
> I could not imagine that it would be so easy  :) 
Here''s the things that blow up--even with this setting--unless
there''s
been some *major* improvements in how Rails'' was structured:

* Classes mysteriously "disappear" or aren''t loaded. 
Ruby''s dynamic
class loading and instance_eval is not thread safe, so when multiple
threads load rails actions, models, and helpers the Dispatcher throws
random exceptions.
* Model objects used in one thread show up in another thread.  Last time
this was because there were problems with how shared connections were
being used between the different threads.
* Requests mysteriously pick up other thread''s request input or output.
No idea why this was.  It''s really really weird.
* High load causes database connections to explode exponentially into
the thousands.  This is caused by use Thread.current[] to store the DB
connection, which causes the rails application to connect one time for
each thread.
* Once connected, the connection doesn''t go away, but the thread does.
This means that when each thread is created, connects, and then finishes
it''s response, the connection is kept around until the GC kills it.
This causes file descriptor leaks that eventually make the whole Mongrel
server stop functioning because the number of open files is greater than
the 1024 select() limit on most systems.
* Long running Mongrel servers start eating lots of memory or stop
running.  This is mostly caused by the connections per thread model not
being cleaned up, but even when this is cleaned out it''s still not good
enough.  Files, other thread, popen calls, fork, and many other
resources or notoriously poorly managed by nearly every Rails
programmer.

I started working on fixing these problems way back in the SCGI days,
but the task was so daunting that I just gave up.  So, yes, your little
patch is very easy.  It probably won''t work without a lot of extra
effort.

If you''re interested in doing some hack-o-tronic work to make this
function, then you''ll have to either rule out the above problems (with
more than one request with a browser) or find a way to implement fixes
outside of rails.  Some possible solutions:

1) Rip out anything left over in a Thread.current[] after the rails
request is done.
2) Find a way to mark what request/response/IO objects are assigned to
each thread, and then blow-up if they change during processing.  This
would most likely be a debugging option for people suspecting crossover.
3) Find a way to keep track of what a rails controller opens, and then
make sure it gets closed completely.  This''ll be a real realy pain in
the ass since people love to open crap at random and not clean it up.
4) Do a completely pre-loading when a rails application is started in
production mode so that all classes are properly loaded ahead of time
and you don''t have to worry about Ruby''s lack of thread safety
here.

If you''re up to the challenge, then give it a shot.  You''ll
want to
setup a test harness with your current code that thrashes the living
hell out of a real rails application for a long period of time and then
fix everything that comes up.


-- 
Zed A. Shaw
http://www.zedshaw.com/
http://mongrel.rubyforge.org/

Zed Shaw a ?crit :
> On Sun, 2006-05-28 at 19:38 +0200, Fran?ois SIMOND wrote:
>   
>> Here is a rudimentary patch against mongrel-0.3.12.4, that basically
works !
>> I could not imagine that it would be so easy  :) 
>>     
>
> Here''s the things that blow up--even with this setting--unless
there''s
> been some *major* improvements in how Rails'' was structured:
>   
Too bad.. i thought that you had already made it but i wanted to try,
encouraged by preliminar test results :)
> * Classes mysteriously "disappear" or aren''t loaded. 
Ruby''s dynamic
> class loading and instance_eval is not thread safe, so when multiple
> threads load rails actions, models, and helpers the Dispatcher throws
> random exceptions.
> * Model objects used in one thread show up in another thread.  Last time
> this was because there were problems with how shared connections were
> being used between the different threads.
> * Requests mysteriously pick up other thread''s request input or
output.
> No idea why this was.  It''s really really weird.
> * High load causes database connections to explode exponentially into
> the thousands.  This is caused by use Thread.current[] to store the DB
> connection, which causes the rails application to connect one time for
> each thread.
> * Once connected, the connection doesn''t go away, but the thread
does.
> This means that when each thread is created, connects, and then finishes
> it''s response, the connection is kept around until the GC kills
it.
> This causes file descriptor leaks that eventually make the whole Mongrel
> server stop functioning because the number of open files is greater than
> the 1024 select() limit on most systems.
> * Long running Mongrel servers start eating lots of memory or stop
> running.  This is mostly caused by the connections per thread model not
> being cleaned up, but even when this is cleaned out it''s still not
good
> enough.  Files, other thread, popen calls, fork, and many other
> resources or notoriously poorly managed by nearly every Rails
> programmer.
>   
For ActiveRecord related weirdness, i hope that it was adressed in 1.1.x
by recent AR allow_concurrency modifications and connection pool patch.
I neigher really have ideas for the wrong input or output.. Do you think
that these threads were incorrectly finished ? ( like the example in
http://groups.google.com/group/comp.lang.ruby/msg/0b79331e7d40e9b8
)
> I started working on fixing these problems way back in the SCGI days,
> but the task was so daunting that I just gave up.  So, yes, your little
> patch is very easy.  It probably won''t work without a lot of extra
> effort.
>   
Were your fixes integrated in rails trunk ?
> If you''re interested in doing some hack-o-tronic work to make this
> function, then you''ll have to either rule out the above problems
(with
> more than one request with a browser) or find a way to implement fixes
> outside of rails.  Some possible solutions:
>   
I love this word, "hack-o-tronic" :) but i do not know if
i''ll like to
do it..
> 1) Rip out anything left over in a Thread.current[] after the rails
> request is done.
>   
Like
Thread.current = nil
GC.start
? :)
> 2) Find a way to mark what request/response/IO objects are assigned to
> each thread, and then blow-up if they change during processing.  This
> would most likely be a debugging option for people suspecting crossover.
>   
Maybe it could be done by re-defining and logging each problematic
operation.
> 3) Find a way to keep track of what a rails controller opens, and then
> make sure it gets closed completely.  This''ll be a real realy pain
in
> the ass since people love to open crap at random and not clean it up.
>   
Right. That''s the reason why rails team has disabled allow_concurrency 
by default.
> 4) Do a completely pre-loading when a rails application is started in
> production mode so that all classes are properly loaded ahead of time
> and you don''t have to worry about Ruby''s lack of thread
safety here.
>   
An appropriate script could do that at mongrel rails startup, looking in
standard or configured paths, and loading every class
first.
> If you''re up to the challenge, then give it a shot. 
You''ll want to
> setup a test harness with your current code that thrashes the living
> hell out of a real rails application for a long period of time and then
> fix everything that comes up.
My primary goal is to find solutions for a rails app that i''m currently
developping (a free personnal and social photo gallery).
This application need some long-time processings and uploads, and will
be installed in shared hosting too.

I''ll maybe use BackgrounDRb for the image resize operations, but I
think
that i can build it better if i''m not much restricted in terms of time
of concurrential requests, so I''ll try !

Do you have testcases that I could use as a quickstart ?

And thank you Zed for a such detailled response !

-- 
 Francois Simond

Any thought for having a preforking mongrel, with a few features like
min_servers, max_servers, min_spare_servers, etc?

 - James

On Mon, 2006-05-29 at 12:06 -0700, James Moore wrote:
> Any thought for having a preforking mongrel, with a few features like
> min_servers, max_servers, min_spare_servers, etc?
> 
>  - James
Pre-forking is not reliable under ruby for some odd reason.  I had this
running with SCGI, but when the IO on the main socket comes in at higher
rates you get the data mixed up between the pre-forked processes.

Only way around it is to write a custom C program that then embeds Ruby
which then runs Mongrel, which is a lot more trouble than it''s worth
right now.

And, none of the preforking will ever work on win32 since windows
doesn''t have the same kind of fork semantics that Unix does.

Zed