[Rails] Re: Re: Re: Database Concurrency Without a Web Server?

snacktime
Wed, 12 Jul 2006 21:30:08 -0000

On 7/12/06, Don Stocks <amillionhitpoints@yahoo.com>
wrote:
> Ezra Zygmuntowicz wrote:
>
> >       Be very carefull with that setting. You do not want your main
rails
> > app running with that setting sewt to true or you will have major
> > problems. When you use ActiveRecord by itself then you can
>
> Thanks for the heads up, Ezra!  I gave it a test drive and so far I
> don''t see any difference.
>
> I''m working on a network monitoring system.  The background
process is a
> Poller service that is used to monitor remote systems.  Each Poller
> object has a Job object in a one-to-one relationship.  The poller is
> quite busy running many simultaneous jobs.  So I need a high level of
> concurrency.
>
> In my tests, the script is instanciating 5000 ActiveRecord Job objects
> per minute to poll remote systems / services.  I am using Francis
> Cianfrocca''s EventMachine[1] to invoke the Job.run() method of
each
> object at a specific time within a 1 minute window.  After the job runs
> it saves the results to the db (Job.save).  I''m trying to run 5000
of
> these Jobs per minute.  Each Job only takes about 5-7 milliseconds to
> comlete.
>
> It just doesn''t look like script/runner is using db connection
pooling.
> And I suspect that the db queries are blocking, causing a serious
> bottleneck.  One reason I think this is system CPU utilization is only
> around 72% and MySQL isn''t even breathing hard.
>
> I really need to figure out what''s causing the delays.  Many of my
jobs
> are running very late and I think it''s because my db access is
being
> serialized.
>
> Any suggestions would be greatly appreciated!

If you are using non blocking IO, then everything needs to be non
blocking.  For example if you use an event framework of some type to
fire off network connections via net/http, it''s not going to work,
because the IO in net/http blocks.  Same for connection pooling.  You
really need to use an event framework that has handlers for everything
that is IO based, or just use threads.  I''ve used python''s
twisted
framework in the past, and it works very well.

Chris