[Rails] Re: Re: Re: Database Concurrency Without a Web Server?

Don Stocks
Wed, 12 Jul 2006 20:41:59 -0000

Ezra Zygmuntowicz wrote:

> 	Be very carefull with that setting. You do not want your main rails
> app running with that setting sewt to true or you will have major
> problems. When you use ActiveRecord by itself then you can

Thanks for the heads up, Ezra!  I gave it a test drive and so far I
don''t see any difference.

I''m working on a network monitoring system.  The background process is
a
Poller service that is used to monitor remote systems.  Each Poller
object has a Job object in a one-to-one relationship.  The poller is
quite busy running many simultaneous jobs.  So I need a high level of
concurrency.

In my tests, the script is instanciating 5000 ActiveRecord Job objects
per minute to poll remote systems / services.  I am using Francis
Cianfrocca''s EventMachine[1] to invoke the Job.run() method of each
object at a specific time within a 1 minute window.  After the job runs
it saves the results to the db (Job.save).  I''m trying to run 5000 of
these Jobs per minute.  Each Job only takes about 5-7 milliseconds to
comlete.

It just doesn''t look like script/runner is using db connection pooling.
And I suspect that the db queries are blocking, causing a serious
bottleneck.  One reason I think this is system CPU utilization is only
around 72% and MySQL isn''t even breathing hard.

I really need to figure out what''s causing the delays.  Many of my jobs
are running very late and I think it''s because my db access is being
serialized.

Any suggestions would be greatly appreciated!

 - Don


[1]  http://rubyforge.org/projects/eventmachine/