Greetings, We are currently working on offloading some long running tasks to backgroundrb, and we are using the thread_pool.defer method. Our intent originally was to have one worker running, and have it process requests as they came in in threads, so that we don''t have a lot of these workers running concurrently. It seems in our testing that backgroundrb is not closing these worker threads as they finish, and will eventually stop queueing new requests with the message: FATAL: sorry, too many clients already. Since we intend to open just one worker, and have it queue and process requests as they come in inside the thread pool, how do we close these threads once we are finished? Or am I misunderstanding how this model should work, and we should be instantiating a new worker per request? Here is the code we are using: http://pastie.org/private/jnyues8lwlhvoqbmqd8w Thanks, dan
On Thu, Jun 26, 2008 at 3:03 AM, Dan Herrera <dan at revelationglobal.com> wrote:> Greetings, > > We are currently working on offloading some long running tasks to > backgroundrb, and we are using the thread_pool.defer method. Our > intent originally was to have one worker running, and have it process > requests as they came in in threads, so that we don''t have a lot of > these workers running concurrently. > > It seems in our testing that backgroundrb is not closing these worker > threads as they finish, and will eventually stop queueing new requests > with the message: FATAL: sorry, too many clients already. > > Since we intend to open just one worker, and have it queue and process > requests as they come in inside the thread pool, how do we close these > threads once we are finished? Or am I misunderstanding how this model > should work, and we should be instantiating a new worker per request? > > Here is the code we are using: > > http://pastie.org/private/jnyues8lwlhvoqbmqd8w >Those worker threads are always active in thread pool ( or rather waiting for new tasks in queue object, enqueued via #defer method) and hence there is no point in stopping them, once they are done, since if no work is there in queue, they will anyways sleep. On which line you are getting the error? Can you paste full exception? Could be some issue with Queue class.
On Wed, Jun 25, 2008 at 7:35 PM, hemant <gethemant at gmail.com> wrote:> On Thu, Jun 26, 2008 at 3:03 AM, Dan Herrera <dan at revelationglobal.com> wrote: >> Greetings, >> >> We are currently working on offloading some long running tasks to >> backgroundrb, and we are using the thread_pool.defer method. Our >> intent originally was to have one worker running, and have it process >> requests as they came in in threads, so that we don''t have a lot of >> these workers running concurrently. >> >> It seems in our testing that backgroundrb is not closing these worker >> threads as they finish, and will eventually stop queueing new requests >> with the message: FATAL: sorry, too many clients already. >> >> Since we intend to open just one worker, and have it queue and process >> requests as they come in inside the thread pool, how do we close these >> threads once we are finished? Or am I misunderstanding how this model >> should work, and we should be instantiating a new worker per request? >> >> Here is the code we are using: >> >> http://pastie.org/private/jnyues8lwlhvoqbmqd8w >> > > Those worker threads are always active in thread pool ( or rather > waiting for new tasks in queue object, enqueued via #defer method) and > hence there is no point in stopping them, once they are done, since if > no work is there in queue, they will anyways sleep. > > On which line you are getting the error? Can you paste full exception? > Could be some issue with Queue class.Here is the full trace: http://pastie.org/222713 We get this error when running the test several times in a row. It could be that our test is not properly closing the connection to backgroundrb. The test reports that the error occurs in the before(:all) block in an RSpec. Thanks for your help! dan
> > http://pastie.org/222713 > > We get this error when running the test several times in a row. It > could be that our test is not properly closing the connection to > backgroundrb. The test reports that the error occurs in the > before(:all) block in an RSpec. > > Thanks for your help! > > dan > > Hi Dan,I''m a bit late to the party on this one, and I''m afraid I can''t get in to pastie at the moment, but I''ve just had a similar-sounding problem myself. If you''re doing anything with the model in your worker the error is probably caused by the sql connection not being released when the thread is complete. Try adding: ActiveRecord::Base.verify_active_connections! to the end of your worker method (and/or upping the number of concurrent connections available on whatever sql server you are using). Best, Todd -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080811/03b3a360/attachment.html>