Backgroundrb devel - Aug 2008 - enqueued job polling configurability

I don''t know if this is where I should be submitting patches, but
it''s
also an idea/feature request, or something.

It bugs me that every worker polls the database every 5 seconds - If I  
wanted something done quickly, I''d just call it with the async prefix,
no? If I''m willing to offload it to 5 seconds later, maybe I''m
willing
to let it sit for 10, 20, maybe a full minute before being run.  
Anyways, this would make that configurable.

line 136 of vendor/plugins/backgroundrb/server/lib/meta_worker.rb :
-  add_periodic_timer(5) { check_for_enqueued_tasks }
+ add_periodic_timer(BDRB_CONFIG[:backgroundrb][:enq_poll_time].nil? ?  
5 : BDRB_CONFIG[:backgroundrb][:enq_poll_time])  
{ check_for_enqueued_tasks }

Same functionality, but then you could say

:backgroundrb:
     :enq_poll_time: 30

And enqueued tasks would poll a lot less. I guess it''s really not that
big of a hit on the database, it just bugs me for some reason. Also,  
my app in particular might have lots and lots of workers, depending on  
how I design the backgroundrb stuff, and 30 workers polling every 5  
seconds would be a bit much.

-Woody

Hello,

I have finished this exact type of thing (plus the ability to disable it all
together) and pushed to Github last night, and sent a pull request to
gnufied.

I''m waiting for him to merge it in to his copy (he said he was going to
about 8 hours or so ago, but I guess something came up).

Regards
Kieran


On Fri, Aug 15, 2008 at 7:10 AM, Woody Peterson
<woody at crystalcommerce.com>wrote:
> I don''t know if this is where I should be submitting patches, but
it''s also
> an idea/feature request, or something.
>
> It bugs me that every worker polls the database every 5 seconds - If I
> wanted something done quickly, I''d just call it with the async
prefix, no?
> If I''m willing to offload it to 5 seconds later, maybe
I''m willing to let it
> sit for 10, 20, maybe a full minute before being run. Anyways, this would
> make that configurable.
>
> line 136 of vendor/plugins/backgroundrb/server/lib/meta_worker.rb :
> -  add_periodic_timer(5) { check_for_enqueued_tasks }
> + add_periodic_timer(BDRB_CONFIG[:backgroundrb][:enq_poll_time].nil? ? 5 :
> BDRB_CONFIG[:backgroundrb][:enq_poll_time]) { check_for_enqueued_tasks }
>
> Same functionality, but then you could say
>
> :backgroundrb:
>    :enq_poll_time: 30
>
> And enqueued tasks would poll a lot less. I guess it''s really not
that big
> of a hit on the database, it just bugs me for some reason. Also, my app in
> particular might have lots and lots of workers, depending on how I design
> the backgroundrb stuff, and 30 workers polling every 5 seconds would be a
> bit much.
>
> -Woody
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080815/4e2cac59/attachment.html>

> On Fri, Aug 15, 2008 at 7:10 AM, Woody Peterson 
> <woody at crystalcommerce.com <mailto:woody at
crystalcommerce.com>> wrote:
>
>     I don''t know if this is where I should be submitting patches,
but
>     it''s also an idea/feature request, or something.
>
>     It bugs me that every worker polls the database every 5 seconds -
>     If I wanted something done quickly, I''d just call it with the
>     async prefix, no? If I''m willing to offload it to 5 seconds
later,
>     maybe I''m willing to let it sit for 10, 20, maybe a full
minute
>     before being run. Anyways, this would make that configurable.
>

What needs to be changed with the queries to ensure they drop into query 
cache?

I think the rough rules for mysql are:

- any updates/deletes/inserts on that table trash the entire cache 
(separate status updates into a separate table if these cause query 
cache to be purged too regularly)
- no column level privs or caching is bypassed
- no queries with volatile functions, eg "NOW()"

Possibly the database cost is extremely small if it can regularly hit 
the query cache?

Ed W
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080814/3e1bc438/attachment.html>