I want to take a moment to talk about the notion of a queue abstraction layer in Rails and put a stake in the ground with my opinions on the topic while at the same time getting some feedback. My understanding is this: at the moment ActiveWorker (which can be seen in this repo: http://github.com/joshbuddy/rails ) is the current track for how to integrate workers (and queues to an extent). There''s nothing wrong with this approach, however my position is that workers != queues, and I don''t necessarily want to define my workers in the context of Rails, yet I do want an abstraction layer for queues. The current design of ActiveWorker has the queues essentially as adapters, similar to how ActiveRecord works, so I can understand leaving this as part of ActiveWorker, but I do wonder if it''s a good idea to couple these things together. Personally I don''t think it is but then again others may disagree. Here''s my proposal: there should be a queue abstraction layer (call it ActiveQueue if you like) with the following high-level requirements: * It should provide a generic interface to queues (messages queues and publish/subscribe) * It should provide a standard interface for drivers to implement and should provide a few basic driver implementations * It should allow other drivers to be added in through simple configuration (config/queue.yml) * It should define an interface for serializing messages (and provide JSON and XML serialization as the default?) * It should NOT define how to handle work units (i.e. this is not for defining asynchronous workers) * It should be used under the hood by ActiveWorker I actually have a need for this right now and would be willing to implement it to scratch my own itch, but I didn''t want to do so without first getting some feedback from the core community to see if this is both interesting and useful. Thoughts? Sincerely, Anthony Eden -- GMU/IT d- s: a32 C++(++++)$ UL@ P--- L+(++) !E W+++$ !N o? K? w--- !O M++ V PS+ PE Y PGP t+ !5 X- R tv b++ DI+ D++ G- e++ h---- r+++ y++++** http://anthony.mp -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com. To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=.
+1 on the concept ... I feel that Rails has been without this for a while. With the recent surge of activity on queue services, I think this is an awesome suggestion ... it would be great to have a standard Rails-ish way of setting up multiple work queues. I think that a good starting point for where to look for inspiration is http://github.com/defunkt/resque#readme ... which has a ton of really good design choices. If there was a common interface to queues such as Resque that could be abstracted in front of any choice of queue backend, I feel like it would be a huge win for Rails. Paul On Sun, Nov 22, 2009 at 4:57 PM, Anthony Eden <anthonyeden@gmail.com> wrote:> I want to take a moment to talk about the notion of a queue > abstraction layer in Rails and put a stake in the ground with my > opinions on the topic while at the same time getting some feedback. > > My understanding is this: at the moment ActiveWorker (which can be > seen in this repo: http://github.com/joshbuddy/rails ) is the current > track for how to integrate workers (and queues to an extent). There''s > nothing wrong with this approach, however my position is that workers > != queues, and I don''t necessarily want to define my workers in the > context of Rails, yet I do want an abstraction layer for queues. The > current design of ActiveWorker has the queues essentially as adapters, > similar to how ActiveRecord works, so I can understand leaving this as > part of ActiveWorker, but I do wonder if it''s a good idea to couple > these things together. Personally I don''t think it is but then again > others may disagree. > > Here''s my proposal: there should be a queue abstraction layer (call it > ActiveQueue if you like) with the following high-level requirements: > > * It should provide a generic interface to queues (messages queues and > publish/subscribe) > * It should provide a standard interface for drivers to implement and > should provide a few basic driver implementations > * It should allow other drivers to be added in through simple > configuration (config/queue.yml) > * It should define an interface for serializing messages (and provide > JSON and XML serialization as the default?) > * It should NOT define how to handle work units (i.e. this is not for > defining asynchronous workers) > * It should be used under the hood by ActiveWorker > > I actually have a need for this right now and would be willing to > implement it to scratch my own itch, but I didn''t want to do so > without first getting some feedback from the core community to see if > this is both interesting and useful. > > Thoughts? > > Sincerely, > Anthony Eden > > -- > GMU/IT d- s: a32 C++(++++)$ UL@ P--- L+(++) !E W+++$ !N o? K? w--- !O > M++ V PS+ PE Y PGP t+ !5 X- R tv b++ DI+ D++ G- e++ h---- r+++ y++++** > > http://anthony.mp > > -- > > You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. > To post to this group, send email to rubyonrails-core@googlegroups.com. > To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=. > > >-- Paul Campbell paul@rushedsunlight.com - - - - - - - - - - - - - - - - - - - blog http://www.pabcas.com twitter http://www.twitter.com/paulca github http://www.github.com/paulca phone +353 87 914 8162 - - - - - - - - - - - - - - - - - - - -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com. To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=.
+1 I would love to see the related scheduled job be part of this as well since some processes can only have a single instance running at a time. On Nov 22, 7:57 pm, Anthony Eden <anthonye...@gmail.com> wrote:> I want to take a moment to talk about the notion of a queue > abstraction layer in Rails and put a stake in the ground with my > opinions on the topic while at the same time getting some feedback. > > My understanding is this: at the moment ActiveWorker (which can be > seen in this repo:http://github.com/joshbuddy/rails) is the current > track for how to integrate workers (and queues to an extent). There''s > nothing wrong with this approach, however my position is that workers > != queues, and I don''t necessarily want to define my workers in the > context of Rails, yet I do want an abstraction layer for queues. The > current design of ActiveWorker has the queues essentially as adapters, > similar to how ActiveRecord works, so I can understand leaving this as > part of ActiveWorker, but I do wonder if it''s a good idea to couple > these things together. Personally I don''t think it is but then again > others may disagree. > > Here''s my proposal: there should be a queue abstraction layer (call it > ActiveQueue if you like) with the following high-level requirements: > > * It should provide a generic interface to queues (messages queues and > publish/subscribe) > * It should provide a standard interface for drivers to implement and > should provide a few basic driver implementations > * It should allow other drivers to be added in through simple > configuration (config/queue.yml) > * It should define an interface for serializing messages (and provide > JSON and XML serialization as the default?) > * It should NOT define how to handle work units (i.e. this is not for > defining asynchronous workers) > * It should be used under the hood by ActiveWorker > > I actually have a need for this right now and would be willing to > implement it to scratch my own itch, but I didn''t want to do so > without first getting some feedback from the core community to see if > this is both interesting and useful. > > Thoughts? > > Sincerely, > Anthony Eden > > -- > GMU/IT d- s: a32 C++(++++)$ UL@ P--- L+(++) !E W+++$ !N o? K? w--- !O > M++ V PS+ PE Y PGP t+ !5 X- R tv b++ DI+ D++ G- e++ h---- r+++ y++++** > > http://anthony.mp-- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com. To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=.
Interesting ideas. I think it be "ActionQueue" since its not a model layer (not sure). Maybe both belong in the same "pack" similar to ActionPack. So we have some root package called "ActionBackgroundShit" and it contains AW and AQ. actionbackgroundshit/lib/action_worker/* actionbackgroundshit/lib/action_queue/* On Mon, Nov 23, 2009 at 5:50 AM, mbiker <andy.w.freeman@gmail.com> wrote:> +1 > > I would love to see the related scheduled job be part of this as well > since some processes can only have a single instance running at a > time. > > On Nov 22, 7:57 pm, Anthony Eden <anthonye...@gmail.com> wrote: >> I want to take a moment to talk about the notion of a queue >> abstraction layer in Rails and put a stake in the ground with my >> opinions on the topic while at the same time getting some feedback. >> >> My understanding is this: at the moment ActiveWorker (which can be >> seen in this repo:http://github.com/joshbuddy/rails) is the current >> track for how to integrate workers (and queues to an extent). There''s >> nothing wrong with this approach, however my position is that workers >> != queues, and I don''t necessarily want to define my workers in the >> context of Rails, yet I do want an abstraction layer for queues. The >> current design of ActiveWorker has the queues essentially as adapters, >> similar to how ActiveRecord works, so I can understand leaving this as >> part of ActiveWorker, but I do wonder if it''s a good idea to couple >> these things together. Personally I don''t think it is but then again >> others may disagree. >> >> Here''s my proposal: there should be a queue abstraction layer (call it >> ActiveQueue if you like) with the following high-level requirements: >> >> * It should provide a generic interface to queues (messages queues and >> publish/subscribe) >> * It should provide a standard interface for drivers to implement and >> should provide a few basic driver implementations >> * It should allow other drivers to be added in through simple >> configuration (config/queue.yml) >> * It should define an interface for serializing messages (and provide >> JSON and XML serialization as the default?) >> * It should NOT define how to handle work units (i.e. this is not for >> defining asynchronous workers) >> * It should be used under the hood by ActiveWorker >> >> I actually have a need for this right now and would be willing to >> implement it to scratch my own itch, but I didn''t want to do so >> without first getting some feedback from the core community to see if >> this is both interesting and useful. >> >> Thoughts? >> >> Sincerely, >> Anthony Eden >> >> -- >> GMU/IT d- s: a32 C++(++++)$ UL@ P--- L+(++) !E W+++$ !N o? K? w--- !O >> M++ V PS+ PE Y PGP t+ !5 X- R tv b++ DI+ D++ G- e++ h---- r+++ y++++** >> >> http://anthony.mp > > -- > > You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. > To post to this group, send email to rubyonrails-core@googlegroups.com. > To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=. > > >-- Joshua Peek -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com. To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=.
Queues != Model. Josh, you are right indeed. +1 from me for AW & AQ :-) Cheers, Amol On Nov 23, 2009, at 8:47 PM, Joshua Peek wrote:> Interesting ideas. > > I think it be "ActionQueue" since its not a model layer (not sure). > > Maybe both belong in the same "pack" similar to ActionPack. > > So we have some root package called "ActionBackgroundShit" and it > contains AW and AQ. > > actionbackgroundshit/lib/action_worker/* > actionbackgroundshit/lib/action_queue/* > > On Mon, Nov 23, 2009 at 5:50 AM, mbiker <andy.w.freeman@gmail.com> wrote: >> +1 >> >> I would love to see the related scheduled job be part of this as well >> since some processes can only have a single instance running at a >> time. >> >> On Nov 22, 7:57 pm, Anthony Eden <anthonye...@gmail.com> wrote: >>> I want to take a moment to talk about the notion of a queue >>> abstraction layer in Rails and put a stake in the ground with my >>> opinions on the topic while at the same time getting some feedback. >>> >>> My understanding is this: at the moment ActiveWorker (which can be >>> seen in this repo:http://github.com/joshbuddy/rails) is the current >>> track for how to integrate workers (and queues to an extent). There''s >>> nothing wrong with this approach, however my position is that workers >>> != queues, and I don''t necessarily want to define my workers in the >>> context of Rails, yet I do want an abstraction layer for queues. The >>> current design of ActiveWorker has the queues essentially as adapters, >>> similar to how ActiveRecord works, so I can understand leaving this as >>> part of ActiveWorker, but I do wonder if it''s a good idea to couple >>> these things together. Personally I don''t think it is but then again >>> others may disagree. >>> >>> Here''s my proposal: there should be a queue abstraction layer (call it >>> ActiveQueue if you like) with the following high-level requirements: >>> >>> * It should provide a generic interface to queues (messages queues and >>> publish/subscribe) >>> * It should provide a standard interface for drivers to implement and >>> should provide a few basic driver implementations >>> * It should allow other drivers to be added in through simple >>> configuration (config/queue.yml) >>> * It should define an interface for serializing messages (and provide >>> JSON and XML serialization as the default?) >>> * It should NOT define how to handle work units (i.e. this is not for >>> defining asynchronous workers) >>> * It should be used under the hood by ActiveWorker >>> >>> I actually have a need for this right now and would be willing to >>> implement it to scratch my own itch, but I didn''t want to do so >>> without first getting some feedback from the core community to see if >>> this is both interesting and useful. >>> >>> Thoughts? >>> >>> Sincerely, >>> Anthony Eden >>> >>> -- >>> GMU/IT d- s: a32 C++(++++)$ UL@ P--- L+(++) !E W+++$ !N o? K? w--- !O >>> M++ V PS+ PE Y PGP t+ !5 X- R tv b++ DI+ D++ G- e++ h---- r+++ y++++** >>> >>> http://anthony.mp >> >> -- >> >> You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. >> To post to this group, send email to rubyonrails-core@googlegroups.com. >> To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com. >> For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=. >> >> >> > > > > -- > Joshua Peek > > -- > > You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. > To post to this group, send email to rubyonrails-core@googlegroups.com. > To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=. > >-- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com. To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=.
Seems to me it''s analogous to caching from a service-oriented architecture pov - ActiveSupport::Queue like ActiveSupport::Cache ? Interface/implements approach? AMPQ-compliant interface? Starling? Also suggest that there may be different levels of the interface, starting with very simple transient push/pop all the way to full AMPQ. This would allow work around ActiveWorker to focus exclusively on worker challenges like polling, starvation, backoff, sleep, wake, deployment, running in context, etc. m On Mon, Nov 23, 2009 at 7:40 AM, Amol Hatwar <amol@hatwar.org> wrote:> Queues != Model. Josh, you are right indeed. +1 from me for AW & AQ :-) > > Cheers, > > Amol > > On Nov 23, 2009, at 8:47 PM, Joshua Peek wrote: > > > Interesting ideas. > > > > I think it be "ActionQueue" since its not a model layer (not sure). > > > > Maybe both belong in the same "pack" similar to ActionPack. > > > > So we have some root package called "ActionBackgroundShit" and it > > contains AW and AQ. > > > > actionbackgroundshit/lib/action_worker/* > > actionbackgroundshit/lib/action_queue/* > > > > On Mon, Nov 23, 2009 at 5:50 AM, mbiker <andy.w.freeman@gmail.com> > wrote: > >> +1 > >> > >> I would love to see the related scheduled job be part of this as well > >> since some processes can only have a single instance running at a > >> time. > >> > >> On Nov 22, 7:57 pm, Anthony Eden <anthonye...@gmail.com> wrote: > >>> I want to take a moment to talk about the notion of a queue > >>> abstraction layer in Rails and put a stake in the ground with my > >>> opinions on the topic while at the same time getting some feedback. > >>> > >>> My understanding is this: at the moment ActiveWorker (which can be > >>> seen in this repo:http://github.com/joshbuddy/rails) is the current > >>> track for how to integrate workers (and queues to an extent). There''s > >>> nothing wrong with this approach, however my position is that workers > >>> != queues, and I don''t necessarily want to define my workers in the > >>> context of Rails, yet I do want an abstraction layer for queues. The > >>> current design of ActiveWorker has the queues essentially as adapters, > >>> similar to how ActiveRecord works, so I can understand leaving this as > >>> part of ActiveWorker, but I do wonder if it''s a good idea to couple > >>> these things together. Personally I don''t think it is but then again > >>> others may disagree. > >>> > >>> Here''s my proposal: there should be a queue abstraction layer (call it > >>> ActiveQueue if you like) with the following high-level requirements: > >>> > >>> * It should provide a generic interface to queues (messages queues and > >>> publish/subscribe) > >>> * It should provide a standard interface for drivers to implement and > >>> should provide a few basic driver implementations > >>> * It should allow other drivers to be added in through simple > >>> configuration (config/queue.yml) > >>> * It should define an interface for serializing messages (and provide > >>> JSON and XML serialization as the default?) > >>> * It should NOT define how to handle work units (i.e. this is not for > >>> defining asynchronous workers) > >>> * It should be used under the hood by ActiveWorker > >>> > >>> I actually have a need for this right now and would be willing to > >>> implement it to scratch my own itch, but I didn''t want to do so > >>> without first getting some feedback from the core community to see if > >>> this is both interesting and useful. > >>> > >>> Thoughts? > >>> > >>> Sincerely, > >>> Anthony Eden > >>> > >>> -- > >>> GMU/IT d- s: a32 C++(++++)$ UL@ P--- L+(++) !E W+++$ !N o? K? w--- !O > >>> M++ V PS+ PE Y PGP t+ !5 X- R tv b++ DI+ D++ G- e++ h---- r+++ y++++** > >>> > >>> http://anthony.mp > >> > >> -- > >> > >> You received this message because you are subscribed to the Google > Groups "Ruby on Rails: Core" group. > >> To post to this group, send email to rubyonrails-core@googlegroups.com. > >> To unsubscribe from this group, send email to > rubyonrails-core+unsubscribe@googlegroups.com<rubyonrails-core%2Bunsubscribe@googlegroups.com> > . > >> For more options, visit this group at > http://groups.google.com/group/rubyonrails-core?hl=. > >> > >> > >> > > > > > > > > -- > > Joshua Peek > > > > -- > > > > You received this message because you are subscribed to the Google Groups > "Ruby on Rails: Core" group. > > To post to this group, send email to rubyonrails-core@googlegroups.com. > > To unsubscribe from this group, send email to > rubyonrails-core+unsubscribe@googlegroups.com<rubyonrails-core%2Bunsubscribe@googlegroups.com> > . > > For more options, visit this group at > http://groups.google.com/group/rubyonrails-core?hl=. > > > > > > -- > > You received this message because you are subscribed to the Google Groups > "Ruby on Rails: Core" group. > To post to this group, send email to rubyonrails-core@googlegroups.com. > To unsubscribe from this group, send email to > rubyonrails-core+unsubscribe@googlegroups.com<rubyonrails-core%2Bunsubscribe@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/rubyonrails-core?hl=. > > >-- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com. To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=.