Hi, I''m running Starling and Workling to process background jobs. I''ve four instances of starling running on the same server, and I''m trying to get Workling to use them all. However, all of the jobs seem to be queued up and carried out one by one, instead of up to four being process simultaneously. As a result, I''ve got a very long job queue which is taking a long time to process, and I''d like to reduce it. I''ve set :multiple to true in script/workling_client.rb, but this doesn''t seem to help. I''m very inexperienced with both Starling and Workling, so it''s quite possible I''m either missing something simple, or approaching this the wrong way entirely... Any suggestions or insight much appreciated! best -- Posted via http://www.ruby-forum.com/. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
I actually just was in the same situation. I also had thought multiple Starling instances would help things, but it didn''t. The bottleneck was Workling, and I imagine it is the same for you. Despite the lack of documentation, it turns out to be extremely easy to run several instances: 1) In script/workling_client.rb, change :multiple from false to true as you already have, so the Daemon knows to allow for multiple instances 2) Just start more instances by calling "script/workling_client start" several times. It will create workling0.pid, workling1.pid, etc. I''m not sure how to properly setup god.rb to monitor the multiple instances so if you do please let me know :) FYI, you could also create multiple worker classes (my_worker.rb, another_worker.rb etc), and each Workling instance would use a thread for each class. So if you split up your methods into several classes, it might help things even more. On Mar 25, 6:37 am, "Rob W." <rails-mailing-l...-ARtvInVfO7ksV2N9l4h3zg@public.gmane.org> wrote:> Hi, > I''m running Starling and Workling to process background jobs. I''ve four > instances of starling running on the same server, and I''m trying to get > Workling to use them all. However, all of the jobs seem to be queued up > and carried out one by one, instead of up to four being process > simultaneously. > > As a result, I''ve got a very long job queue which is taking a long time > to process, and I''d like to reduce it. > > I''ve set :multiple to true in script/workling_client.rb, but this > doesn''t seem to help. > > I''m very inexperienced with both Starling and Workling, so it''s quite > possible I''m either missing something simple, or approaching this the > wrong way entirely... > > Any suggestions or insight much appreciated! > > best > -- > Posted viahttp://www.ruby-forum.com/.--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
PS. Here is an extremely helpful series of blog posts for starling and workling: http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/ On Mar 29, 3:49 pm, lampy <lampligh...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> I actually just was in the same situation. I also had thought > multiple Starling instances would help things, but it didn''t. The > bottleneck was Workling, and I imagine it is the same for you. > Despite the lack of documentation, it turns out to be extremely easy > to run several instances: > > 1) In script/workling_client.rb, change :multiple from false to true > as you already have, so the Daemon knows to allow for multiple > instances > 2) Just start more instances by calling "script/workling_client start" > several times. It will create workling0.pid, workling1.pid, etc. I''m > not sure how to properly setup god.rb to monitor the multiple > instances so if you do please let me know :) > > FYI, you could also create multiple worker classes (my_worker.rb, > another_worker.rb etc), and each Workling instance would use a thread > for each class. So if you split up your methods into several classes, > it might help things even more. > > On Mar 25, 6:37 am, "Rob W." <rails-mailing-l...-ARtvInVfO7ksV2N9l4h3zg@public.gmane.org> wrote: > > > > > Hi, > > I''m running Starling and Workling to process background jobs. I''ve four > > instances of starling running on the same server, and I''m trying to get > > Workling to use them all. However, all of the jobs seem to be queued up > > and carried out one by one, instead of up to four being process > > simultaneously. > > > As a result, I''ve got a very long job queue which is taking a long time > > to process, and I''d like to reduce it. > > > I''ve set :multiple to true in script/workling_client.rb, but this > > doesn''t seem to help. > > > I''m very inexperienced with both Starling and Workling, so it''s quite > > possible I''m either missing something simple, or approaching this the > > wrong way entirely... > > > Any suggestions or insight much appreciated! > > > best > > -- > > Posted viahttp://www.ruby-forum.com/.--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
I have modified my workling_client script to better handle multiple
worklings. This approach gives you full control over the workling
process names and pid files, which you can then use to cleanly manage
the worklings using God or Monit.
See below for my (admittedly hacky) solution:
#!/usr/bin/env ruby
require ''rubygems''
require ''daemons''
# This is a bit hacky, but we don''t have a clean way to hook into the
parsed
# options, as Daemons is managing these internally.
number = 0
for i in 0..ARGV.length
if ARGV[i] == ''--number''
number = ARGV[i + 1]
2.times {ARGV.delete_at(i)}
end
end
puts "Workling Number: #{number}"
workling = File.join(File.dirname(__FILE__), ''..'',
''vendor'',
''plugins'', ''workling'',
''script'', ''listen.rb'')
options = {
:app_name => "workling-#{number}-client",
:ARGV => ARGV,
:dir_mode => :normal,
:dir => File.join(File.dirname(__FILE__), ''..'',
''log''),
:log_output => true,
:multiple => false,
:backtrace => true,
:monitor => false
}
Daemons.run(workling, options)
On Mar 29, 12:57 pm, lampy
<lampligh...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:> PS. Here is an extremely helpful series of blog posts for starling
> and
workling:http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/
>
> On Mar 29, 3:49 pm, lampy
<lampligh...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> > I actually just was in the same situation. I also had thought
> > multiple Starling instances would help things, but it didn''t.
The
> > bottleneck was Workling, and I imagine it is the same for you.
> > Despite the lack of documentation, it turns out to be extremely easy
> > to run several instances:
>
> > 1) In script/workling_client.rb, change :multiple from false to true
> > as you already have, so the Daemon knows to allow for multiple
> > instances
> > 2) Just start more instances by calling "script/workling_client
start"
> > several times. It will create workling0.pid, workling1.pid, etc.
I''m
> > not sure how to properly setup god.rb to monitor the multiple
> > instances so if you do please let me know :)
>
> > FYI, you could also create multiple worker classes (my_worker.rb,
> > another_worker.rb etc), and each Workling instance would use a thread
> > for each class. So if you split up your methods into several classes,
> > it might help things even more.
>
> > On Mar 25, 6:37 am, "Rob W."
<rails-mailing-l...-ARtvInVfO7ksV2N9l4h3zg@public.gmane.org> wrote:
>
> > > Hi,
> > > I''m running Starling and Workling to process background
jobs. I''ve four
> > > instances of starling running on the same server, and
I''m trying to get
> > > Workling to use them all. However, all of the jobs seem to be
queued up
> > > and carried out one by one, instead of up to four being process
> > > simultaneously.
>
> > > As a result, I''ve got a very long job queue which is
taking a long time
> > > to process, and I''d like to reduce it.
>
> > > I''ve set :multiple to true in script/workling_client.rb,
but this
> > > doesn''t seem to help.
>
> > > I''m very inexperienced with both Starling and Workling,
so it''s quite
> > > possible I''m either missing something simple, or
approaching this the
> > > wrong way entirely...
>
> > > Any suggestions or insight much appreciated!
>
> > > best
> > > --
> > > Posted viahttp://www.ruby-forum.com/.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---