Rails - Nov 2009 - Simultaneous long-running tasks; Rake, Daemons, Delayed::Jobs?

Hi everyone,

I have an application where an essential part is communicating with a
remote API via SOAP. I want to be able to run 3-4 of these tasks at
once, and one task might take anything from one minute to several
hours, depending on the size of the data being sent. The app relies
heavily on ActiveRecord fetching fetching data (in_batches) and
updating it once the response from the API has been received.

I''m using Ruby 1.8.6 and Rails 2.3.4.

In an earlier version of this application (written in PHP), this was
set up as a cron job which called the relevant scripts via curl. This
was obviously not the best implementation, and so it was decided to
move to Rails.

After having done quite a bit of research, I can''t for the life of me
figure out what the best approach to handle these long-running tasks
would be. Initially I was going to set them up as Rake tasks, and call
them with cron every minute. I had a database table with active tasks
and would only start the task if less than 4 tasks were already
running.

Then I discovered Daemons, and was trying to figure out if that would
be a better approach to my issue. And then, I discovered
Delayed::Jobs, which also could be helpful.

Naturally, I want to perform these tasks as efficiently as possible,
with the least amount of resources. The tasks will be more-or-less
around the clock, so I would probably rather have it run continually
(like a Daemon) than having to reload the libraries every time (which
I would have to do with Rake). On the other hand, I''m not sure how a
Daemon would handle concurrent tasks--threading, possibly?

With all of the different options out there, I''m getting more and more
confused by the minute. Help, anyone?

Best regards,
Sebastian

I''m using delayed_job and am happy with the ease of use.
I haven''t had the need to run more than 1 queued job at a time, but
you can start any number of workers using the following syntax:

# This would allow up to four simultaneous jobs
$ RAILS_ENV=production script/delayed_job -n 4 start


On Nov 3, 5:52 pm, Sebastian von Conrad
<sebastian.von.con...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> Hi everyone,
>
> I have an application where an essential part is communicating with a
> remote API via SOAP. I want to be able to run 3-4 of these tasks at
> once, and one task might take anything from one minute to several
> hours, depending on the size of the data being sent. The app relies
> heavily on ActiveRecord fetching fetching data (in_batches) and
> updating it once the response from the API has been received.
>
> I''m using Ruby 1.8.6 and Rails 2.3.4.
>
> In an earlier version of this application (written in PHP), this was
> set up as a cron job which called the relevant scripts via curl. This
> was obviously not the best implementation, and so it was decided to
> move to Rails.
>
> After having done quite a bit of research, I can''t for the life of
me
> figure out what the best approach to handle these long-running tasks
> would be. Initially I was going to set them up as Rake tasks, and call
> them with cron every minute. I had a database table with active tasks
> and would only start the task if less than 4 tasks were already
> running.
>
> Then I discovered Daemons, and was trying to figure out if that would
> be a better approach to my issue. And then, I discovered
> Delayed::Jobs, which also could be helpful.
>
> Naturally, I want to perform these tasks as efficiently as possible,
> with the least amount of resources. The tasks will be more-or-less
> around the clock, so I would probably rather have it run continually
> (like a Daemon) than having to reload the libraries every time (which
> I would have to do with Rake). On the other hand, I''m not sure how
a
> Daemon would handle concurrent tasks--threading, possibly?
>
> With all of the different options out there, I''m getting more and
more
> confused by the minute. Help, anyone?
>
> Best regards,
> Sebastian

Thank you for your reply.

I do like the look of Delayed::Job, and after seeing your reply I
discovered that I must have glossed over something important: That you
can set up multiple workers on one computer.

I also noticed that you can use Delayed::Job together with a Daemon:
http://wiki.github.com/tobi/delayed_job/running-delayedworker-as-a-daemon

Does anyone have experience with this setup? If so, how is it working
for you?

Best regards,
Sebastian

On Nov 5, 2:41 am, "E. Litwin"
<elit...-ur4TIblo6goN+BqQ9rBEUg@public.gmane.org>
wrote:
> I''m using delayed_job and am happy with the ease of use.
> I haven''t had the need to run more than 1 queued job at a time,
but
> you can start any number of workers using the following syntax:
>
> # This would allow up to four simultaneous jobs
> $ RAILS_ENV=production script/delayed_job -n 4 start
>
> On Nov 3, 5:52 pm, Sebastian von Conrad
>
> <sebastian.von.con...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > Hi everyone,
>
> > I have an application where an essential part is communicating with a
> > remote API via SOAP. I want to be able to run 3-4 of these tasks at
> > once, and one task might take anything from one minute to several
> > hours, depending on the size of the data being sent. The app relies
> > heavily on ActiveRecord fetching fetching data (in_batches) and
> > updating it once the response from the API has been received.
>
> > I''m using Ruby 1.8.6 and Rails 2.3.4.
>
> > In an earlier version of this application (written in PHP), this was
> > set up as a cron job which called the relevant scripts via curl. This
> > was obviously not the best implementation, and so it was decided to
> > move to Rails.
>
> > After having done quite a bit of research, I can''t for the
life of me
> > figure out what the best approach to handle these long-running tasks
> > would be. Initially I was going to set them up as Rake tasks, and call
> > them with cron every minute. I had a database table with active tasks
> > and would only start the task if less than 4 tasks were already
> > running.
>
> > Then I discovered Daemons, and was trying to figure out if that would
> > be a better approach to my issue. And then, I discovered
> > Delayed::Jobs, which also could be helpful.
>
> > Naturally, I want to perform these tasks as efficiently as possible,
> > with the least amount of resources. The tasks will be more-or-less
> > around the clock, so I would probably rather have it run continually
> > (like a Daemon) than having to reload the libraries every time (which
> > I would have to do with Rake). On the other hand, I''m not
sure how a
> > Daemon would handle concurrent tasks--threading, possibly?
>
> > With all of the different options out there, I''m getting more
and more
> > confused by the minute. Help, anyone?
>
> > Best regards,
> > Sebastian