Backgroundrb devel - Jul 2006 - BackgrounDRb New release.

Howdy Folks-

	I''m happy to announce a new release of BackgrounDRb! I have added  
quite a few new features and included some nice patches from folks on  
the list.

$ script/plugin install svn://rubyforge.org//var/svn/backgroundrb

	All of the code now stays within the plugin and the start and stop  
scripts are now just stubs. This makes it easier to tweak or figure  
out how it works. The backgroundrb.rb file in the root of the plugin  
dir is the drb server part of MiddleMan and the pluginroot/lib/ 
backgroundrb.rb file is the rails client side MiddleMan.

	One of the new things is much better threading job control. This is  
mostly invisible to you as a user of the plugin but it makes the  
whole system a lot more sturdy. It gives your worker objects the  
chance to clean up after themselves when they get deleted instead of  
just getting the kill brick dropped on them ;) Thanks to David  
Lemstra for the job management code!

	The other big new feature is a built in timer and a time to live  
argument for workers and caches. This allows you to set how long a  
worker or a cache survives before it gets reaped.  The :ttl param  
defaults to 30 minutes for worker classes and 10 minutes for cached  
objects. If you don;t specify the :ttl param then thats how long your  
stuff will last. If you want longer or shorter ttl''s then  
specify :ttl in *seconds* like this:

#worker
session[:job_key] = MiddleMan.new_worker(:class => :foo_worker,
                                                                         
           :args => "Bar",
                                                                         
           :ttl => 300) # 300 seconds == 5 minutes

#caching

#fill_cache, expire in 5 minutes
@posts = Post.find(:all, :include => :comments)
MiddleMan.cache_as(:post_cache, 300, @posts)
#OR
@posts = MiddleMan.cache_as :post_cache, 300 do
   Post.find(:all, :include => :comments)
end

#retrieve_cache
@posts = MiddleMan.cache_get :post_cache do
   Post.find(:all, :include => :comments)
end

	By default the timer_sleep is set to 60 seconds. This means that the  
timer will wakeup every 60 seconds and delete caches and worker that  
are past their time to live. You can change this in the config file  
if you want it to run faster or slower.

	All your worker classes should now inherit from BackgrounDRb::Rails  
so they get all the good new thread management features. If you are  
already using the plugin you should delete the plugin from your rails  
app and reinstall it. You should also delete the script/backgroundrb  
directory and the config/backgroundrb.yml file. Then run:

# to install the new code.
$ rake backgroundrb:setup

Your workers should not need to change as long as they inherit from  
BackgrounDRb::Rails. Please test this and give me feedback if you are  
using it. I would also like to start collecting worker classes or at  
least hear what folks are using this plugin for.

*Home page: http://backgroundrb.rubyforge.org
*svn repo: svn://rubyforge.org//var/svn/backgroundrb
*Mailing list: http://rubyforge.org/mailman/listinfo/backgroundrb-devel


Thanks
-Ezra

Nifty stuff Ezra!
Haven''t had time to test yet, but just want to point out something that
I''m worried could cause people some grief, unless I''m missing
something.
> One of the new things is much better threading job control. This is
> mostly invisible to you as a user of the plugin but it makes the
> whole system a lot more sturdy. It gives your worker objects the
> chance to clean up after themselves when they get deleted instead of
> just getting the kill brick dropped on them ;)
The assumption is that your worker class has some kind of a loop and
calls ''terminate'' somewhere in the loop where it is safe to
kill (or
uses ''check_terminate'' to see if we should cleanup and then
call
''terminate''). ''terminate'' then raises a
signal saying it is OK to be
killed.

The problem:
by default, if you don''t call ''terminate'' anywhere in
the worker class,
the OK-to-Kill signal is never raised, so ''delete_worker''
never follows
through and kills the worker, but instead sits there waiting for the
OK-to-kill signal.

Maybe the worker class should have a variable (off by default) that
enables this behaviour? When I sent out the code, I assumed it''d be a
reference and people would know that they should use terminate if they
went that route. Now I''m afraid people will upgrade but not realize
their processes are no longer being killed.

I think a simple condition shoved in like:
if (@jobs[key].jb_ctrl) then do_the_signal_stuff end
Then @jb_ctrl must be enabled in worker class.

ie.
def delete_worker(key)
  @mutex.synchronize {
    if @jobs[key]
      if @jobs[key].respond_to? :thread
        if @jobs[key].thread.alive?
	  if @jobs[key].jb_ctrl
            @jobs[key].thread[:kill] = true;
            @jobs[key].thread[:safe_to_kill].wait(@mutex)
          end
          @jobs[key].thread.kill
        end
      end
      @jobs.delete(key)
    end
    @timestamps.delete(key) if @timestamps.has_key?(key)
  }
end
> I would also like to start collecting worker classes or at
> least hear what folks are using this plugin for.
I''m using it to do a hole whack of calculations w/ the database in the
background. BackgrounDRb keeps my www server responsive and lets me use
periodically_call_remote() to update a progress bar. W/ the job control,
www clients can even cancel the calculations.

cheers,
David Lemstra

How about jobs that I would like to run forever?

I collect newsfeeds from various sites in background and would like to run
them forever!!

In fact...i was doing this, but what to do now?
So..when a job gets deleted after 30 minutes..does it also deletes the
session key? If not...then we won''t be able to detect within our rails
app,
if the client has died!!


On 7/2/06, Ezra Zygmuntowicz <ezmobius@gmail.com>
wrote:
>
> Howdy Folks-
>
>         I''m happy to announce a new release of BackgrounDRb! I
have added
> quite a few new features and included some nice patches from folks on
> the list.
>
> $ script/plugin install svn://rubyforge.org//var/svn/backgroundrb
>
>         All of the code now stays within the plugin and the start and stop
> scripts are now just stubs. This makes it easier to tweak or figure
> out how it works. The backgroundrb.rb file in the root of the plugin
> dir is the drb server part of MiddleMan and the pluginroot/lib/
> backgroundrb.rb file is the rails client side MiddleMan.
>
>         One of the new things is much better threading job control. This
> is
> mostly invisible to you as a user of the plugin but it makes the
> whole system a lot more sturdy. It gives your worker objects the
> chance to clean up after themselves when they get deleted instead of
> just getting the kill brick dropped on them ;) Thanks to David
> Lemstra for the job management code!
>
>         The other big new feature is a built in timer and a time to live
> argument for workers and caches. This allows you to set how long a
> worker or a cache survives before it gets reaped.  The :ttl param
> defaults to 30 minutes for worker classes and 10 minutes for cached
> objects. If you don;t specify the :ttl param then thats how long your
> stuff will last. If you want longer or shorter ttl''s then
> specify :ttl in *seconds* like this:
>
> #worker
> session[:job_key] = MiddleMan.new_worker(:class => :foo_worker,
>
>            :args => "Bar",
>
>            :ttl => 300) # 300 seconds == 5 minutes
>
> #caching
>
> #fill_cache, expire in 5 minutes
> @posts = Post.find(:all, :include => :comments)
> MiddleMan.cache_as(:post_cache, 300, @posts)
> #OR
> @posts = MiddleMan.cache_as :post_cache, 300 do
>    Post.find(:all, :include => :comments)
> end
>
> #retrieve_cache
> @posts = MiddleMan.cache_get :post_cache do
>    Post.find(:all, :include => :comments)
> end
>
>         By default the timer_sleep is set to 60 seconds. This means that
> the
> timer will wakeup every 60 seconds and delete caches and worker that
> are past their time to live. You can change this in the config file
> if you want it to run faster or slower.
>
>         All your worker classes should now inherit from
> BackgrounDRb::Rails
> so they get all the good new thread management features. If you are
> already using the plugin you should delete the plugin from your rails
> app and reinstall it. You should also delete the script/backgroundrb
> directory and the config/backgroundrb.yml file. Then run:
>
> # to install the new code.
> $ rake backgroundrb:setup
>
> Your workers should not need to change as long as they inherit from
> BackgrounDRb::Rails. Please test this and give me feedback if you are
> using it. I would also like to start collecting worker classes or at
> least hear what folks are using this plugin for.
>
> *Home page: http://backgroundrb.rubyforge.org
> *svn repo: svn://rubyforge.org//var/svn/backgroundrb
> *Mailing list: http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
>
> Thanks
> -Ezra
>
> _______________________________________________
> Rails mailing list
> Rails@lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
>


-- 
nothing much to talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://wrath.rubyonrails.org/pipermail/rails/attachments/20060703/ef34b1aa/attachment.html

On Jul 3, 2006, at 7:04 AM, hemant wrote:
> How about jobs that I would like to run forever?
>
> I collect newsfeeds from various sites in background and would like  
> to run them forever!!
>
> In fact...i was doing this, but what to do now?
> So..when a job gets deleted after 30 minutes..does it also deletes  
> the session key? If not...then we won''t be able to detect within  
> our rails app, if the client has died!!
>
>
> On 7/2/06, Ezra Zygmuntowicz <ezmobius@gmail.com> wrote:
> Howdy Folks-
>
>         I''m happy to announce a new release of BackgrounDRb! I
have
> added
> quite a few new features and included some nice patches from folks on
> the list.
>
> $ script/plugin install svn://rubyforge.org//var/svn/backgroundrb
>
Hemant-

	Thats a good point. I will add an option to allow for
''immortal''
workers. Does this syntax look good to you?

session[:job_key] = MiddleMan.new_worker(:class => :long_worker,
                                                                         
          :args => ''whatever'',
                                                                         
          :ttl => :immortal)

	I will see what the best way to add this is and get another release  
out today.

Thanks
-Ezra

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://wrath.rubyonrails.org/pipermail/rails/attachments/20060703/aa09003d/attachment.html

On 03.07.2006, at 22:34, Ezra Zygmuntowicz wrote:
> Hemant-
>
> 	Thats a good point. I will add an option to allow for
''immortal''
> workers. Does this syntax look good to you?
>
> session[:job_key] = MiddleMan.new_worker(:class => :long_worker,
>                                                                        
>           :args => ''whatever'',
>                                                                        
>           :ttl => :immortal)
>
Wouldn''t it be better to just assume ''immortality'' if
there is
no :ttl argument? Ie. the default is that the worker is never killed?  
Seems more logical somehow :) To me, anyway.

G?nter

On 03.07.2006, at 22:34, Ezra Zygmuntowicz wrote:

>
> Hemant-
>
> 	Thats a good point. I will add an option to allow for
''immortal''
> workers. Does this syntax look good to you?
>
> session[:job_key] = MiddleMan.new_worker(:class => :long_worker,
>                                                                        
>           :args => ''whatever'',
>                                                                        
>           :ttl => :immortal)
>
> 	I will see what the best way to add this is and get another  
> release out today.
Wouldn''t it be better to just assume ''immortality'' if
there is
no :ttl argument? Seems more logical (to me, anyway) and doesn''t  
change the semantics to the first release :)

G?nter
P.S.: I replied to this message before, using a different account and  
the message now seems to wait for moderator approval... just delete  
it ;)

Yeah looks fine.

BTW, you didn''t tell me abt, what happens to the session object, when
after
30 minutes that thread gets deleted?


On 7/4/06, Ezra Zygmuntowicz <ezmobius at gmail.com>
wrote:
>
>
> On Jul 3, 2006, at 7:04 AM, hemant wrote:
>
> How about jobs that I would like to run forever?
>
> I collect newsfeeds from various sites in background and would like to run
> them forever!!
>
> In fact...i was doing this, but what to do now?
> So..when a job gets deleted after 30 minutes..does it also deletes the
> session key? If not...then we won''t be able to detect within our
rails app,
> if the client has died!!
>
>
> On 7/2/06, Ezra Zygmuntowicz <ezmobius at gmail.com> wrote:
> >
> > Howdy Folks-
> >
> >         I''m happy to announce a new release of BackgrounDRb!
I have
> > added
> > quite a few new features and included some nice patches from folks on
> > the list.
> >
> > $ script/plugin install svn://rubyforge.org//var/svn/backgroundrb
> >
> >
> Hemant-
>
> Thats a good point. I will add an option to allow for
''immortal'' workers.
> Does this syntax look good to you?
>
> session[:job_key] = MiddleMan.new_worker(:class => :long_worker,
>
>        :args => ''whatever'',
>
>        :ttl => :immortal)
>
> I will see what the best way to add this is and get another release out
> today.
>
> Thanks
> -Ezra
>
>

-- 
nothing much to talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060703/d6ccd3d5/attachment.html

On Jul 3, 2006, at 5:36 PM, hemant wrote:
> Yeah looks fine.
>
> BTW, you didn''t tell me abt, what happens to the session object,  
> when after 30 minutes that thread gets deleted?
>
>
Hemant-

	Ok after thinking about things this is what it in the current svn. I  
think you are right to wonder about what to do with a stale session.  
So I changed things to behave like this:

When you do:
session[:job_key] = MiddleMan.new_worker(:class => :long_worker,
                                                                         
           :args => ''whatever'')


It will create an immortal worker by default if you don''t include  
the :ttl param. But you can include the :ttl if you want to be specific.

#So this will create an immortal worker too:
session[:job_key] = MiddleMan.new_worker(:class => :long_worker,
                                                                         
          :args => ''whatever'',
                                                                         
          :ttl => :immortal)

So now you have to specify the :ttl param if you want your worker to  
be killed after a certain amount of time.

session[:job_key] = MiddleMan.new_worker(:class => :long_worker,
                                                                         
          :args => ''whatever'',
                                                                         
          :ttl => 900) # 15 minutes

When you create immortal workers you are responsible for deleting  
them when you are done. When you do use caching you still get a  
default :ttl of 10 minutes. And you should use the block form of  
cache_get so that if the cache has expired, it will get filled again  
and behave like a cache hit.

So to answer your question about what to do with the session if your  
worker dies is something like this:

unless MiddleMan[session[:jobkey]].nil?
    MiddleMan[session[:jobkey]].some_method
end

So I recommend either using immortal workers and deleting them with  
delete_worker or being careful about checking that your job is alive  
before calling methods on it.


Cheers-
-Ezra

Thanks Erza,

Just updated the SVN....
Yeah...i had the idea that, i must either reset the rails session after half
an hour, or check for availability of the Middleman class object!!

Anyway..thanks for the immortal worker threads, they are a required feature
i guess.


On 7/4/06, Ezra Zygmuntowicz <ezmobius at gmail.com>
wrote:
>
>
> On Jul 3, 2006, at 5:36 PM, hemant wrote:
>
> > Yeah looks fine.
> >
> > BTW, you didn''t tell me abt, what happens to the session
object,
> > when after 30 minutes that thread gets deleted?
> >
> >
>
> Hemant-
>
>         Ok after thinking about things this is what it in the current svn.
> I
> think you are right to wonder about what to do with a stale session.
> So I changed things to behave like this:
>
> When you do:
> session[:job_key] = MiddleMan.new_worker(:class => :long_worker,
>
>            :args => ''whatever'')
>
>
> It will create an immortal worker by default if you don''t include
> the :ttl param. But you can include the :ttl if you want to be specific.
>
> #So this will create an immortal worker too:
> session[:job_key] = MiddleMan.new_worker(:class => :long_worker,
>
>           :args => ''whatever'',
>
>           :ttl => :immortal)
>
> So now you have to specify the :ttl param if you want your worker to
> be killed after a certain amount of time.
>
> session[:job_key] = MiddleMan.new_worker(:class => :long_worker,
>
>           :args => ''whatever'',
>
>           :ttl => 900) # 15 minutes
>
> When you create immortal workers you are responsible for deleting
> them when you are done. When you do use caching you still get a
> default :ttl of 10 minutes. And you should use the block form of
> cache_get so that if the cache has expired, it will get filled again
> and behave like a cache hit.
>
> So to answer your question about what to do with the session if your
> worker dies is something like this:
>
> unless MiddleMan[session[:jobkey]].nil?
>     MiddleMan[session[:jobkey]].some_method
> end
>
> So I recommend either using immortal workers and deleting them with
> delete_worker or being careful about checking that your job is alive
> before calling methods on it.
>
>
> Cheers-
> -Ezra
>
>

-- 
nothing much to talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060704/299fbfbc/attachment-0001.html

Hi,

I would like to do something simple yet I''m confused.

 

Lets say I have a table with 2 fields: rank, updated_at.

 

I would like to show all records matching this condition:

 

Rank = rank - (Time.now - Updated_at.to_time)

 

As you can see this is supposed to be a calculation on the field.

 

How do I write a query to show this?

 

I''ve tried with_scope, and regular find but did not succeed.

 

Please advice

 

Thanks

Adi

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://wrath.rubyonrails.org/pipermail/rails/attachments/20060806/69efd788/attachment.html