Backgroundrb devel - Jul 2006 - Using BackgrounDRb to replace nested loops?

In response to my Rails Weenie post today[1], someone mentioned that  
I try BackgrounDRb to help take the strain off my application.

1: http://rails.techno-weenie.net/question/2006/7/13/nested-loops- 
best-way-to-send-messages-to-blog-subscribers

I''ve just tried to install and use BackgrounDRb with my app, but
I''m
afraid that I''m not too versed in running processes like these, and I  
can''t get the caching to work--I get this error: private method  
`cache'' called for #<BackgrounDRb::MiddleMan: 
0x254f9b0>:BackgrounDRb::MiddleMan

Would someone help me by looking over my Rails Weenie question,  
below, and recommend to me whether BackgrounDRb will help me with my  
problem?

(Thanks for the great site, as always, Ezra, and it was good meeting  
you after the Workshop for Good Event, too!)

 From the Rails Weenie post:

I run a members-only blog where subscribers can opt to receive  
messages in given categories via email in real-time or daily/weekly  
digests. What?s the best way to deliver the digest messages without  
pushing my app into an ugly, processor-hungry loop?
Each message belongs to a given category, and each subscriber can  
subscribe to 0-n categories to receive updates. Right now the daily  
emails are sent as soon as a message is posted:

    1. # within the create action...after the message is saved:
    2.
    3. # get subscribers to this category
    4. @subscribers = @msg.category.subscribers
    5.
    6. # loop through the subscribers, delivering the message to each  
one
    7. for subscriber in @subscribers
    8. MsgNotifier.deliver_update(@msg)
    9. end

This causes a slight delay upon posting--the message has to be sent  
to all the subscribers for this category--but it works. (I''m not  
terribly concerned about the delay in this part, for now, but if  
anyone has a suggestion on how to improve things, I''d love to hear  
it.) The real problem arises, however, when I''m trying to compile the  
messages for daily and weekly digests:

    1. # get all real-time subscribers
    2. @subscribers = User.find_all_by_freq(''real-time'')
    3.
    4. # loop through subscribers
    5. for subscriber in @subscribers
    6.
    7. @text = nil
    8.
    9. # get categories for subscriber
   10. @categories = subscriber.categories
   11.
   12. # loop through categories
   13. for category in @categories
   14.
   15. # get messages posted in that category in the past day
   16. @msgs = category.messages(:conditions => "[created_at > ?],  
1.day.ago")
   17.
   18. # loop through messages
   19. for msg in @msgs
   20.
   21. # concatenate the messages for that category
   22. @text += msg.body
   23. end
   24. end
   25. end
   26.
   27. # deliver subscriber''s update containing all of the past days  
messages
   28. MsgNotifier.deliver_daily(@subscriber, @text)
   29.
   30. # unset @text
   31. @text = nil
   32. end

As you can see, this is really, really ugly. Is there a better way to  
gather together the content for the digest messages? I''m open to any  
suggestions--this code is very, very slow right now...

Best,

Jacob Patton

Hi-

	Answers inline:

On Jul 13, 2006, at 5:22 PM, Jacob Patton wrote:
> In response to my Rails Weenie post today[1], someone mentioned that
> I try BackgrounDRb to help take the strain off my application.
>
> 1: http://rails.techno-weenie.net/question/2006/7/13/nested-loops-
> best-way-to-send-messages-to-blog-subscribers
>
> I''ve just tried to install and use BackgrounDRb with my app, but
I''m
> afraid that I''m not too versed in running processes like these,
and I
> can''t get the caching to work--I get this error: private method
> `cache'' called for #<BackgrounDRb::MiddleMan:
> 0x254f9b0>:BackgrounDRb::MiddleMan
	I apologize for this. A recent change introduced this bug. Please  
svn up or co the plugin again and try again. Caching will work as  
advertised now. But I don''t think caching is the answer to your
problem.

	I think you would be best served by creating a worker class that did  
the compilation of messages and sent the emails in the backgroundrb  
server. I also think that every 5 or 10 minutes is real time enough  
for most things like this. So you might be better served by building  
and sending out the email digests at intervals and not with every  
time a post gets updated.

	So since you need to use ActionMailer, which in turn needs  
ActionView, you will want to pull most of rails into your drb server  
so you can make all this happen. So add this line to your script/ 
backgroundrb/start script at the top after the boot.rb require line:

require File.dirname(__FILE__) + "/../../config/environment.rb"

Then build a worker class that does the processing and sending of  
emails in the do_work method.

class EmailWorker < BackgrounDRb::Rails

   def do_work(args)
      # loop over emails and build digest
      # loop over subscribers and send email
      #
   end

end

You will need a cron job to fire off this workers method every 5 or  
10 minutes like this:

#!/usr/bin/env ruby
require "drb"
DRb.start_service
MiddleMan = DRbObject.new(nil, "druby://localhost:22222")
# setting :ttl to 600 seconds/10 minutes. Make sure you set this for  
long enough time for your worker to finish the emails
# but you need to set this so the workers will get garbage collected  
after they are done.
MiddleMan.new_worker(:class => :email_worker, :ttl => 600)



See if this gets you started and ask if you get stuck.

Cheers-
-Ezra

> <snip code>
> Best,
>
> Jacob Patton
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>

Ezra thanks for your help and advice.  I''ll try out the steps you  
suggest, and I''ll report back to the list.

Best,

Jacob

On Jul 13, 2006, at 11:11 PM, Ezra Zygmuntowicz wrote:
> Hi-
>
> 	Answers inline:
>
> On Jul 13, 2006, at 5:22 PM, Jacob Patton wrote:
>
>> In response to my Rails Weenie post today[1], someone mentioned that
>> I try BackgrounDRb to help take the strain off my application.
>>
>> 1: http://rails.techno-weenie.net/question/2006/7/13/nested-loops-
>> best-way-to-send-messages-to-blog-subscribers
>>
>> I''ve just tried to install and use BackgrounDRb with my app,
but I''m
>> afraid that I''m not too versed in running processes like
these, and I
>> can''t get the caching to work--I get this error: private
method
>> `cache'' called for #<BackgrounDRb::MiddleMan:
>> 0x254f9b0>:BackgrounDRb::MiddleMan
>
> 	I apologize for this. A recent change introduced this bug. Please  
> svn up or co the plugin again and try again. Caching will work as  
> advertised now. But I don''t think caching is the answer to your  
> problem.
>
> 	I think you would be best served by creating a worker class that  
> did the compilation of messages and sent the emails in the  
> backgroundrb server. I also think that every 5 or 10 minutes is  
> real time enough for most things like this. So you might be better  
> served by building and sending out the email digests at intervals  
> and not with every time a post gets updated.
>
> 	So since you need to use ActionMailer, which in turn needs  
> ActionView, you will want to pull most of rails into your drb  
> server so you can make all this happen. So add this line to your  
> script/backgroundrb/start script at the top after the boot.rb  
> require line:
>
> require File.dirname(__FILE__) + "/../../config/environment.rb"
>
> Then build a worker class that does the processing and sending of  
> emails in the do_work method.
>
> class EmailWorker < BackgrounDRb::Rails
>
>   def do_work(args)
>      # loop over emails and build digest
>      # loop over subscribers and send email
>      #
>   end
>
> end
>
> You will need a cron job to fire off this workers method every 5 or  
> 10 minutes like this:
>
> #!/usr/bin/env ruby
> require "drb"
> DRb.start_service
> MiddleMan = DRbObject.new(nil, "druby://localhost:22222")
> # setting :ttl to 600 seconds/10 minutes. Make sure you set this  
> for long enough time for your worker to finish the emails
> # but you need to set this so the workers will get garbage  
> collected after they are done.
> MiddleMan.new_worker(:class => :email_worker, :ttl => 600)
>
>
>
> See if this gets you started and ask if you get stuck.
>
> Cheers-
> -Ezra
>
>
>> <snip code>
>> Best,
>>
>> Jacob Patton
>> _______________________________________________
>> Backgroundrb-devel mailing list
>> Backgroundrb-devel at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>
>

Hi Ezra,

Thanks for your advice regarding using BackgrounDRb to handle long- 
running emailing processes for my blog application.

I''m having a little trouble getting BackgrounDRb to work--I''m
just
using it to post a message to the logs for now, to make sure I can  
get the code running.

My questions are inline:
> [You said:] I think you would be best served by creating a worker  
> class that did the compilation of messages and sent the emails in  
> the backgroundrb server. I also think that every 5 or 10 minutes is  
> real time enough for most things like this. So you might be better  
> served by building and sending out the email digests at intervals  
> and not with every time a post gets updated.
>
> So since you need to use ActionMailer, which in turn needs  
> ActionView, you will want to pull most of rails into your drb  
> server so you can make all this happen. So add this line to your  
> script/backgroundrb/start script at the top after the boot.rb  
> require line:
>
> require File.dirname(__FILE__) + "/../../config/environment.rb"
I''ve done this...
> Then build a worker class that does the processing and sending of  
> emails in the do_work method.
>
> class EmailWorker < BackgrounDRb::Rails
>
>   def do_work(args)
>      # loop over emails and build digest
>      # loop over subscribers and send email
>      #
>   end
>
> end
I did this, too, but my file looks like:

# in lib/workers/checker_worker.rb

class CheckerWorker < BackgrounDRb::Rails

   def do_work(args)
     logger.debug "BackgrounDRb run at #{Time.now}"
   end

end
> You will need a cron job to fire off this workers method every 5 or  
> 10 minutes like this:
>
> #!/usr/bin/env ruby
> require "drb"
> DRb.start_service
> MiddleMan = DRbObject.new(nil, "druby://localhost:22222")
Here is the file I''m using

# in lib/checker.rb

#!/usr/bin/env ruby
require "drb"
DRb.start_service
MiddleMan = DRbObject.new(nil, "druby://localhost:22222")
MiddleMan.new_worker(:class => :checker_worker, :ttl => 600)

I think I''ve set up everything correctly, and that by running  
checker.rb by typing ruby checker.rb while within the lib directory,  
the BackgrounDRb object should fire, and the debug message should be  
posted to the log.

When I run the command, though, I get this error message:

(druby://localhost:22222) /usr/local/lib/ruby/gems/1.8/gems/ 
activesupport-1.3.1/
lib/active_support/dependencies.rb:100:in `const_missing'':  
uninitialized constan
t CheckerWorker (NameError)
...

Can you, Ezra, or anyone else help me discover where I''ve gone wrong?

Thanks very much for your help,

Jacob Patton

On Jul 18, 2006, at 11:24 AM, Jacob Patton wrote:
> Hi Ezra,
>
> Thanks for your advice regarding using BackgrounDRb to handle long- 
> running emailing processes for my blog application.
>
> I''m having a little trouble getting BackgrounDRb to
work--I''m just
> using it to post a message to the logs for now, to make sure I can  
> get the code running.
>
> My questions are inline:
>
>> [You said:] I think you would be best served by creating a worker  
>> class that did the compilation of messages and sent the emails in  
>> the backgroundrb server. I also think that every 5 or 10 minutes  
>> is real time enough for most things like this. So you might be  
>> better served by building and sending out the email digests at  
>> intervals and not with every time a post gets updated.
>>
>> So since you need to use ActionMailer, which in turn needs  
>> ActionView, you will want to pull most of rails into your drb  
>> server so you can make all this happen. So add this line to your  
>> script/backgroundrb/start script at the top after the boot.rb  
>> require line:
>>
>> require File.dirname(__FILE__) +
"/../../config/environment.rb"
>
> I''ve done this...
>
>> Then build a worker class that does the processing and sending of  
>> emails in the do_work method.
>>
>> class EmailWorker < BackgrounDRb::Rails
>>
>>   def do_work(args)
>>      # loop over emails and build digest
>>      # loop over subscribers and send email
>>      #
>>   end
>>
>> end
>
> I did this, too, but my file looks like:
>
> # in lib/workers/checker_worker.rb
>
> class CheckerWorker < BackgrounDRb::Rails
>
>   def do_work(args)
>     logger.debug "BackgrounDRb run at #{Time.now}"

   needs to be @logger

>   end
>
> end
>
>> You will need a cron job to fire off this workers method every 5  
>> or 10 minutes like this:
>>
>> #!/usr/bin/env ruby
>> require "drb"
>> DRb.start_service
>> MiddleMan = DRbObject.new(nil, "druby://localhost:22222")
>
> Here is the file I''m using
>
> # in lib/checker.rb
>
> #!/usr/bin/env ruby
> require "drb"
> DRb.start_service
> MiddleMan = DRbObject.new(nil, "druby://localhost:22222")
> MiddleMan.new_worker(:class => :checker_worker, :ttl => 600)
>
> I think I''ve set up everything correctly, and that by running  
> checker.rb by typing ruby checker.rb while within the lib  
> directory, the BackgrounDRb object should fire, and the debug  
> message should be posted to the log.
>
> When I run the command, though, I get this error message:
>
> (druby://localhost:22222) /usr/local/lib/ruby/gems/1.8/gems/ 
> activesupport-1.3.1/
> lib/active_support/dependencies.rb:100:in `const_missing'':  
> uninitialized constan
> t CheckerWorker (NameError)
> ...
Hrmm.. I wonder if this is caused when you require the config/ 
environment.rb file? Can you test that for me? comment out the  
require File.dirname(__FILE__) + "/../../config/environment.rb" line  
and stop/start the drb server. Then try this again and see if you get  
errors.

Also you need to realize that Backgroundrb requires all your worker  
classes in lib/workers when it starts up. So if you didn''t restart  
the drb server after you put the checker_worker.rb file in lib/ 
workers then that would cause this exact problem. So first restart  
your drb server. If that doesnt fix it then try what I stated above  
about the environemtn.rb require. Then report back if you still have  
errors.

THanks
-Ezra


>
> Can you, Ezra, or anyone else help me discover where I''ve gone
wrong?
>
> Thanks very much for your help,
>
> Jacob Patton