Backgroundrb devel - Jun 2006 - no view involved...

The canonical example of using BackgrounDRb involves updating a  
progress bar on some user''s web page. Therefore, the long running  
background process has some affiliation with a view.

How can backgroundrb be used for long running processes that are NOT  
associated with a controller/view? Is this even possible? I  
understand the nature of Rails is such that it does "just in time"  
work, but I''m hoping to twist it to my own nefarious purposes.

Am I even making sense? Anyone have any ideas on how to do long  
running processes that are unattached to any specific controller/view  
instance?

cr

On Jun 6, 2006, at 6:10 PM, Ezra Zygmuntowicz wrote:
>
> On Jun 6, 2006, at 3:50 PM, cremes.devlist at mac.com wrote:
>
>> The canonical example of using BackgrounDRb involves updating a
>> progress bar on some user''s web page. Therefore, the long
running
>> background process has some affiliation with a view.
>>
>> How can backgroundrb be used for long running processes that are NOT
>> associated with a controller/view? Is this even possible? I
>> understand the nature of Rails is such that it does "just in
time"
>> work, but I''m hoping to twist it to my own nefarious purposes.
>>
>> Am I even making sense? Anyone have any ideas on how to do long
>> running processes that are unattached to any specific controller/view
>> instance?
> Hey-
>
> 	You absolutely do not have to use progress bars or any sort of  
> view updating. If you just want to kick off some long running tasks  
> then just use MiddleMan.new_worker to create a worker and let it  
> go. You can reconnect any time later by just keeping the return  
> value of the new_worker call in a session or somewhere else.
>
> 	But I don''t think I totally understand what you are trying to do.
> Can you explain your problem in more detail? If you want long tasks  
> to run but they don''t get kicked off by rails then there might not
> be any need for backgroundrb. You could just run a ruby daemon. If  
> you tell me what you are trying to accomplish I can help you out.
(Moving this back to the list.)

Sure, I''ll provide more detail.

For a project I''m working on I''ve been developing a web
service for
uploading large files. POSTing large files (large as in hundreds of  
MB or several GB) is pointless, and doing it on top of SOAP even  
moreso. So I think I have a workaround that will fit my environment.

The web service receives a request (say #AddFile) that contains  
various bits of info like filename, filesize, md5 checksum, some  
metadata, and a URI for fetching the file. The web service (running  
as the Rails app) will sanity check the submitted information, verify  
sufficient space exists for the file, generate a unique file id  
(uuid) and kick-off a background process to actually go fetch the  
file. It returns the uuid to the caller and gets ready to receive the  
next request. I guess you could say its a web service front-end to a  
storage subsystem.

The long running process now needs to go out and fetch the file. It  
could take minutes or hours to transfer the file and I certainly  
don''t want anyone hung up waiting for that to finish.

So the background process is essentially decoupled from the Rails app  
once the fetch request gets handed off to it. I suppose I could add  
another web service call for someone to check on "bytes transferred"  
or something which the MM could deliver, but it isn''t a necessary  
service.

Let me know if you need more detail or some clarification.

cr

On Jun 6, 2006, at 5:06 PM, Chuck Remes wrote:
>
> On Jun 6, 2006, at 6:10 PM, Ezra Zygmuntowicz wrote:
>
>>
>> On Jun 6, 2006, at 3:50 PM, cremes.devlist at mac.com wrote:
>>
>>> The canonical example of using BackgrounDRb involves updating a
>>> progress bar on some user''s web page. Therefore, the long
running
>>> background process has some affiliation with a view.
>>>
>>> How can backgroundrb be used for long running processes that are
NOT
>>> associated with a controller/view? Is this even possible? I
>>> understand the nature of Rails is such that it does "just in
time"
>>> work, but I''m hoping to twist it to my own nefarious
purposes.
>>>
>>> Am I even making sense? Anyone have any ideas on how to do long
>>> running processes that are unattached to any specific controller/ 
>>> view
>>> instance?
>
>> Hey-
>>
>> 	You absolutely do not have to use progress bars or any sort of
>> view updating. If you just want to kick off some long running tasks
>> then just use MiddleMan.new_worker to create a worker and let it
>> go. You can reconnect any time later by just keeping the return
>> value of the new_worker call in a session or somewhere else.
>>
>> 	But I don''t think I totally understand what you are trying to
do.
>> Can you explain your problem in more detail? If you want long tasks
>> to run but they don''t get kicked off by rails then there might
not
>> be any need for backgroundrb. You could just run a ruby daemon. If
>> you tell me what you are trying to accomplish I can help you out.
>
> (Moving this back to the list.)
>
> Sure, I''ll provide more detail.
>
> For a project I''m working on I''ve been developing a web
service for
> uploading large files. POSTing large files (large as in hundreds of
> MB or several GB) is pointless, and doing it on top of SOAP even
> moreso. So I think I have a workaround that will fit my environment.
>
> The web service receives a request (say #AddFile) that contains
> various bits of info like filename, filesize, md5 checksum, some
> metadata, and a URI for fetching the file. The web service (running
> as the Rails app) will sanity check the submitted information, verify
> sufficient space exists for the file, generate a unique file id
> (uuid) and kick-off a background process to actually go fetch the
> file. It returns the uuid to the caller and gets ready to receive the
> next request. I guess you could say its a web service front-end to a
> storage subsystem.
>
> The long running process now needs to go out and fetch the file. It
> could take minutes or hours to transfer the file and I certainly
> don''t want anyone hung up waiting for that to finish.
>
> So the background process is essentially decoupled from the Rails app
> once the fetch request gets handed off to it. I suppose I could add
> another web service call for someone to check on "bytes
transferred"
> or something which the MM could deliver, but it isn''t a necessary
> service.
>
> Let me know if you need more detail or some clarification.
>
> cr
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>


	Yeah that sounds like a perfect use case for backgroundrb. You can  
just kick off the backend worker from a normal rails action after you  
have validated all the stuff and the worker can stay alive and  
working as long as you want to leave it in there. You could have a  
worker something like this:



class FileFetcher

   def initialize(options)
      @options = options
      @done = false
      start_file_fetch
   end

   def start_file_fetch
      Thread.new do
          # parse the details out of the options hash and start your  
file download
          # once your script has the complete file, set @done to true
      end
   end

   def done?
      @done
   end
end


	This is of course a simplification. But you could do something like  
this and just kick off the job and let it go. Then you could have a  
cron job that runs every hour or so and iterates over the running  
FileFetcher workers and id done? is true, delete them from the pool  
of workers. I will soon be adding a time to live variable that you  
can give to new_worker so that when the job is done working and it is  
older then the ttl, it will be automatically deleted.


-Ezra

On Jun 6, 2006, at 5:56 PM, cremes.devlist at mac.com wrote:
>
> On Jun 6, 2006, at 7:40 PM, Ezra Zygmuntowicz wrote:
>
>> 	Yeah that sounds like a perfect use case for backgroundrb. You can
>> just kick off the backend worker from a normal rails action after you
>> have validated all the stuff and the worker can stay alive and
>> working as long as you want to leave it in there. You could have a
>> worker something like this:
>>
>
> Sounds good!
>
> Since a lot of backgroundrb relies on DRb, is it safe to assume I  
> could push these workers out to other DRb instances running on  
> other machines? Or is there a dependency on MiddleMan that would  
> prevent that?
>
> (Thanks for being so responsive!)
>
> cr

	No dependency that I can forsee, you can keep pushing things out to  
more drb servers on other computers. In fact the next release will  
allow your worker classes to have ''slaves'' a slave is another
drb
server run in a forked process that will only stay alive as long as  
your worker stays alive. So your worker class could potentially start  
a bunch of new processes to do heavy lifting and this slave feature  
will make it very easy. Details soon.

-Ezra

On Jun 6, 2006, at 8:06 PM, Ezra Zygmuntowicz wrote:
>
> 	No dependency that I can forsee, you can keep pushing things out  
> to more drb servers on other computers. In fact the next release  
> will allow your worker classes to have ''slaves'' a slave
is another
> drb server run in a forked process that will only stay alive as  
> long as your worker stays alive. So your worker class could  
> potentially start a bunch of new processes to do heavy lifting and  
> this slave feature will make it very easy. Details soon.
>
> -Ezra
Ah, the slave gem! I''ve been futzing with that lately too...

You''re doing great work. Better yet, you''re doing work that I
would
be doing if it wasn''t for you. Er, or something like that! :-)

cr