Hello everyone, I am very interested in off-loading tasks from the request/response cycle in order to speed up my application. My application is heavily connected to using web services both external and from other servers we host. My application does the following long-running tasks: - accesses / pushes to the twitter api''s (with possible need for status responses) - accesses / pushes the facebook api''s (with need for status responses) - requesting data from another server we run (rails needs access data) - pushes information to aim using ruburple I am just curious and grateful for any help people can give for what would suit me best. I am familar with backgroundrb and bj along with starling, etc. I don''t know which one will suit me best based on the tasks I need to queue. My main concern is how to properly request data and then show it to the user. For instance, when the user logs into the rails application, I need to show them certain information about them which comes from complex analyzing of data done by another server we run. When a user logs in, I need to show them the current results of this analysis. To do this, I need to perform a GET request to another server and get the results and then show them to the user. What is the "best practice" way to achieve this in which I can offload this task to a background worker and then get the results and display them to the user. For instance, say the user logs in and then the background worker kicks off to retrieve their current analyzed data from the server. How do I then wait for the data to be retrieved by the worker and when it''s ready display it to the user? I feel like I am missing something because I don''t see how the worker can "tell" the main rails application that it now has the data and then somehow show it when its "ready". Do I need to constantly poll the server or how is this usually done. Also, I need to allow the user to make certain changes to data fields and then I need to push those changes to twitter and facebook via a background process. What solution is best suited based on what I described and could someone please help me understand the pattern for how to do this. I will be eternally grateful. -- Posted via http://www.ruby-forum.com/.
On Sat, Mar 1, 2008 at 7:10 AM, Nathan Esquenazi <lists at ruby-forum.com> wrote:> Hello everyone, > > I am very interested in off-loading tasks from the request/response > cycle in order to speed up my application. My application is heavily > connected to using web services both external and from other servers we > host. > > My application does the following long-running tasks: > > - accesses / pushes to the twitter api''s (with possible need for status > responses) > - accesses / pushes the facebook api''s (with need for status responses) > - requesting data from another server we run (rails needs access data) > - pushes information to aim using ruburple > > I am just curious and grateful for any help people can give for what > would suit me best. I am familar with backgroundrb and bj along with > starling, etc. I don''t know which one will suit me best based on the > tasks I need to queue. > > My main concern is how to properly request data and then show it to the > user. For instance, when the user logs into the rails application, I > need to show them certain information about them which comes from > complex analyzing of data done by another server we run. > > When a user logs in, I need to show them the current results of this > analysis. To do this, I need to perform a GET request to another server > and get the results and then show them to the user. What is the "best > practice" way to achieve this in which I can offload this task to a > background worker and then get the results and display them to the user. > > For instance, say the user logs in and then the background worker kicks > off to retrieve their current analyzed data from the server. How do I > then wait for the data to be retrieved by the worker and when it''s ready > display it to the user? > > I feel like I am missing something because I don''t see how the worker > can "tell" the main rails application that it now has the data and then > somehow show it when its "ready". Do I need to constantly poll the > server or how is this usually done. > > Also, I need to allow the user to make certain changes to data fields > and then I need to push those changes to twitter and facebook via a > background process. > > What solution is best suited based on what I described and could someone > please help me understand the pattern for how to do this. I will be > eternally grateful. > --For what you are describing, I think BackgrounDRb will be best fit. Since, its the only solution ( that I know of ) provides interfacing with rails so as you can actually fetch results retrieved by a worker in rails. Bj is a worker queue to offload tasks. AFAIK, you can''t really interact seamlessly from rails. I don''t know much about startling. But this is coming from guy who maintains bdrb to take it with a grain of salt.
At 06:17 PM 3/1/2008, mongrel-users-request at rubyforge.org wrote:>Date: Sun, 2 Mar 2008 07:47:43 +0530 >From: hemant <gethemant at gmail.com> >Subject: Re: [Mongrel] Offloading Background Tasks >To: mongrel-users at rubyforge.org >Message-ID: ><fb283d5e0803011817t764d1b12w7fdafda64ea7ebc5 at mail.gmail.com> >Content-Type: text/plain; charset=ISO-8859-1 > >On Sat, Mar 1, 2008 at 7:10 AM, Nathan Esquenazi ><lists at ruby-forum.com> wrote: > > Hello everyone, > > > > I am very interested in off-loading tasks from the > request/response > > cycle in order to speed up my application. My application is > heavily > > connected to using web services both external and from other > servers we > > host. >[snip] > > > > I feel like I am missing something because I don''t see how the > worker > > can "tell" the main rails application that it now has the data and > then > > somehow show it when its "ready". Do I need to constantly poll the > > server or how is this usually done. > > > > Also, I need to allow the user to make certain changes to data > fields > > and then I need to push those changes to twitter and facebook via > a > > background process. > > > > What solution is best suited based on what I described and could > someone > > please help me understand the pattern for how to do this. I will > be > > eternally grateful. > > -- > >For what you are describing, I think BackgrounDRb will be best fit. >Since, its the only solution ( that I know of ) provides interfacing >with rails so as you can actually fetch results retrieved by a worker >in rails. > >Bj is a worker queue to offload tasks. AFAIK, you can''t really >interact seamlessly from rails. I don''t know much about startling. But >this is coming from guy who maintains bdrb to take it with a grain of >salt.Hi, Firstly - apologies to Hemant for misinformation from me a while ago about the current status of BackgrounDrb. It''s clear now of course to me that this an active and healthy project. I have been messing around with Ara Howards BackgroundJob (bj) and it''s got a way to spin out a bunch of jobs and then wait until they all return, which is similar to the way threads work:> jobs = Bj.submit list_of_jobs, :tag => ''important'' > ... > > jobs.each do |job| > if job.finished? > p job.exit_status > p job.stdout > p job.stderr > end > endYou can see that if by the time you get down to the "if jobs.finished?" section, you could choose to ignore whatever jobs haven''t returned yet (or possibly even TERM them if that''s appropriate). I wonder also if you''ve considered adding some architecture to your system so that you can collect whatever you can from your external services from within Rails but then whatever is left over you could pull down via Ajax after the page had mostly loaded (so some parts of the page might be left with placeholders and then get filled in with the correct info later as the Ajax calls retrieve it from Rails.. That''d be my thought on this: build something (with BackgrounDRb, Bj or whatever) that lets you spawn a bunch of a async queries. Give those queries some kind of unique handle. When you''re done building the whole page, have your view fill in data from whichever queries have returned. Then send the incomplete page to the browser and have ajax queries pull down the remaining query results (using a webservice interface passing in those unique handles you set up above to ID which job you want). Hope this helps! Steve
We''ve been testing using Spawn. Has anyone had any success in a production environment with the spawn plugin? It''s very simple but seems to work well although there doesn''t seem to be any recent activity around it. Eventually I''d like to use Amazons SQS for our application but we still have other architecture issues to sort out before going that route. Mike On Sat, Mar 1, 2008 at 9:51 PM, Steve Midgley <public at misuse.org> wrote:> At 06:17 PM 3/1/2008, mongrel-users-request at rubyforge.org wrote: > >Date: Sun, 2 Mar 2008 07:47:43 +0530 > >From: hemant <gethemant at gmail.com> > >Subject: Re: [Mongrel] Offloading Background Tasks > >To: mongrel-users at rubyforge.org > >Message-ID: > ><fb283d5e0803011817t764d1b12w7fdafda64ea7ebc5 at mail.gmail.com> > >Content-Type: text/plain; charset=ISO-8859-1 > > > > >On Sat, Mar 1, 2008 at 7:10 AM, Nathan Esquenazi > ><lists at ruby-forum.com> wrote: > > > Hello everyone, > > > > > > I am very interested in off-loading tasks from the > > request/response > > > cycle in order to speed up my application. My application is > > heavily > > > connected to using web services both external and from other > > servers we > > > host. > >[snip] > > > > > > > I feel like I am missing something because I don''t see how the > > worker > > > can "tell" the main rails application that it now has the data and > > then > > > somehow show it when its "ready". Do I need to constantly poll the > > > server or how is this usually done. > > > > > > Also, I need to allow the user to make certain changes to data > > fields > > > and then I need to push those changes to twitter and facebook via > > a > > > background process. > > > > > > What solution is best suited based on what I described and could > > someone > > > please help me understand the pattern for how to do this. I will > > be > > > eternally grateful. > > > -- > > > >For what you are describing, I think BackgrounDRb will be best fit. > >Since, its the only solution ( that I know of ) provides interfacing > >with rails so as you can actually fetch results retrieved by a worker > >in rails. > > > >Bj is a worker queue to offload tasks. AFAIK, you can''t really > >interact seamlessly from rails. I don''t know much about startling. But > >this is coming from guy who maintains bdrb to take it with a grain of > >salt. > > Hi, > > Firstly - apologies to Hemant for misinformation from me a while ago > about the current status of BackgrounDrb. It''s clear now of course to > me that this an active and healthy project. > > I have been messing around with Ara Howards BackgroundJob (bj) and it''s > got a way to spin out a bunch of jobs and then wait until they all > return, which is similar to the way threads work: > > > jobs = Bj.submit list_of_jobs, :tag => ''important'' > > ... > > > > jobs.each do |job| > > if job.finished? > > p job.exit_status > > p job.stdout > > p job.stderr > > end > > end > > You can see that if by the time you get down to the "if jobs.finished?" > section, you could choose to ignore whatever jobs haven''t returned yet > (or possibly even TERM them if that''s appropriate). > > I wonder also if you''ve considered adding some architecture to your > system so that you can collect whatever you can from your external > services from within Rails but then whatever is left over you could > pull down via Ajax after the page had mostly loaded (so some parts of > the page might be left with placeholders and then get filled in with > the correct info later as the Ajax calls retrieve it from Rails.. > > That''d be my thought on this: build something (with BackgrounDRb, Bj or > whatever) that lets you spawn a bunch of a async queries. Give those > queries some kind of unique handle. When you''re done building the whole > page, have your view fill in data from whichever queries have returned. > Then send the incomplete page to the browser and have ajax queries pull > down the remaining query results (using a webservice interface passing > in those unique handles you set up above to ID which job you want). > > Hope this helps! > > Steve > > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >
Thanks for your responses. Just wanted to reply back that I installed backgroundrb (the new version) and I definitely appreciate how useful it is and how it is fairly easy to use. Thanks for all your hard work Hermant. -- Posted via http://www.ruby-forum.com/.
On Sun, 2008-03-09 at 05:58 +0100, Nathan Esquenazi wrote:> Thanks for your responses. Just wanted to reply back that I installed > backgroundrb (the new version) and I definitely appreciate how useful it > is and how it is fairly easy to use. > > Thanks for all your hard work Hermant.No problems! If you are having any problems feel free to shoot your questions on BackgrounDRb mailing list.
Hemant Kumar wrote:> On Sun, 2008-03-09 at 05:58 +0100, Nathan Esquenazi wrote: >> Thanks for your responses. Just wanted to reply back that I installed >> backgroundrb (the new version) and I definitely appreciate how useful it >> is and how it is fairly easy to use. >> >> Thanks for all your hard work Hermant. > > No problems! > > If you are having any problems feel free to shoot your questions on > BackgrounDRb mailing list.Hi! Im using this backgroundrb plugin! The steps i followed for using this plugin are as follows: Kindly correct me where i went wrong! 1)rails_apps/app_name/ruby script/plugin install http://svn.devjavu.com/backgroundrb/trunk 2) rake backgroundrb:setup 3) ruby script/generate worker example 4)opened example_worker.rb and just wrote : 1. class ExampleWorker < BackgrounDRb::MetaWorker 2. set_worker_name :example_worker 3. def create(args = nil) 4. # this method is called, when worker is loaded for the first time 5. logger.info "here" 6. end 7. end after that i tried to run #ruby script/backgroundrb start and it gave me load error :bdrb_config.rb not found! I tried copying it to some directory and then ultimately it popped up another error : fork function is umimplemented on this machine! I have also installed the required gems :packets and chronic.. Can someone help me where i went wrong! or give me a step by step procedure to start backgroundrb running. Thanks in advance! Sup -- Posted via http://www.ruby-forum.com/.
Quoting Supriya Agarwal <lists at ruby-forum.com>:> after that i tried to run #ruby script/backgroundrb start > and it gave me load error :bdrb_config.rb not found! I tried copying it > to some directory and then ultimately it popped up another error : > fork function is umimplemented on this machine! > I have also installed the required gems :packets and chronic..This isn''t really the right list for this but... "fork function is umimplemented on this machine!" Are you running on Windows? I think the error above indicates that your OS fundamentally can''t do what backgroundRb tries to do. See Wikipedia for more info on forking http://en.wikipedia.org/wiki/Fork_%28operating_system%29 -- Cynthia Kiser
Supriya Agarwal wrote:> Hi! > > Im using this backgroundrb plugin! The steps i followed for using this > plugin are as follows: Kindly correct me where i went wrong! > > 1)rails_apps/app_name/ruby script/plugin install > http://svn.devjavu.com/backgroundrb/trunk > > 2) rake backgroundrb:setupYou need to make a db:migrate # rake db:migrate Regards -- Posted via http://www.ruby-forum.com/.