So today at work it looks like I lost the debate on using an rdbms versus a distributed data store for our product, which btw has 0 users so far. But anyways that''s another story. So along with this is the need to do async http requests from within rails. Basically I have to make one http request per data object I want from the backend. So normally I will need to make 1-4 async http requests, wait for them to return (or timeout), then continue processing as normal in the rails controller. I think this can probably be done with EM. Can anyone think of a reason not to start up Em within rails, perform the http requests, then stop the reactor and continue on? Is there any significant overhead with starting the reactor? Chris
On Oct 23, 2007, at 2:01 AM, snacktime wrote:> So today at work it looks like I lost the debate on using an rdbms > versus a distributed data store for our product, which btw has 0 users > so far. But anyways that''s another story. > > So along with this is the need to do async http requests from within > rails. Basically I have to make one http request per data object I > want from the backend. So normally I will need to make 1-4 async > http requests, wait for them to return (or timeout), then continue > processing as normal in the rails controller. I think this can > probably be done with EM. Can anyone think of a reason not to start > up Em within rails, perform the http requests, then stop the reactor > and continue on? Is there any significant overhead with starting the > reactor?I''m no EM expert, but I have two alternative suggestions. 1. Use BackgrounDRb. It was tailor made for this exact scenario. It hasn''t gotten a lot of love in the last few months, but the code works. 2. Merb allows the programmer to "render" a Proc which releases the internal mutex (so it can handle a new request) and Mongrel takes care of streaming the response back to the client in a thread-safe manner. Of course, you can "chain" multiple Procs together, so if your asynch requests require any kind of serialization then this would enforce that. (I''m a Merb newbie, so take the above with the appropriate grain of salt.) Other than that, EM would very likely work just fine. I would highly recommend against starting/stopping it on each asynch call. What would be the point? Open a socket connection between your Rails controller and EM and tear down the connection when you are done. But leave EM running. The overhead is the socket setup/teardown instead of a *process* setup/teardown which is what you suggested. cr
On 10/23/07, snacktime <snacktime at gmail.com> wrote:> > So today at work it looks like I lost the debate on using an rdbms > versus a distributed data store for our product, which btw has 0 users > so far. But anyways that''s another story. > > So along with this is the need to do async http requests from within > rails. Basically I have to make one http request per data object I > want from the backend. So normally I will need to make 1-4 async > http requests, wait for them to return (or timeout), then continue > processing as normal in the rails controller. I think this can > probably be done with EM. Can anyone think of a reason not to start > up Em within rails, perform the http requests, then stop the reactor > and continue on? Is there any significant overhead with starting the > reactor?Compared to the overhead of Rails itself and the fact that you''re kicking off one or more external requests which may timeout, I''d guess the startup overhead of EM is trivial. Your application is a perfect case for the Deferrable pattern or for EM''s new Sparwned Process object. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20071023/5b54353d/attachment.html
On 10/23/07, Chuck Remes <cremes.devlist at mac.com> wrote:> > > On Oct 23, 2007, at 2:01 AM, snacktime wrote: > > > So today at work it looks like I lost the debate on using an rdbms > > versus a distributed data store for our product, which btw has 0 users > > so far. But anyways that''s another story. > > > > So along with this is the need to do async http requests from within > > rails. Basically I have to make one http request per data object I > > want from the backend. So normally I will need to make 1-4 async > > http requests, wait for them to return (or timeout), then continue > > processing as normal in the rails controller. I think this can > > probably be done with EM. Can anyone think of a reason not to start > > up Em within rails, perform the http requests, then stop the reactor > > and continue on? Is there any significant overhead with starting the > > reactor? > > Other than that, EM would very likely work just fine. I would highly > recommend against starting/stopping it on each asynch call. What > would be the point? Open a socket connection between your Rails > controller and EM and tear down the connection when you are done. But > leave EM running. The overhead is the socket setup/teardown instead > of a *process* setup/teardown which is what you suggested.Good comments, but I understood what Chris is trying to do a little differently. I thought he was going to call EM.run inside of his Rails controller. That doesn''t start up a new process. It does block the Rails process, but that''s going to happen anyway because Rails is single-threaded. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20071023/bd64d16f/attachment.html
On 10/23/07, Chuck Remes <cremes.devlist at mac.com> wrote:> > Other than that, EM would very likely work just fine. I would highly > recommend against starting/stopping it on each asynch call. What > would be the point? Open a socket connection between your Rails > controller and EM and tear down the connection when you are done. But > leave EM running. The overhead is the socket setup/teardown instead > of a *process* setup/teardown which is what you suggested.Another point: I have an EM-based web framework that was designed specifically for REST operations that would . In it, every HTTP response is automatically an EM Deferrable, so it''s trivial to do what Chris needs to do: kick off one or more operations that require access to external resources (like other web servers or distributed data stores) and keep merrily processing other incoming client requests while waiting for the responses. I''ve been hesitant to release this thing for a couple of reasons. First, it''s married to the EM HTTP frontend, and it really should be modified to work with webrick, apache, lighty, nginx, or whatever. And second, the Ruby world already has Rails and Merb and probably doesn''t need yet another Web framework, even though the emphasis of this one (REST and external-services) is different. (Over in Javaland, there are hundreds specialized web frameworks.) Any opinions? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20071023/11c84ea0/attachment.html
On 10/23/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> On 10/23/07, Chuck Remes <cremes.devlist at mac.com> wrote: > > Other than that, EM would very likely work just fine. I would highly > > recommend against starting/stopping it on each asynch call. What > > would be the point? Open a socket connection between your Rails > > controller and EM and tear down the connection when you are done. But > > leave EM running. The overhead is the socket setup/teardown instead > > of a *process* setup/teardown which is what you suggested. > > > Another point: I have an EM-based web framework that was designed > specifically for REST operations that would . In it, every HTTP response is > automatically an EM Deferrable, so it''s trivial to do what Chris needs to > do: kick off one or more operations that require access to external > resources (like other web servers or distributed data stores) and keep > merrily processing other incoming client requests while waiting for the > responses. > > I''ve been hesitant to release this thing for a couple of reasons. First, > it''s married to the EM HTTP frontend, and it really should be modified to > work with webrick, apache, lighty, nginx, or whatever. And second, the Ruby > world already has Rails and Merb and probably doesn''t need yet another Web > framework, even though the emphasis of this one (REST and external-services) > is different. (Over in Javaland, there are hundreds specialized web > frameworks.) > > Any opinions? >Release it Francis, I have several use cases of a simple HTTP frontend without any heavy weight framework. I don''t know, but unless you intend run entire rails on top of EventMachine as swiftply of Kirk Haines, its a bad idea to have EventMachine.run in your controller. For Chris, No you can''t do async requests in rails. There is a way using mongrel_handlers and sleep. The code looks something like this: require ''active_record'' class StatusHandler < Mongrel::HttpHandler def process(request, response) id = request.params[''PATH_INFO''].slice(1, 20) current = request.params[''QUERY_STRING''] while status(id) == current do sleep 0.2 end response.start(200) do |head, out| head["Content-Type"] = "text/html" out.write status(id) end end def status(id) connection.select_value("select status from auctions where id=#{id.to_i}") end def connection ActiveRecord::Base.connection end end uri "/status", :handler => StatusHandler.new, :in_front => true Taken verbatim from : http://adam.blogs.bitscribe.net/2007/05/08/comet-with-rails-mongrel/ All the best. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org
On 10/23/07, snacktime <snacktime at gmail.com> wrote:> So today at work it looks like I lost the debate on using an rdbms > versus a distributed data store for our product, which btw has 0 users > so far. But anyways that''s another story. > > So along with this is the need to do async http requests from within > rails. Basically I have to make one http request per data object I > want from the backend. So normally I will need to make 1-4 async > http requests, wait for them to return (or timeout), then continue > processing as normal in the rails controller. I think this can > probably be done with EM. Can anyone think of a reason not to start > up Em within rails, perform the http requests, then stop the reactor > and continue on? Is there any significant overhead with starting the > reactor?If you use evented_mongrel, then your app would already be running inside of an EM loop. No need to start and stop the reactor. Kirk Haines
On 10/23/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> I''ve been hesitant to release this thing for a couple of reasons. First, > it''s married to the EM HTTP frontend, and it really should be modified to > work with webrick, apache, lighty, nginx, or whatever. And second, the Ruby > world already has Rails and Merb and probably doesn''t need yet another Web > framework, even though the emphasis of this one (REST and external-services) > is different. (Over in Javaland, there are hundreds specialized web > frameworks.)I have a bunch of optimizations for the HTTP parser, as well as a change to allow the header to be processed before the POST/PUT body is touched, that I need to get over to you sometime, Francis. I''ve been hacking on it heavily because I am using it as the basis for a sort of micro-framework specifically aimed at web app edge cases -- things like upload handling, which are long, slow, IO intensive requests, as well as ajax, service APIs, and simple request handling that benefits more from being very fast than from from being feature heavy. On an AMD 4200+ CPU, it will do about 14k hello worlds a second and on an Intel 3.0Ghz Xeon CPU, it''s doing over 20k/second right now. Kirk Haines
On 10/23/07, Kirk Haines <wyhaines at gmail.com> wrote:> > On 10/23/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote: > > > I''ve been hesitant to release this thing for a couple of reasons. First, > > it''s married to the EM HTTP frontend, and it really should be modified > to > > work with webrick, apache, lighty, nginx, or whatever. And second, the > Ruby > > world already has Rails and Merb and probably doesn''t need yet another > Web > > framework, even though the emphasis of this one (REST and > external-services) > > is different. (Over in Javaland, there are hundreds specialized web > > frameworks.) > > I have a bunch of optimizations for the HTTP parser, as well as a > change to allow the header to be processed before the POST/PUT body is > touched, that I need to get over to you sometime, Francis. I''ve been > hacking on it heavily because I am using it as the basis for a sort of > micro-framework specifically aimed at web app edge cases -- things > like upload handling, which are long, slow, IO intensive requests, as > well as ajax, service APIs, and simple request handling that benefits > more from being very fast than from from being feature heavy. > > On an AMD 4200+ CPU, it will do about 14k hello worlds a second and on > an Intel 3.0Ghz Xeon CPU, it''s doing over 20k/second right now.That sounds great! It may be what''s needed to get more attention for the EM HTTP server. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20071023/c33723df/attachment.html
> > I have a bunch of optimizations for the HTTP parser, .. > > > > On an AMD 4200+ CPU, it will do about 14k hello worlds a second and on > > an Intel 3.0Ghz Xeon CPU, it''s doing over 20k/second right now. > > That sounds great! It may be what''s needed to get more attention for the EM > HTTP server. >Get Roy Fielding''s attention - before he starts trying to sell Waka: "" I went through the readable bits with HTTP and it turned out to be a big mistake. Nobody reads HTTP in real practice, yet the overhead of parsing HTTP messages is huge."" http://lists.w3.org/Archives/Public/uri/2007Oct/0054.html =0) Cheers! Duncan __________________________ Duncan Cragg The Financial Times Group (UK) http://duncan-cragg.org/blog/