Hi I''m a Java developer working with a team that''s moving over to Ruby and Ruby on Rails - we''re really excited! We are writing a replacement for a large, java-based, e-commerce website for a client who is based in the United Kingdom. This will be the first web site of its kind written in RoR. One of the things we need to do is to access various external API''s to help us build each web page. We are required to support a number of different text-based APIs (XML, key/value pairs etc etc) via HTTP. Typically, we will call an API to get some data, let''s say a list of Countries, for example. When we receive the data we build a list of Country objects. If we simulate the same thing using ActiveRecord (SELECT * FROM Countries), we find that the task is about 5 times faster. I am assuming that the difference is that ActiveRecord creates its objects in-line, while the data is arriving and the HTTP method does not start until the entire request is complete. Can anyone suggest ideas of how we might go about doing this type of in-line processing while reading the HTTP response. I do not want to have this kind of low-level code in every controller, so we could probably do some sort of helper. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
On Feb 17, 2008, at 10:26 AM, Swordfish wrote:> Can anyone suggest ideas of how we might go about doing this type of > in-line processing while reading the HTTP response. I do not want to > have this kind of low-level code in every controller, so we could > probably do some sort of helper.I would recommend caching the results. Enduring HTTP request/response cycles for infrequently changed data such as lists of countries is very time-consuming. Another strategy you might examine is dRb. In your worker, hit the Web services periodically for a refresh and from your Rails application make the request to the local dRb instance. If dRb is too big a hammer, then just run a cron job that periodically updates a memcache server. To extract the code from your controllers, use a before_filter in app/ controllers/application.rb and the request to construct your objects will be made before each action is begun. Still, it seems like a very expensive way to retrieve what seems to be somewhat static data. Hope this helps. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Many thanks for the response. I only used a country list as an example. In fact, the API calls will retrieve dynamic data. All I want to do is to start reading the stream of data that comes down the socket before the stream has finished. On Feb 17, 9:35 pm, "s.ross" <cwdi...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> On Feb 17, 2008, at 10:26 AM, Swordfish wrote: > > > Can anyone suggest ideas of how we might go about doing this type of > > in-line processing while reading the HTTP response. I do not want to > > have this kind of low-level code in every controller, so we could > > probably do some sort of helper. > > I would recommend caching the results. Enduring HTTP request/response > cycles for infrequently changed data such as lists of countries is > very time-consuming. Another strategy you might examine is dRb. In > your worker, hit the Web services periodically for a refresh and from > your Rails application make the request to the local dRb instance. If > dRb is too big a hammer, then just run a cron job that periodically > updates a memcache server. > > To extract the code from your controllers, use a before_filter in app/ > controllers/application.rb and the request to construct your objects > will be made before each action is begun. > > Still, it seems like a very expensive way to retrieve what seems to be > somewhat static data. > > Hope this helps.--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Have you considered serving each section of the page asynchronously as an ajax request? Basically, load the page with a bunch of empty divs. Then with your Prototype''s dom:loaded event, issue a bunch of Ajax requests to fill these divs in. What happens is that each is populated by a different HTTP request, allowing for a perceived performance improvement because the page loads and content populates as it is available. On Feb 17, 2008, at 12:49 PM, Swordfish wrote:> > Many thanks for the response. I only used a country list as an > example. In fact, the API calls will retrieve dynamic data. All I > want to do is to start reading the stream of data that comes down the > socket before the stream has finished. > > On Feb 17, 9:35 pm, "s.ross" <cwdi...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: >> On Feb 17, 2008, at 10:26 AM, Swordfish wrote: >> >>> Can anyone suggest ideas of how we might go about doing this type of >>> in-line processing while reading the HTTP response. I do not want >>> to >>> have this kind of low-level code in every controller, so we could >>> probably do some sort of helper. >> >> I would recommend caching the results. Enduring HTTP request/response >> cycles for infrequently changed data such as lists of countries is >> very time-consuming. Another strategy you might examine is dRb. In >> your worker, hit the Web services periodically for a refresh and from >> your Rails application make the request to the local dRb instance. If >> dRb is too big a hammer, then just run a cron job that periodically >> updates a memcache server. >> >> To extract the code from your controllers, use a before_filter in >> app/ >> controllers/application.rb and the request to construct your objects >> will be made before each action is begun. >> >> Still, it seems like a very expensive way to retrieve what seems to >> be >> somewhat static data. >> >> Hope this helps. > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
On 17 Feb 2008, at 22:17, s.ross wrote:> Have you considered serving each section of the page asynchronously as > an ajax request? Basically, load the page with a bunch of empty divs. > Then with your Prototype''s dom:loaded event, issue a bunch of Ajax > requests to fill these divs in. What happens is that each is populated > by a different HTTP request, allowing for a perceived performance > improvement because the page loads and content populates as it is > available.Do know that although it is perceived as loading faster, it''ll probably be slower and put a bigger strain on your server and database. Three factors come into play: - Page deflating by apache will be less, since there''s less data to work with and compress (the less data, the less efficient the compression usually is, especially with text) - browser limiting the number of simultaneous connections to the same domain, that''s why Rails 2 has the asset host feature nowadays, to work around that issue - the number of extra database hits you''ll make: example: loading page in one request: fetch article with id specified in url and eagerly load the associated comments, pictures: 1 database hit, 1 rendering cycle loading page using several requests: fetch article, render page, fetch article and comments, render comments section, fetch article and pictures, render picture section: 3 database hits, 3 rendering cycles The last two can be worked around and optimized (balancing over several virtual hostnames, caching of asynchronous pages), but one may start wondering if it''s worth going through all this trouble. On top of that, browsers that have javascript disabled will get a largely empty page. Best regards Peter De Berdt --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Swordfish wrote:> Hi > > I''m a Java developer working with a team that''s moving over to Ruby > and Ruby on Rails - we''re really excited! > > We are writing a replacement for a large, java-based, e-commerce > website for a client who is based in the United Kingdom. This will be > the first web site of its kind written in RoR. > > One of the things we need to do is to access various external API''s to > help us build each web page. We are required to support a number of > different text-based APIs (XML, key/value pairs etc etc) via HTTP. >If you do XML parsing with Ruby it can become slow enough to be noticeable in an interactive context. I''ve seen 100ms delay for in memory parsing of small and simple contents (3 or four levels of elements, probably less than 100 elements) recently with the built in parser and I was really surprised. I''m not sure how you can go around that. Manual parsing with regexs might be at least one order of magnitude faster but the code can become a mess if the XML is complex. In your position I''d bench your XML parsing on actual data in memory to avoid any network-related latency problem to find out if this is one of the things that slow you down. I''ve seen some XPath parsing being really slow on Ruby compiled with pthread support too. All my systems now have Ruby compiled without pthread support for a minor global speed boost so I can''t test if the problem was really in the Xpath queries or in the basic XML parsing but leaving pthread out of the way gave me 100x the perf on the Ruby script I have in mind. Lionel --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
On Feb 17, 2008, at 1:32 PM, Peter De Berdt wrote:> > On 17 Feb 2008, at 22:17, s.ross wrote: > >> Have you considered serving each section of the page asynchronously >> as >> an ajax request? Basically, load the page with a bunch of empty divs. >> Then with your Prototype''s dom:loaded event, issue a bunch of Ajax >> requests to fill these divs in. What happens is that each is >> populated >> by a different HTTP request, allowing for a perceived performance >> improvement because the page loads and content populates as it is >> available. > > Do know that although it is perceived as loading faster, it''ll > probably be slower and put a bigger strain on your server and > database. > > Three factors come into play: > - Page deflating by apache will be less, since there''s less data to > work with and compress (the less data, the less efficient the > compression usually is, especially with text) > - browser limiting the number of simultaneous connections to the > same domain, that''s why Rails 2 has the asset host feature nowadays, > to work around that issue > - the number of extra database hits you''ll make: > example: > loading page in one request: fetch article with id specified in url > and eagerly load the associated comments, pictures: 1 database hit, > 1 rendering cycle > loading page using several requests: fetch article, render page, > fetch article and comments, render comments section, fetch article > and pictures, render picture section: 3 database hits, 3 rendering > cycles > > The last two can be worked around and optimized (balancing over > several virtual hostnames, caching of asynchronous pages), but one > may start wondering if it''s worth going through all this trouble. > > On top of that, browsers that have javascript disabled will get a > largely empty page. > > Best regards > > Peter De BerdtYou raise good points. However, unless I completely missed Swordfish''s question, the goal was to have the web service requests fulfilled quickly and probably asynchronously. With Rails, options are limited WRT any asynchronous execution. Seems one architecture that stays inside the Rails framework would be something along the lines of: before_filter :do_ws_fetches after_filter :wait_for_fetches # ... def do_ws_fetches @ws = [] @ws << Thread do # code to do first ws fetch end @ws << Thread do # code to do second ws fetch end # ... and so on ... # don''t join so everything else can go ahead pretty much as planned, except render. end def fetches_still_happening? @ws.detect{|d| d.alive?} end def wait_for_fetches # subject to some give-up / timeout criteria 0.upto(timeout_in_milliseconds / 10) break if !fetches_still_happening? sleep(.01) end end Of course, this whole idea leads us down the Rails and concurrency path, but if all that being done is foraging for data from remote sources, some careful programming can reduce the risk of stepping on data. WDYT? --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Very interesting use of filters. This is exactly the kind of approach I was thinking about. I will have a go. Many thanks. On Feb 18, 1:48 am, "s.ross" <cwdi...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> On Feb 17, 2008, at 1:32 PM, Peter De Berdt wrote: > > > > > > > On 17 Feb 2008, at 22:17, s.ross wrote: > > >> Have you considered serving each section of the page asynchronously > >> as > >> an ajax request? Basically, load the page with a bunch of empty divs. > >> Then with your Prototype''s dom:loaded event, issue a bunch of Ajax > >> requests to fill these divs in. What happens is that each is > >> populated > >> by a different HTTP request, allowing for a perceived performance > >> improvement because the page loads and content populates as it is > >> available. > > > Do know that although it is perceived as loading faster, it''ll > > probably be slower and put a bigger strain on your server and > > database. > > > Three factors come into play: > > - Page deflating by apache will be less, since there''s less data to > > work with and compress (the less data, the less efficient the > > compression usually is, especially with text) > > - browser limiting the number of simultaneous connections to the > > same domain, that''s why Rails 2 has the asset host feature nowadays, > > to work around that issue > > - the number of extra database hits you''ll make: > > example: > > loading page in one request: fetch article with id specified in url > > and eagerly load the associated comments, pictures: 1 database hit, > > 1 rendering cycle > > loading page using several requests: fetch article, render page, > > fetch article and comments, render comments section, fetch article > > and pictures, render picture section: 3 database hits, 3 rendering > > cycles > > > The last two can be worked around and optimized (balancing over > > several virtual hostnames, caching of asynchronous pages), but one > > may start wondering if it''s worth going through all this trouble. > > > On top of that, browsers that have javascript disabled will get a > > largely empty page. > > > Best regards > > > Peter De Berdt > > You raise good points. However, unless I completely missed Swordfish''s > question, the goal was to have the web service requests fulfilled > quickly and probably asynchronously. With Rails, options are limited > WRT any asynchronous execution. Seems one architecture that stays > inside the Rails framework would be something along the lines of: > > before_filter :do_ws_fetches > after_filter :wait_for_fetches > > # ... > > def do_ws_fetches > @ws = [] > @ws << Thread do > # code to do first ws fetch > end > @ws << Thread do > # code to do second ws fetch > end > # ... and so on ... > # don''t join so everything else can go ahead pretty much as > planned, except render. > end > > def fetches_still_happening? > -LSz0kx4RPVReoWH0uzbU5w@public.gmane.org{|d| d.alive?} > end > > def wait_for_fetches > # subject to some give-up / timeout criteria > 0.upto(timeout_in_milliseconds / 10) > break if !fetches_still_happening? > sleep(.01) > end > end > > Of course, this whole idea leads us down the Rails and concurrency > path, but if all that being done is foraging for data from remote > sources, some careful programming can reduce the risk of stepping on > data. > > WDYT?--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---