Ok Ok Ok... first off, I apologize for the sliiightly misleading subject line. But, I thought I''d try to draw some attention and see if I could get some discussion going (hopefully no flames though!). I was browsing around and I came across this interview with Alex Payne from twitter: http://www.radicalbehavior.com/5-question-interview-with-twitter-developer-alex-payne The second interview question is what particularly drew my attention where he says that they have been running into a number of scaling issues that they probably wouldn''t have with other frameworks. There are three main points that caught my attention in the response. 1) "At this point in time there''s no facility in Rails to talk to more than one database at a time." 2) "setting up multiple read-only slave databases [is not a quick fix to implement]" 3) Ruby + Rails'' syntactical sugar = slow Point #3 is pretty well known, the solutions always mentioned before is scale out. However, Alex says that they can''t because of 1 & 2. I''ve been under the impression (and still am) that doing 1 & 2 really isn''t that hard. So, the question is, what to do if you have a rails app and are in twitters place? -carl -- EPA Rating: 3000 Lines of Code / Gallon (of coffee) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Carl Lerche wrote:> 1) "At this point in time there''s no facility in Rails to talk to more > than one database at a time."You can set up another model to use another DB connection that''s configured in your database.yml file. Like so: class EmailModel < ActiveRecord::Base self.abstract_class = true establish_connection "#{RAILS_ENV}_email" end I have actual models that then subclass from EmailModel that actually interact with tables in that connection''s DB. In this case, the *_email connection just _happens_ to be to another SQL Server (gasp!) database on the same server, but there isn''t any reason that I can think of that you shouldn''t be able to configure another connection to go wherever you want and deal with it. You just have to keep track of your models, etc. Does that sound right to everyone? Wes -- Posted via http://www.ruby-forum.com/. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
On 4/12/07, Carl Lerche <carl.lerche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> > Ok Ok Ok... first off, I apologize for the sliiightly misleading > subject line. But, I thought I''d try to draw some attention and see if > I could get some discussion going (hopefully no flames though!).> 1) "At this point in time there''s no facility in Rails to talk to more > than one database at a time."I have seen Rails deployments that use multiple databases, theres an illustration of one on the Ruby on Rails site for a german web community. Basically there is two databases, each is attached to a farm of Rails Pizza boxes. Each Pizza box can only ever see one DB, but the DBs increment their PKs in 2s (one is even the other odd), so that they can periodically replicate/merge their data sets. In principle you could use multiple DBs and scale horizontally, but replication overhead will probably eat into you very quickly.> 2) "setting up multiple read-only slave databases [is not a quick fix > to implement]" > > 3) Ruby + Rails'' syntactical sugar = slow> Point #3 is pretty well known, the solutions always mentioned before > is scale out. However, Alex says that they can''t because of 1 & 2. > I''ve been under the impression (and still am) that doing 1 & 2 really > isn''t that hard.Well there are different options for 2. I am quite sure that you actually spend lots of money on your DB to achieve it. He did say or imply though that scaling out incurs additional DB overhead per Rails instance. Which implies an optimal ratio between DBs & Rails instances.> So, the question is, what to do if you have a rails app and are in > twitters place?Pretty much what he is doing already. But then, its not like scaling is easy in the first place. You have to ask yourself though - Twitter got up and running in ~ 9-12 months or so? In a Java/.NET version would he even be in production by now? How would the PHP equivalent fare? --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
I''m taking the multiple-databases idea even one further: reading from external databases and dynamically *building* an AR heirarchy to ease access to that database (this is an administration site used to administer many websites). The real question needs to be: what server is he running? Is he using capistrano to help with distributed installs? I''ve heard good things about mongrel clusters and other bits of load balancing. Rails is most definitely scalable, it just may not be as easy as with other (read: older) technologies. Jason On 4/12/07, Richard Conroy <richard.conroy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> > > On 4/12/07, Carl Lerche <carl.lerche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > > Ok Ok Ok... first off, I apologize for the sliiightly misleading > > subject line. But, I thought I''d try to draw some attention and see if > > I could get some discussion going (hopefully no flames though!). > > > 1) "At this point in time there''s no facility in Rails to talk to more > > than one database at a time." > > I have seen Rails deployments that use multiple databases, theres > an illustration of one on the Ruby on Rails site for a german web > community. > > Basically there is two databases, each is attached to a farm > of Rails Pizza boxes. Each Pizza box can only ever see one DB, > but the DBs increment their PKs in 2s (one is even the other odd), > so that they can periodically replicate/merge their data sets. > > In principle you could use multiple DBs and scale horizontally, > but replication overhead will probably eat into you very quickly. > > > 2) "setting up multiple read-only slave databases [is not a quick fix > > to implement]" > > > > 3) Ruby + Rails'' syntactical sugar = slow > > > Point #3 is pretty well known, the solutions always mentioned before > > is scale out. However, Alex says that they can''t because of 1 & 2. > > I''ve been under the impression (and still am) that doing 1 & 2 really > > isn''t that hard. > > Well there are different options for 2. I am quite sure that you actually > spend lots of money on your DB to achieve it. > > He did say or imply though that scaling out incurs additional DB overhead > per Rails instance. Which implies an optimal ratio between DBs & Rails > instances. > > > So, the question is, what to do if you have a rails app and are in > > twitters place? > > Pretty much what he is doing already. But then, its not like scaling is > easy in > the first place. > > You have to ask yourself though - Twitter got up and running in ~ 9-12 > months or so? > In a Java/.NET version would he even be in production by now? How would > the PHP > equivalent fare? > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Preface: I''ve been working at Obvious (Twitter''s parent company) since it was Odeo, and Alex is one of my co-workers. We''re definitely in a distributed setup, using capistrano, etc, etc. Our database is not currently a limiting factor, nor is ActiveRecord - scaling those components was certainly a challenge, but one that we were glad to have. That we have such a thriving and active community is, as many have suggested, a testament to the tools that Rails and the Ruby community in general offer us as developers and designers. I''ll be giving a talk at the SDForum Silicon Valley Ruby Conference next weekend on the subject of Scaling Twitter. More details at http://romeda.org/blog/2007/04/scaling-twitter-talk.html Blaine It''s Obvious - http://twitter.com/blaine On Apr 12, 11:48 am, "Jason Roelofs" <jameskil...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> I''m taking the multiple-databases idea even one further: reading from > external databases and dynamically *building* an AR heirarchy to ease access > to that database (this is an administration site used to administer many > websites). > > The real question needs to be: what server is he running? Is he using > capistrano to help with distributed installs? I''ve heard good things about > mongrel clusters and other bits of load balancing. Rails is most definitely > scalable, it just may not be as easy as with other (read: older) > technologies. > > Jason > > On 4/12/07, Richard Conroy <richard.con...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > > > On 4/12/07, Carl Lerche <carl.ler...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > > Ok Ok Ok... first off, I apologize for the sliiightly misleading > > > subject line. But, I thought I''d try to draw some attention and see if > > > I could get some discussion going (hopefully no flames though!). > > > > 1) "At this point in time there''s no facility in Rails to talk to more > > > than one database at a time." > > > I have seen Rails deployments that use multiple databases, theres > > an illustration of one on the Ruby on Rails site for a german web > > community. > > > Basically there is two databases, each is attached to a farm > > of Rails Pizza boxes. Each Pizza box can only ever see one DB, > > but the DBs increment their PKs in 2s (one is even the other odd), > > so that they can periodically replicate/merge their data sets. > > > In principle you could use multiple DBs and scale horizontally, > > but replication overhead will probably eat into you very quickly. > > > > 2) "setting up multiple read-only slave databases [is not a quick fix > > > to implement]" > > > > 3) Ruby + Rails'' syntactical sugar = slow > > > > Point #3 is pretty well known, the solutions always mentioned before > > > is scale out. However, Alex says that they can''t because of 1 & 2. > > > I''ve been under the impression (and still am) that doing 1 & 2 really > > > isn''t that hard. > > > Well there are different options for 2. I am quite sure that you actually > > spend lots of money on your DB to achieve it. > > > He did say or imply though that scaling out incurs additional DB overhead > > per Rails instance. Which implies an optimal ratio between DBs & Rails > > instances. > > > > So, the question is, what to do if you have a rails app and are in > > > twitters place? > > > Pretty much what he is doing already. But then, its not like scaling is > > easy in > > the first place. > > > You have to ask yourself though - Twitter got up and running in ~ 9-12 > > months or so? > > In a Java/.NET version would he even be in production by now? How would > > the PHP > > equivalent fare?--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
You do what you do with any other bottleneck. You take measurements, understand what your application is doing and then address it. What you don''t do is panic: Just because Rails might not be able to run Ebay ''out of the box'' doesn''t mean that it is a pile of junk. There are loads of solutions. A few examples are: - Scale, replicate and cluster your database. Oracle cracked this years ago. - Does all of your data in the database need to talk to all of the other data? If it doesn''t then partition the data along the appropriate fault lines. Technologies such as Amazon EC2 allow you to launch a complete machine per client per application per database if you want. ''Shared nothing'' is more than just avoiding threads. It means don''t share anything unless you have to - because bottlenecks happen at the coupling points. - Split the application and glue it together with REST calls. It doesn''t all have to live in one stack. Splitting the stack gives you multiple databases. For example you could have a historical application that looks after what was happening yesterday, and a live application that looks after what happened today, with an archiving process moving rows between the two. Today is what is important and that has the fast response, triggering an AJAX call to the ''historical'' stack which may take a little longer to respond. But most importantly don''t throw the baby out with the bathwater. Ruby/ Rails gives you productivity by solving the problems inherent in web- apps. If you suddenly find you have a hit on your hand then that is a good problem to have - spend some of the money on fixing and improving the framework/infrastructure/language interpreter, or more likely your application design. But in the vast majority of cases scaling is a non-issue. It is much more important just to get the app written and written quickly - that''s where Rails still wins. NeilW On Apr 12, 5:36 pm, "Carl Lerche" <carl.ler...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> So, the question is, what to do if you have a rails app and are in > twitters place?--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---