Hi list, I''ve just published a post detailing our Unicorn setup, migration process, and reasons for choosing it: http://github.com/blog/517-unicorn Thanks again! -- Chris Wanstrath http://github.com/defunkt
Chris Wanstrath <chris at ozmm.org> wrote:> Hi list, > > I''ve just published a post detailing our Unicorn setup, migration > process, and reasons for choosing it: > http://github.com/blog/517-unicornCool! "The Unicorn master manages the workers and balancing" Actually, the master never manages balancing, just the workers. The diagram is a little inaccurate as it looks like the master sees the requests, it never does. The request flow is like this: requests | | | shared socket(s) /|\ / | \ | | | | | | worker pool While the shared socket is opened and configured by the master, but the master does nothing else with the sockets. You''re completely right about the pull balancing, it''s one of the most distinctive differences between Unicorn and other setups. Also, for the 502s, do you get more 502s after the initial worker times out? I think nginx may (still)[1] mark a backend as completely dead/down when a backend dies. That may cause nginx to stop forwarding to that backend entirely and throw more 502s for a few seconds until nginx decides to actually try that backend again. Setting fail_timeout=0 causes nginx to never mark backends as down and always try to send a request to them: upstream github { server unix:/data/github/current/tmp/sockets/unicorn.sock fail_timeout=0; } I''ll add this bit somewhere in the Unicorn docs and release 0.93.3 with the OpenBSD fix for Jeremy in a bit.> Thanks again!No problem :) [1] - I''m not 100% sure if nginx still does this, but I don''t see anything in the 0.6.x changelog that indicates otherwise. I don''t see a huge amount of harm in doing this always, we''ve been using fail_timeout=0 for ~2 years now regardless of the backend. -- Eric Wong
Thanks for this post Chris, it was very informative and has answered a few questions that I''ve had in my head over the last couple of days. I''ve been testing unicorn with a few apps for a couple days and actually already moved one over to it. I have a question for list. We are currently setup with a load balancer that runs nginx and haproxy. Nginx, simply proxies to haproxy, which then balances that across multiple mongrel or thin instances that span several servers. We simply include the public directory on our load balancer so nginx can serve static files right there. We don''t have nginx running on the app servers, they are just mongrel or thin. So, my question. How would you do a Unicorn deployment when you have multiple app servers? 1. Simply use mongrels upstream and let it round-robin between all the unicorn instances on the different servers? Or, perhaps use the fair-upstream plugin? nginx -> [unicorns] 2. Keep haproxy in the middle? nginx -> haproxy -> [unicorns] 3. Stick haproxy in front and have it balance between the app servers that run their own nginx? haproxy -> [nginxs] -> unicorn # could use socket instead of tcp in this case I would love to hear any opinions. Thanks! Dusty Doris On Fri, Oct 9, 2009 at 3:42 PM, Chris Wanstrath <chris at ozmm.org> wrote:> Hi list, > > I''ve just published a post detailing our Unicorn setup, migration > process, and reasons for choosing it: > http://github.com/blog/517-unicorn > > Thanks again! > > -- > Chris Wanstrath > http://github.com/defunkt > _______________________________________________ > mongrel-unicorn mailing list > mongrel-unicorn at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-unicorn >
On Fri, Oct 9, 2009 at 1:30 PM, Eric Wong <normalperson at yhbt.net> wrote:> "The Unicorn master manages the workers and balancing" > > Actually, the master never manages balancing, just the workers. ?The > diagram is a little inaccurate as it looks like the master sees the > requests, it never does. > > The request flow is like this: > > ? ? ? ? ? requests > ? ? ? ? ? ? ?| > ? ? ? ? ? ? ?| > ? ? ? ? ? ? ?| > ? ? ? ?shared socket(s) > ? ? ? ? ? ? /|\ > ? ? ? ? ? ?/ | \ > ? ? ? ? ? | ?| ?| > ? ? ? ? ? | ?| ?| > ? ? ? ? worker pool > > While the shared socket is opened and configured by the master, but the > master does nothing else with the sockets. ?You''re completely right > about the pull balancing, it''s one of the most distinctive differences > between Unicorn and other setups.Thanks! I''ve updated the diagram and some of the language to be accurate.> Also, for the 502s, do you get more 502s after the initial worker times > out? ?I think nginx may (still)[1] mark a backend as completely dead/down > when a backend dies. ?That may cause nginx to stop forwarding to that > backend entirely and throw more 502s for a few seconds until nginx > decides to actually try that backend again. > > Setting fail_timeout=0 causes nginx to never mark backends as down and > always try to send a request to them: > > ?upstream github { > ? ?server unix:/data/github/current/tmp/sockets/unicorn.sock fail_timeout=0; > ?}We''ll try this out, thanks for letting us know about it. Cheers, -- Chris Wanstrath http://github.com/defunkt
Dusty Doris <unicorn at dusty.name> wrote:> Thanks for this post Chris, it was very informative and has answered a > few questions that I''ve had in my head over the last couple of days. > I''ve been testing unicorn with a few apps for a couple days and > actually already moved one over to it. > > I have a question for list.First off, please don''t top post, thanks :)> We are currently setup with a load balancer that runs nginx and > haproxy. Nginx, simply proxies to haproxy, which then balances that > across multiple mongrel or thin instances that span several servers. > We simply include the public directory on our load balancer so nginx > can serve static files right there. We don''t have nginx running on > the app servers, they are just mongrel or thin. > > So, my question. How would you do a Unicorn deployment when you have > multiple app servers?For me, it depends on the amount of static files you serve with nginx and also the traffic you hit. Can I assume you''re running Linux 2.6 (with epoll + awesome VFS layer)? May I also assume your load balancer box is not very stressed right now?> 1. Simply use mongrels upstream and let it round-robin between all > the unicorn instances on the different servers? Or, perhaps use the > fair-upstream plugin? > > nginx -> [unicorns]Based on your description of your current setup, this would be the best way to go. I would configure a lowish listen() :backlog for the Unicorns, fail_timeout=0 in nginx for every server This setup means round-robin by default, but if one machine gets a :backlog overflow, then nginx will automatically retry on a different backend.> 2. Keep haproxy in the middle? > > nginx -> haproxy -> [unicorns]This is probably not necessary, but it can''t hurt a whole lot either. Also an option for balancing. If you''re uncomfortable with the first approach you can also configure haproxy as a backup server: upstream unicorn_failover { # round-robin between unicorn app servers on the LAN: server 192.168.0.1:8080 fail_timeout=0; server 192.168.0.2:8080 fail_timeout=0; server 192.168.0.3:8080 fail_timeout=0; # haproxy, configured the same way as you do now # the "backup" parameter means nginx won''t hit haproxy unless # all the direct unicorn connections have backlog overflows # or other issues server 127.0.0.1:8080 fail_timeout=0 backup; # haproxy backup } So your traffic flow may look like the first for the common case, but you may have a slightly more balanced/queueing solution in case you''re completely overloaded.> 3. Stick haproxy in front and have it balance between the app servers > that run their own nginx? > > haproxy -> [nginxs] -> unicorn # could use socket instead of tcp in this caseThis is probably only necessary if: 1) you have a lot of static files that don''t all fit in the VFS caches 2) you handle a lot of large uploads/responses and nginx buffering will thrash one box I know some sites that run this (or similar) config, but it''s mainly because this is what they''ve had for 5-10 years and don''t have time/resources to test new setups.> I would love to hear any opinions.You can also try the following, which is similar to what I describe in: http://article.gmane.org/gmane.comp.lang.ruby.unicorn.general/31 Pretty much all the above setups are valid. The important part is that nginx must sit *somewhere* in between Unicorn and the rest of the world. -- Eric Wong
On Fri, Oct 9, 2009 at 6:01 PM, Eric Wong <normalperson at yhbt.net> wrote:> Dusty Doris <unicorn at dusty.name> wrote: >> Thanks for this post Chris, it was very informative and has answered a >> few questions that I''ve had in my head over the last couple of days. >> I''ve been testing unicorn with a few apps for a couple days and >> actually already moved one over to it. >> >> I have a question for list. > > First off, please don''t top post, thanks :)Sorry about that.> >> We are currently setup with a load balancer that runs nginx and >> haproxy. ?Nginx, simply proxies to haproxy, which then balances that >> across multiple mongrel or thin instances that span several servers. >> We simply include the public directory on our load balancer so nginx >> can serve static files right there. ?We don''t have nginx running on >> the app servers, they are just mongrel or thin. >> >> So, my question. ?How would you do a Unicorn deployment when you have >> multiple app servers? > > For me, it depends on the amount of static files you serve with nginx > and also the traffic you hit. > > Can I assume you''re running Linux 2.6 (with epoll + awesome VFS layer)? > > May I also assume your load balancer box is not very stressed right now? >Yep. We server our css, javascript, and some images out of the load balancer. But for the majority of our images and all the dynamically created ones, we serve those from dedicated image servers that have their own nginx instance running on each one.>> 1. ?Simply use mongrels upstream and let it round-robin between all >> the unicorn instances on the different servers? ?Or, perhaps use the >> fair-upstream plugin? >> >> nginx -> [unicorns] > > Based on your description of your current setup, this would be the best > way to go. ?I would configure a lowish listen() :backlog for the > Unicorns, fail_timeout=0 in nginx for every server ?This setup means > round-robin by default, but if one machine gets a :backlog overflow, > then nginx will automatically retry on a different backend. >Thanks for the recommendation. I was going to give that a shot first to see how it went, as it would also be the easiest to manage. When you say a lowish backlog? What kind of numbers are you talking about? Say, we had 8 workers running that stayed pretty active. They are usually quick to respond, with an occasional 2 second response (say 1/100) due to a bad sql query that we need to fix. Would lowish be 16, 32, 64, 128, 1024? Oh and thanks for the tip on the fail_timeout.>> 2. ?Keep haproxy in the middle? >> >> nginx -> haproxy -> [unicorns] > > This is probably not necessary, but it can''t hurt a whole lot either. > > Also an option for balancing. ?If you''re uncomfortable with the first > approach you can also configure haproxy as a backup server: > > ?upstream unicorn_failover { > ? ?# round-robin between unicorn app servers on the LAN: > ? ?server 192.168.0.1:8080 fail_timeout=0; > ? ?server 192.168.0.2:8080 fail_timeout=0; > ? ?server 192.168.0.3:8080 fail_timeout=0; > > ? ?# haproxy, configured the same way as you do now > ? ?# the "backup" parameter means nginx won''t hit haproxy unless > ? ?# all the direct unicorn connections have backlog overflows > ? ?# or other issues > ? ?server 127.0.0.1:8080 fail_timeout=0 backup; # haproxy backup > ?} > > So your traffic flow may look like the first for the common case, but > you may have a slightly more balanced/queueing solution in case you''re > completely overloaded. > >> 3. ?Stick haproxy in front and have it balance between the app servers >> that run their own nginx? >> >> haproxy -> [nginxs] -> unicorn # could use socket instead of tcp in this case > > This is probably only necessary if: > > ?1) you have a lot of static files that don''t all fit in the VFS caches > > ?2) you handle a lot of large uploads/responses and nginx buffering will > ? ? thrash one box > > I know some sites that run this (or similar) config, but it''s mainly > because this is what they''ve had for 5-10 years and don''t have > time/resources to test new setups. > >> I would love to hear any opinions. > > You can also try the following, which is similar to what I describe in: > > ?http://article.gmane.org/gmane.comp.lang.ruby.unicorn.general/31 >Thats an interesting idea, thanks for sharing it. I like how the individual server also acts as a load balancer, but only if its having trouble itself. Otherwise, it just handles the requests through the socket connection.> Pretty much all the above setups are valid. ?The important part is that > nginx must sit *somewhere* in between Unicorn and the rest of the world. > > -- > Eric Wong >I appreciate your reply and especially for Unicorn. Thanks!
Dusty Doris <unicorn at dusty.name> wrote:> On Fri, Oct 9, 2009 at 6:01 PM, Eric Wong <normalperson at yhbt.net> wrote: > > Dusty Doris <unicorn at dusty.name> wrote: > >> 1. ?Simply use mongrels upstream and let it round-robin between all > >> the unicorn instances on the different servers? ?Or, perhaps use the > >> fair-upstream plugin? > >> > >> nginx -> [unicorns] > > > > Based on your description of your current setup, this would be the best > > way to go. ?I would configure a lowish listen() :backlog for the > > Unicorns, fail_timeout=0 in nginx for every server ?This setup means > > round-robin by default, but if one machine gets a :backlog overflow, > > then nginx will automatically retry on a different backend. > > Thanks for the recommendation. I was going to give that a shot first > to see how it went, as it would also be the easiest to manage. > > When you say a lowish backlog? What kind of numbers are you talking > about? Say, we had 8 workers running that stayed pretty active. They > are usually quick to respond, with an occasional 2 second response > (say 1/100) due to a bad sql query that we need to fix. Would lowish > be 16, 32, 64, 128, 1024?1024 is the default in Mongrel and Unicorn which is very generous. 5 is the default value that Ruby initializes the sockets at, so picking something in between is recommended. It really depends on your app and comfort level. You can also tune and refine it over time safely without worrying too much about dropping connections by configuring multiple listeners per-instance (see below). Keep in mind the backlog is rarely an exact setting, it''s more of a recommendation to the kernel (and the actual value is often higher than specified).> Oh and thanks for the tip on the fail_timeout.No problem, I somehow thought it was widely-known by now...> > You can also try the following, which is similar to what I describe in: > > > > ?http://article.gmane.org/gmane.comp.lang.ruby.unicorn.general/31 > > > > Thats an interesting idea, thanks for sharing it. I like how the > individual server also acts as a load balancer, but only if its having > trouble itself. Otherwise, it just handles the requests through the > socket connection. > I appreciate your reply and especially for Unicorn.You can also try a combination of (1) above and my proposed idea in $gmane/31 by configuring two listeners per-Unicorn instance: # primary listen 8080, :backlog => 10, :tcp_nopush => true # only when all servers overflow the backlog=10 above listen 8081, :backlog => 1024, :tcp_nopush => true And then putting the 8081s as a backup in nginx like this: ?upstream unicorn_failover { ? ?# round-robin between unicorns with small backlogs # as the primary option ? ?server 192.168.0.1:8080 fail_timeout=0; ? ?server 192.168.0.2:8080 fail_timeout=0; ? ?server 192.168.0.3:8080 fail_timeout=0; ? ?# the "backup" parameter means nginx won''t ever try these # unless the set of listeners above fail. ? ?server 192.168.0.1:8081 fail_timeout=0 backup; ? ?server 192.168.0.2:8081 fail_timeout=0 backup; ? ?server 192.168.0.3:8081 fail_timeout=0 backup; ?} You can monitor the nginx error logs and see how often it fails on the low backlog listener, and then increment/decrement the backlog of the primary listeners as needed to get better load-balancing. -- Eric Wong
On Fri, Oct 9, 2009 at 7:11 PM, Eric Wong <normalperson at yhbt.net> wrote:> Dusty Doris <unicorn at dusty.name> wrote: >> On Fri, Oct 9, 2009 at 6:01 PM, Eric Wong <normalperson at yhbt.net> wrote: >> > Dusty Doris <unicorn at dusty.name> wrote: >> >> 1. ?Simply use mongrels upstream and let it round-robin between all >> >> the unicorn instances on the different servers? ?Or, perhaps use the >> >> fair-upstream plugin? >> >> >> >> nginx -> [unicorns] >> > >> > Based on your description of your current setup, this would be the best >> > way to go. ?I would configure a lowish listen() :backlog for the >> > Unicorns, fail_timeout=0 in nginx for every server ?This setup means >> > round-robin by default, but if one machine gets a :backlog overflow, >> > then nginx will automatically retry on a different backend. >> >> Thanks for the recommendation. ?I was going to give that a shot first >> to see how it went, as it would also be the easiest to manage. >> >> When you say a lowish backlog? ?What kind of numbers are you talking >> about? ?Say, we had 8 workers running that stayed pretty active. ?They >> are usually quick to respond, with an occasional 2 second response >> (say 1/100) due to a bad sql query that we need to fix. ?Would lowish >> be 16, 32, 64, 128, 1024? > > 1024 is the default in Mongrel and Unicorn which is very generous. ?5 is > the default value that Ruby initializes the sockets at, so picking > something in between is recommended. ?It really depends on your app and > comfort level. ?You can also tune and refine it over time safely > without worrying too much about dropping connections by configuring > multiple listeners per-instance (see below). > > Keep in mind the backlog is rarely an exact setting, it''s more of a > recommendation to the kernel (and the actual value is often higher > than specified). > >> Oh and thanks for the tip on the fail_timeout. > > No problem, I somehow thought it was widely-known by now... > >> > You can also try the following, which is similar to what I describe in: >> > >> > ?http://article.gmane.org/gmane.comp.lang.ruby.unicorn.general/31 >> > >> >> Thats an interesting idea, thanks for sharing it. ?I like how the >> individual server also acts as a load balancer, but only if its having >> trouble itself. ?Otherwise, it just handles the requests through the >> socket connection. >> I appreciate your reply and especially for Unicorn. > > You can also try a combination of (1) above and my proposed idea in > $gmane/31 by configuring two listeners per-Unicorn instance: > > ? # primary > ? listen 8080, :backlog => 10, :tcp_nopush => true > > ? # only when all servers overflow the backlog=10 above > ? listen 8081, :backlog => 1024, :tcp_nopush => true > > And then putting the 8081s as a backup in nginx like this: > > ??upstream unicorn_failover { > ?? ?# round-robin between unicorns with small backlogs > ? ?# as the primary option > ?? ?server 192.168.0.1:8080 fail_timeout=0; > ?? ?server 192.168.0.2:8080 fail_timeout=0; > ?? ?server 192.168.0.3:8080 fail_timeout=0; > > ?? ?# the "backup" parameter means nginx won''t ever try these > ? ?# unless the set of listeners above fail. > ?? ?server 192.168.0.1:8081 fail_timeout=0 backup; > ?? ?server 192.168.0.2:8081 fail_timeout=0 backup; > ?? ?server 192.168.0.3:8081 fail_timeout=0 backup; > ??} > > You can monitor the nginx error logs and see how often it fails on the > low backlog listener, and then increment/decrement the backlog of > the primary listeners as needed to get better load-balancing. > > -- > Eric Wong >Awesome! I am going to give that a shot.