-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi I recently came across http://www.brainspl.at/articles/2007/11/09/a-fair-proxy-balancer-for-nginx-and-mongrel and wanted to share the experience with it: I tested it on my nginx-mongrel setup and it improved the speed of my client runs heavily. If you have a look at how the improved balancer is working then this balancer is what you should use for an nginx->puppetmaster setup: Puppet clients are asking masters for: * catalogs (rather long and big requests) * file metadata (rather small and fast requests) * file sources (can be offloaded to nginx or apache thanks to the new rest interface and on 2.6 it''s also "faster") Furthermore, they PUT "reports" on the master. I think these points cover more or less all relevant parts of the communication between client and master. During a pupet client run, there are first a bunch of file metadata/sources (plugins/facts) request, then the catalog request, then a lot of file metadata requests and probably some file source requests. At the end the reports (if configured) are PUT on the server. The default load balancer of nginx is just round-robin, which means more or less that the the first requests goes to the first backend server, the second to the second and so on. This is fine in most cases. However as catalogs requests can lost for more than a couple dozen seconds the mongrel distance dealing with the catalog request is busy compiling and not responding to any other requests. This means that if one client is asking for a catalog every 4th probably small and fast request will be stuck in the queue for the busy instance. If there are currently 2 clients asking for catalogs every second request gets already stuck. etc. This means that clients asking for metadata are slowed down, because their requests get stuck in the busy queue. The fair proxy scheduler tries to keep track of the busy instances and does redirect requests to other (probably more free) instances, hence the requests are served much faster. For further discussion have a look at http://www.ruby-forum.com/topic/132604 Before using the fair proxy scheduler I had a lot of clients that were just sitting there and waiting for something. If your ran them with debug, you could see that they were often waiting for file metadata requests. Using the fair proxy scheduler I could more or less get rid off these waiting requests and the client runs pass now much faster and without getting stucked. I have added a paragraph about the fair proxy scheduler to the wiki page about the nginx-mongrel setup. If you have any comments or further notes, just add them. cheers pete -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkxr2DgACgkQbwltcAfKi3+E/QCgmvAIFroqFdA4eTyJDV2BW/ZZ zD0AnAwvrkR1yniZh9x3Xg+IpK20JEx5 =5hKX -----END PGP SIGNATURE----- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Brice Figureau
2010-Aug-18 13:51 UTC
Re: [Puppet Users] nginx - mongrel - fair proxy balancer
On Wed, 2010-08-18 at 14:55 +0200, Peter Meier wrote:> I recently came across > http://www.brainspl.at/articles/2007/11/09/a-fair-proxy-balancer-for-nginx-and-mongrel > and wanted to share the experience with it: > > I tested it on my nginx-mongrel setup and it improved the speed of my > client runs heavily. If you have a look at how the improved balancer is > working then this balancer is what you should use for an > nginx->puppetmaster setup: > > Puppet clients are asking masters for: > > * catalogs (rather long and big requests) > * file metadata (rather small and fast requests) > * file sources (can be offloaded to nginx or apache thanks to the new > rest interface and on 2.6 it''s also "faster") > > Furthermore, they PUT "reports" on the master. > > I think these points cover more or less all relevant parts of the > communication between client and master.This is a nice (and correct) summary of the client/master interactions. You can add to this list the remote file-bucket, certificates signing and a few other esoterical interactions.> During a pupet client run, there are first a bunch of file > metadata/sources (plugins/facts) request, then the catalog request, then > a lot of file metadata requests and probably some file source requests. > At the end the reports (if configured) are PUT on the server. > > The default load balancer of nginx is just round-robin, which means more > or less that the the first requests goes to the first backend server, > the second to the second and so on. This is fine in most cases. However > as catalogs requests can lost for more than a couple dozen seconds the > mongrel distance dealing with the catalog request is busy compiling and > not responding to any other requests. This means that if one client is > asking for a catalog every 4th probably small and fast request will be > stuck in the queue for the busy instance. If there are currently 2 > clients asking for catalogs every second request gets already stuck. > etc. This means that clients asking for metadata are slowed down, > because their requests get stuck in the busy queue.The whole issue is that ruby threading model is really poor (ie green threads). Mongrel accepts the downstream connections, create a new ruby thread and let it run. Unfortunately the ruby interpreter doesn''t schedule this one, since another thread (the compilation one) is still using the CPU (and by definition green threads are not preempted). So this new thread just waits the other to finish.> The fair proxy scheduler tries to keep track of the busy instances and > does redirect requests to other (probably more free) instances, hence > the requests are served much faster. For further discussion have a look > at http://www.ruby-forum.com/topic/132604Yes, but that''s just a cure of a symptom that really shouldn''t exist. Using JRuby (and its native threading) instead of MRI you should be able to serve several compilations and file metadatas/sources at the same time (indeed improving the concurrency). You can try puppet-load (in 2.6) to measure the concurrency of your master.> Before using the fair proxy scheduler I had a lot of clients that were > just sitting there and waiting for something. If your ran them with > debug, you could see that they were often waiting for file metadata > requests. Using the fair proxy scheduler I could more or less get rid > off these waiting requests and the client runs pass now much faster and > without getting stucked.I think you might also see better concurrency with Nginx+Unicorn (even with the regular nginx load balancer).> I have added a paragraph about the fair proxy scheduler to the wiki page > about the nginx-mongrel setup. If you have any comments or further > notes, just add them.Thanks. -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.