I''m in the early stages of implementing puppet for a medium-sized educational environment. The environment is going to support around 100 servers, and between 2,000 and 3,000 end-users. I''m worried about performance with Webrick, especially in the mornings when everybody''s laptops wake up. I''m not really keen on spending time trying to smooth out the check-in schedule on such a large # of nodes. What would the list suggest? From reading through the mailing list and the wiki, it seems that Mongrel front-ended by either Pound or Nginx is going to be the right solution. My gut instinct is to go with Nginx since it''s a known entity for our organization, and we won''t have the extra step of keeping track of patches for Pound.
On Jan 30, 2008, at 10:27 AM, Michael T. Halligan wrote:> What would the list suggest? From reading through the mailing list and > the wiki, it seems that Mongrel front-ended by either Pound or Nginx > is going to be the right solution. My gut instinct is to go with Nginx > since it''s a known entity for our organization, and we won''t have the > extra step of keeping track of patches for Pound.The primary question is whether Nginx now supports client certificates; it did not when I first looked into it. If it does, then just go with what you''re familiar with, as long as you can get it to provide the attributes Puppet needs (documented on the UsingMongrel page). I basically consider Mongrel required for production sites at this point, and I definitely don''t recommend Webrick if you care. -- Getting caught is the mother of invention. --Robert Byrne --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Wed, Jan 30, 2008 at 10:44:56AM +1100, Luke Kanies wrote:> I basically consider Mongrel required for production sites at this > point, and I definitely don''t recommend Webrick if you care.Is there some movement in that direction concretely? In other words: is Mongrel going to become mandatory? Or at least explicitely suggested in the install process (e.g. Recommends on Debian and automatically using it if found)? A. -- Man really attains the state of complete humanity when he produces, without being forced by physical need to sell himself as a commodity. - Ernesto "Che" Guevara _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Jan 30, 2008, at 11:06 AM, The Anarcat wrote:> Is there some movement in that direction concretely? > > In other words: is Mongrel going to become mandatory? Or at least > explicitely suggested in the install process (e.g. Recommends on > Debian > and automatically using it if found)?Mongrel will never become mandatory because Webrick is so easy to set up it makes it easy for people to give Puppet a try. You''re right that the docs should be updated to reflect this recommendation, though. Any volunteers? -- The chief lesson I have learned in a long life is that the only way to make a man trustworthy is to trust him; and the surest way to make him untrustworthy is to distrust him and show your distrust. -- Henry L. Stimson --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Jan 29, 2008, at 3:44 PM, Luke Kanies wrote:> On Jan 30, 2008, at 10:27 AM, Michael T. Halligan wrote: > >> What would the list suggest? From reading through the mailing list >> and >> the wiki, it seems that Mongrel front-ended by either Pound or Nginx >> is going to be the right solution. My gut instinct is to go with >> Nginx >> since it''s a known entity for our organization, and we won''t have the >> extra step of keeping track of patches for Pound. > > > The primary question is whether Nginx now supports client > certificates; it did not when I first looked into it. > > If it does, then just go with what you''re familiar with, as long as > you can get it to provide the attributes Puppet needs (documented on > the UsingMongrel page). > > I basically consider Mongrel required for production sites at this > point, and I definitely don''t recommend Webrick if you care.Have you considered adding this Best Practices document? While preparing for this deployment, there are a few areas I''ve found ambiguous, and a default recommendation would definitely be helpful while defining the architecture. The main decisions I''m going over right now are: - Using Mongrel or Webrick - Which proxy server to use if Mongrel - Whether to split fileserving into it''s own - Whether to automatically sign CSRs or not - Which method for manifest storage (LDAP, RESTful webserver, local file system) Obviously the manifest storage decision is influenced heavily the presence of an existing LDAP infrastructure.
On Jan 30, 2008, at 11:20 AM, Michael T. Halligan wrote:> Have you considered adding this Best Practices document? While > preparing for this deployment, > there are a few areas I''ve found ambiguous, and a default > recommendation would definitely be helpful > while defining the architecture. > > The main decisions I''m going over right now are: > > - Using Mongrel or Webrick > - Which proxy server to use if Mongrel > - Whether to split fileserving into it''s own > - Whether to automatically sign CSRs or not > - Which method for manifest storage (LDAP, RESTful webserver, local > file system) > > Obviously the manifest storage decision is influenced heavily the > presence of an existing LDAP infrastructure.I actually work pretty hard not to make these kinds of recommendations. The community has consistently agreed on practices I wouldn''t follow, and as more developer than sysadmin I''m not in the best position these days to even know what''s best. Part of the consulting I do is to help customers work through these questions, but they make the decisions for themselves in the end. -- Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. -- (attributed to) Brian W. Kernighan (unconfirmed) --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Jan 29, 2008, at 4:33 PM, Luke Kanies wrote:> On Jan 30, 2008, at 11:20 AM, Michael T. Halligan wrote: > >> Have you considered adding this Best Practices document? While >> preparing for this deployment, >> there are a few areas I''ve found ambiguous, and a default >> recommendation would definitely be helpful >> while defining the architecture. >> >> The main decisions I''m going over right now are: >> >> - Using Mongrel or Webrick >> - Which proxy server to use if Mongrel >> - Whether to split fileserving into it''s own >> - Whether to automatically sign CSRs or not >> - Which method for manifest storage (LDAP, RESTful webserver, local >> file system) >> >> Obviously the manifest storage decision is influenced heavily the >> presence of an existing LDAP infrastructure. > > > I actually work pretty hard not to make these kinds of > recommendations. The community has consistently agreed on practices I > wouldn''t follow, and as more developer than sysadmin I''m not in the > best position these days to even know what''s best. > > Part of the consulting I do is to help customers work through these > questions, but they make the decisions for themselves in the end.Not making these recommendations does make sense. If not a recommendation, then what could be rather helpful would be some more community input on what has been tried successfully, or how viable some of these efforts are for current deployment rather than 6-12 months down the road. One specific topic that seems to not have been sufficiently discussed are the available RESTful applications and their current state of development/maturity.
On Jan 30, 2008, at 11:47 AM, Michael T. Halligan wrote:> One specific topic that seems to not have been sufficiently discussed > are the available > RESTful applications and their current state of development/maturity.That''s because they''re not yet released. They will be released in 0.25.0, which I hope to have out this quarter, but I''ve no real idea. -- Man is the only animal that can remain on friendly terms with the victims he intends to eat until he eats them. -- Samuel Butler (1835-1902) --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Jan 29, 2008 4:20 PM, Michael T. Halligan <michael@halligan.org> wrote:> The main decisions I''m going over right now are: > > - Using Mongrel or WebrickGiven the size of your deployment, I''d be really surprised if Webrick was a good decision :)> - Which proxy server to use if MongrelYou will be fine with any of them. If you already deploy Apache, it''s not a bad choice. I didn''t find that I needed to switch to Pound until I hit somewhere around 3-4,000 clients, each checking in every 30 min when on network. As far as the Pound and patch situation goes, Jeff McCune sent the patches to Pound upstream and had no response. Yesterday I started asking them again about this, as they''ve had several dev releases since then that haven''t incorporated it.> - Whether to split fileserving into it''s own > - Whether to automatically sign CSRs or not > - Which method for manifest storage (LDAP, RESTful webserver, local > file system) > > Obviously the manifest storage decision is influenced heavily the > presence of an existing LDAP infrastructure. > > > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users >-- Nigel Kersten Systems Administrator MacOps
On 1/29/08, Michael T. Halligan <michael@halligan.org> wrote:> Not making these recommendations does make sense. If not a > recommendation, > then what could be rather helpful would be some more community input > on what > has been tried successfully, or how viable some of these efforts are > for current > deployment rather than 6-12 months down the road. > > One specific topic that seems to not have been sufficiently discussed > are the available > RESTful applications and their current state of development/maturity.Even though it lacks documentation, we''ve been using Puppet, iClassify (for Node classification and as a centralized point of authority for systems information,) Apache and Mongrel. Puppet is very liberal in what it allows you to do. For example, I love using files and templates to a degree that makes Luke a bit queasy, but that makes a lot of sense for me (and the target of what we build for people with Puppet.) What you have now, in terms of integration points with Puppet, is really external node classification and the capability to build custom reporting engines. The future is bright here, with Puppet being able to query different back ends for much of it''s work (the RESTful API and Terminus stuff.) It works just fine today, though. If you''re looking for an off-the-shelf external node classification tool, I''m not aware of any others besides iClassify. (You can get it at http://oss.hjksolutions.com/iclassify) We''re using it in half a dozen different production sites, ranging from tens of systems to hundreds. We''re actively improving it, and would certainly welcome feedback. We use it for more than just puppet -- we have hooks into Capistrano too, the combination of which makes for smokin'' hot ad-hoc change tools.. things like: cap QUERY="tag:application_server" COMMAND="/etc/init.d/apache2 restart" invoke Become an easy reality. The worst part about iClassify is installation. :) It is otherwise stable for us. Whether you use iClassify or not, I can''t recommend highly enough building a single centralized external nodes tool. You''ll find yourself using it more and more often, especially if you give yourself excellent programmatic hooks into the data it stores. Adam -- HJK Solutions - We Launch Startups - http://www.hjksolutions.com Adam Jacob, Senior Partner T: (206) 508-4759 E: adam@hjksolutions.com
On Jan 29, 2008, at 7:54 PM, Nigel Kersten wrote:> On Jan 29, 2008 4:20 PM, Michael T. Halligan <michael@halligan.org> > wrote: > >> The main decisions I''m going over right now are: >> >> - Using Mongrel or Webrick > > Given the size of your deployment, I''d be really surprised if Webrick > was a good decision :) > >> - Which proxy server to use if Mongrel > > You will be fine with any of them. If you already deploy Apache, it''s > not a bad choice. > > I didn''t find that I needed to switch to Pound until I hit somewhere > around 3-4,000 clients, each checking in every 30 min when on network.This statement has spawned a couple of more questions. It looks like the environment we''re building for the customer is continuously growing in scope, and is likely to settle somewhere around 7.5k workstations/laptops, and 100 servers, so performance is becoming more of a concern. Are you breaking Mongrel out onto multiple physical servers to handle your load, as well as just spawning extra mongrel instances?
On Feb 4, 2008 3:01 PM, Michael T. Halligan <michael@halligan.org> wrote:> > > I didn''t find that I needed to switch to Pound until I hit somewhere > > around 3-4,000 clients, each checking in every 30 min when on network. > > > This statement has spawned a couple of more questions. It looks like > the environment > we''re building for the customer is continuously growing in scope, and > is likely to settle > somewhere around 7.5k workstations/laptops, and 100 servers, so > performance is > becoming more of a concern. > > Are you breaking Mongrel out onto multiple physical servers to handle > your load, > as well as just spawning extra mongrel instances?We''ve found 8 mongrel instances with one Pound in front has been optimal for us, but we''re starting to stretch that... I''m planning to split this onto multiple servers in the next few weeks, but I was hoping to avoid that as much as possible, as it introduces a bit more complexity wrt syncing up SSL certs etc. -- Nigel Kersten Systems Administrator MacOps _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Feb 4, 2008, at 8:23 PM, Nigel Kersten wrote:> We''ve found 8 mongrel instances with one Pound in front has been > optimal for us, but we''re starting to stretch that... I''m planning > to split this onto multiple servers in the next few weeks, but I was > hoping to avoid that as much as possible, as it introduces a bit > more complexity wrt syncing up SSL certs etc.Note that you can specify a single CA server with the ''caserver'' and ''caport'' configuration attributes (which just have the same defaults as the normal server and port). -- Hollywood is a place where they''ll pay you a thousand dollars for a kiss and fifty cents for your soul. -- Marilyn Monroe --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
<Derek.Whayman@barclayscapital.com>
2008-Feb-05 08:12 UTC
Re: Making decision between Webrick and Mongrel
True, true, in fact I hadn''t thought of using a single server for CA. What you can also do is simply rdist/whatever the CA components to all of your Puppet Masters. You can have unique server certificates, the client doesn''t mind as long as the Ruby SSL checks are passed (i.e. hostname matches the server cert) and the server cert is signed by the same CA. Derek -----Original Message----- From: puppet-users-bounces@madstop.com [mailto:puppet-users-bounces@madstop.com] On Behalf Of Luke Kanies Sent: 05 February 2008 03:19 To: Puppet User Discussion Subject: Re: [Puppet-users] Making decision between Webrick and Mongrel On Feb 4, 2008, at 8:23 PM, Nigel Kersten wrote:> We''ve found 8 mongrel instances with one Pound in front has been > optimal for us, but we''re starting to stretch that... I''m planning to > split this onto multiple servers in the next few weeks, but I was > hoping to avoid that as much as possible, as it introduces a bit more > complexity wrt syncing up SSL certs etc.Note that you can specify a single CA server with the ''caserver'' and ''caport'' configuration attributes (which just have the same defaults as the normal server and port). ------------------------------------------------------------------------ For important statutory and regulatory disclosures and more information about Barclays Capital, please visit our web site at http://www.barcap.com. Internet communications are not secure and therefore the Barclays Group does not accept legal responsibility for the contents of this message. Although the Barclays Group operates anti-virus programmes, it does not accept responsibility for any damage whatsoever that is caused by viruses being passed. Any views or opinions presented are solely those of the author and do not necessarily represent those of the Barclays Group. Replies to this email may be monitored by the Barclays Group for operational or business reasons. Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP. This email may relate to or be sent from other members of the Barclays Group. ------------------------------------------------------------------------
L.A.Hurst@lboro.ac.uk
2008-Feb-05 09:26 UTC
Re: Making decision between Webrick and Mongrel
What sort of average load do you see on your server with 8 mongrel instances? Also, correct me if I''m wrong, but I make 4,000 clients every 30mins only 2 per second, which does not sound very much to me - especially as most of the time you would expect the clients to be in sync and therefore not much data would be transferred. -Laurence _____ From: puppet-users-bounces@madstop.com [mailto:puppet-users-bounces@madstop.com] On Behalf Of Nigel Kersten Sent: 05 February 2008 02:23 To: Puppet User Discussion Subject: Re: [Puppet-users] Making decision between Webrick and Mongrel On Feb 4, 2008 3:01 PM, Michael T. Halligan <michael@halligan.org> wrote:> I didn''t find that I needed to switch to Pound until I hit somewhere > around 3-4,000 clients, each checking in every 30 min when on network.This statement has spawned a couple of more questions. It looks like the environment we''re building for the customer is continuously growing in scope, and is likely to settle somewhere around 7.5k workstations/laptops, and 100 servers, so performance is becoming more of a concern. Are you breaking Mongrel out onto multiple physical servers to handle your load, as well as just spawning extra mongrel instances? We''ve found 8 mongrel instances with one Pound in front has been optimal for us, but we''re starting to stretch that... I''m planning to split this onto multiple servers in the next few weeks, but I was hoping to avoid that as much as possible, as it introduces a bit more complexity wrt syncing up SSL certs etc. -- Nigel Kersten Systems Administrator MacOps _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Feb 5, 2008, at 2:12 AM, <Derek.Whayman@barclayscapital.com> wrote:> True, true, in fact I hadn''t thought of using a single server for CA. > What you can also do is simply rdist/whatever the CA components to all > of your Puppet Masters. You can have unique server certificates, the > client doesn''t mind as long as the Ruby SSL checks are passed (i.e. > hostname matches the server cert) and the server cert is signed by the > same CA.There''s no need to do this if you specify that all clients use a single CA -- the clients and servers only need the CA cert, which is one of the primary benefits over SSH. If you mean that you can rdist the CA files to each server and skip specifying a single server, then you''ll essentially not be able to revoke certs, because different certs are likely to end up with the same serial number, and revocation is serial based. -- When one admits that nothing is certain one must, I think, also admit that some things are much more nearly certain than others. -- Bertrand Russell --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Feb 5, 2008, at 2:12 AM, <Derek.Whayman@barclayscapital.com> wrote:> True, true, in fact I hadn''t thought of using a single server for CA. > What you can also do is simply rdist/whatever the CA components to all > of your Puppet Masters. You can have unique server certificates, the > client doesn''t mind as long as the Ruby SSL checks are passed (i.e. > hostname matches the server cert) and the server cert is signed by the > same CA.Oh, and certificates will be one of the many classes newly using REST, which means it will be straightforward to use, say, a DB for your certs in the mythical 0.25.0, thus making it easy to horizontally scale certs. -- I have never met a man so ignorant that I couldn''t learn something from him. --Galileo Galilei --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
<Derek.Whayman@barclayscapital.com>
2008-Feb-05 15:40 UTC
Re: Making decision between Webrick and Mongrel
Ah, so the server can ask another host to sign it''s cert? Excellent. Unfortunately another reason for doing what we did is site-independence... Each geographic region cannot depend on another for run-time activities. Management have decreed that building hosts (and hence Puppet runs) count as run-time activities, especially if a DR situation should materialise. We could use a different CA in each region but don''t. Derek -----Original Message----- From: puppet-users-bounces@madstop.com [mailto:puppet-users-bounces@madstop.com] On Behalf Of Luke Kanies Sent: 05 February 2008 14:09 To: Puppet User Discussion Subject: Re: [Puppet-users] Making decision between Webrick and Mongrel There''s no need to do this if you specify that all clients use a single CA -- the clients and servers only need the CA cert, which is one of the primary benefits over SSH. If you mean that you can rdist the CA files to each server and skip specifying a single server, then you''ll essentially not be able to revoke certs, because different certs are likely to end up with the same serial number, and revocation is serial based. ------------------------------------------------------------------------ For important statutory and regulatory disclosures and more information about Barclays Capital, please visit our web site at http://www.barcap.com. Internet communications are not secure and therefore the Barclays Group does not accept legal responsibility for the contents of this message. Although the Barclays Group operates anti-virus programmes, it does not accept responsibility for any damage whatsoever that is caused by viruses being passed. Any views or opinions presented are solely those of the author and do not necessarily represent those of the Barclays Group. Replies to this email may be monitored by the Barclays Group for operational or business reasons. Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP. This email may relate to or be sent from other members of the Barclays Group. ------------------------------------------------------------------------
On Feb 5, 2008, at 9:40 AM, <Derek.Whayman@barclayscapital.com> wrote:> Ah, so the server can ask another host to sign it''s cert? Excellent.Hrm, I didn''t mean to imply that; we may even be talking past each other a bit here. All clients and servers that need to community must have their certs signed by the same CA. The serial file for the CA is the only part of Puppet that does not currently easily scale horizontally, so Puppet provides special parameters for telling all machines to send their CA operations (which consist entirely of getting a signed cert) to a single host. I''m not sure what you thought I meant; when 0.25.0, it probably will be possible to have chains where clientA asks serverA for a cert and serverA then asks serverB to do the actual signing, but it''s not possible right now. -- Good judgment comes from experience, and experience comes from bad judgment. --Barry LePatner --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
<Derek.Whayman@barclayscapital.com>
2008-Feb-05 16:27 UTC
Re: Making decision between Webrick and Mongrel
Basically I was trying to figure out how to "bootstrap" the Puppet Masters to get their server certs. I.e. from an mostly empty $ssldir - say where they only have the CA cert (not the keys) because the CA is a separate machine. I wrongly guessed they would ask the CA machine via $ca_server and $ca_port. Cheers, Derek -----Original Message----- From: puppet-users-bounces@madstop.com [mailto:puppet-users-bounces@madstop.com] On Behalf Of Luke Kanies Sent: 05 February 2008 16:04 To: Puppet User Discussion Subject: Re: [Puppet-users] Making decision between Webrick and Mongrel On Feb 5, 2008, at 9:40 AM, <Derek.Whayman@barclayscapital.com> wrote:> Ah, so the server can ask another host to sign it''s cert? Excellent.Hrm, I didn''t mean to imply that; we may even be talking past each other a bit here. ------------------------------------------------------------------------ For important statutory and regulatory disclosures and more information about Barclays Capital, please visit our web site at http://www.barcap.com. Internet communications are not secure and therefore the Barclays Group does not accept legal responsibility for the contents of this message. Although the Barclays Group operates anti-virus programmes, it does not accept responsibility for any damage whatsoever that is caused by viruses being passed. Any views or opinions presented are solely those of the author and do not necessarily represent those of the Barclays Group. Replies to this email may be monitored by the Barclays Group for operational or business reasons. Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP. This email may relate to or be sent from other members of the Barclays Group. ------------------------------------------------------------------------
On Feb 5, 2008, at 10:27 AM, <Derek.Whayman@barclayscapital.com> wrote:> Basically I was trying to figure out how to "bootstrap" the Puppet > Masters to get their server certs. I.e. from an mostly empty > $ssldir - > say where they only have the CA cert (not the keys) because the CA > is a > separate machine. I wrongly guessed they would ask the CA machine via > $ca_server and $ca_port.Your puppetmaster is pretty much just another Puppet client, so when you first start puppetd on it, it *will* ask its own server for a signed cert. This doesn''t work very well, though, because once you start puppetmasterd on this host, it will create the whole $ssldir/ca subdir, with a completely new CA. If this server''s clients ask it for certificates, they will get certs that are part of a different trust domain than the server itself. That is, given this situation: Corp: ServerA, ClientA DC1: ServerB, ClientB ServerB is a client of ServerA, thus is part of ServerA''s trust domain. ClientB is a client of ServerB, thus is part of ServerB''s trust domain. ClientB can''t talk to ServerA, nor can ServerB talk to itself, because of this split in trust domains. The only real way to fix this problem right now is to have every host use the same server as its CA. You can instead manually create a sub- CA cert for ServerB (Puppet''s CA does not know how to create this type of cert), so that ServerB can at least talk to itself, but ClientB still won''t be able to talk to ServerA. Note that this isn''t really a Puppet thing, it''s a certificate thing, so there''s not much Puppet can do about it. -- The surest sign that intelligent life exists elsewhere in the universe is that it has never tried to contact us. --Calvin and Hobbes (Bill Watterson) --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com