Hi All,
I''ve been exploring puppet and mcollective recently and I was
wondering if people here might be able to point me towards some
information on a potential use case. It seems like puppet is primarily
used at the moment as a tool for managing individual machine
configurations (''What do I need for a web server in
production?''). At
the same time there is work going into imperative, distributed
solutions for provisioning machines and getting them set up with
puppet, either based on mcollective or custom built with tools like
fog (e.g. the new cloud provisioner). I''m wondering, though, has
anyone gone a step further and used the puppet declarative approach to
define the structure of a whole network? This would be particularly
useful for local CI environments when combined with virtualisation.
I''m assuming that a start point would either be exported resources or
(better, in my opinion) something vaguely like this -
http://www.devco.net/archives/2010/09/18/puppet_search_engine_with_mcollective.php
- using fact-based discovery to find appropriate services to link up
machines together. But the next logical step would be to go beyond
simply querying the state of existing resources ("what is the ip of
the database?") and say something more like "Ensure there is a
postgres 9.1 database running on my local subnet". I accept that this
could be a bit terrifying for a production environment, but in a local/
testing environment I could see it being hugely helpful. Clearly it
relies on puppet itself being able to control a variety of
virtualisation options, and perhaps a richer array of network service
controls as well.
Is this a crazy idea? Has anyone tried it? Or would sticking to e.g.
TheForeman be a better plan (I have only looked briefly at it, not
really tried it yet - at first glance it still seems a bit more
imperative than what I''m suggesting + I get the impression that it
relies on DNS to glue a lot of things together, which isn''t
necessarily so appropriate in a heavily virtualised environment.
Lucian
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
I''ve done some work on creating entire networks using defined resources or 100% exported resources for the nodes you build. The following is an example environment with classes included: https://github.com/kbarber/puppet-onedemo/tree/master/manifests If you take a look at the nodes.pp you can see my use of app_* cases. The resources are defined in the ''apps'' module path ... and they all use the pattern described in: https://github.com/kbarber/puppet-onedemo/blob/master/manifests/solutions.pp Here I have a wrapper for launching vm''s using an OpenNebula resource, and I handle exporting configurations for each node. Its not perfect and lacks time-based orchestration which is the major hick-up for such a solution. Not to mention that you would have to have stored configurations scale very well ... but if you keep the layer you export down to a minimum little information needs to live in the stored configurations themselves. So largely this is a proof of concept. The app_stuck example actually is meant to build a complete working ''pastie'' style application - with databases, app servers and load balancers. Generally though - this topic is a hot discussion these days I''ve been having with quite a few individuals. ken. On Tue, Oct 11, 2011 at 10:13 AM, Lucian Holland <lucian@symposion.co.uk> wrote:> Hi All, > > I''ve been exploring puppet and mcollective recently and I was > wondering if people here might be able to point me towards some > information on a potential use case. It seems like puppet is primarily > used at the moment as a tool for managing individual machine > configurations (''What do I need for a web server in production?''). At > the same time there is work going into imperative, distributed > solutions for provisioning machines and getting them set up with > puppet, either based on mcollective or custom built with tools like > fog (e.g. the new cloud provisioner). I''m wondering, though, has > anyone gone a step further and used the puppet declarative approach to > define the structure of a whole network? This would be particularly > useful for local CI environments when combined with virtualisation. > > I''m assuming that a start point would either be exported resources or > (better, in my opinion) something vaguely like this - > http://www.devco.net/archives/2010/09/18/puppet_search_engine_with_mcollective.php > - using fact-based discovery to find appropriate services to link up > machines together. But the next logical step would be to go beyond > simply querying the state of existing resources ("what is the ip of > the database?") and say something more like "Ensure there is a > postgres 9.1 database running on my local subnet". I accept that this > could be a bit terrifying for a production environment, but in a local/ > testing environment I could see it being hugely helpful. Clearly it > relies on puppet itself being able to control a variety of > virtualisation options, and perhaps a richer array of network service > controls as well. > > Is this a crazy idea? Has anyone tried it? Or would sticking to e.g. > TheForeman be a better plan (I have only looked briefly at it, not > really tried it yet - at first glance it still seems a bit more > imperative than what I''m suggesting + I get the impression that it > relies on DNS to glue a lot of things together, which isn''t > necessarily so appropriate in a heavily virtualised environment. > > Lucian > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Tue, Oct 11, 2011 at 02:13:36AM -0700, Lucian Holland wrote:> Hi All, > > I''ve been exploring puppet and mcollective recently and I was > wondering if people here might be able to point me towards some > information on a potential use case. It seems like puppet is primarily > used at the moment as a tool for managing individual machine > configurations (''What do I need for a web server in production?''). At > the same time there is work going into imperative, distributed > solutions for provisioning machines and getting them set up with > puppet, either based on mcollective or custom built with tools like > fog (e.g. the new cloud provisioner). I''m wondering, though, has > anyone gone a step further and used the puppet declarative approach to > define the structure of a whole network? This would be particularly > useful for local CI environments when combined with virtualisation.I''ve been pondering something not dissimiliar to this recently, a sort of "config introspection", for some of our live services (e.g. load balancers). My initial thoughts were geared around collected resources, but I''m not sure the model works well for them. We already use puppet manifests to provide control info to mcollective, so feeding mcollective discovery back into puppet might be an interesting way to do some of this information gathering. The scope of what we''re thinking about is a fair bit smaller than the sort of holistic, whole system stuff Lucian is talking about though. Stu -- From the prompt of Stu Teasdale Computer programmers never die, they just get lost in the processing. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Wow, thanks for this - there''s a huge amount in there. It will take me a while to get unpick it all (I''m still a puppet novice really!) but that looks like exactly the sort of thing I was talking about. Will have a play and see how I get on! Lucian -- Lucian Holland Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Tuesday, 11 October 2011 at 17:25, Ken Barber wrote:> I''ve done some work on creating entire networks using defined > resources or 100% exported resources for the nodes you build. The > following is an example environment with classes included: > > https://github.com/kbarber/puppet-onedemo/tree/master/manifests > > If you take a look at the nodes.pp you can see my use of app_* cases. > The resources are defined in the ''apps'' module path ... and they all > use the pattern described in: > > https://github.com/kbarber/puppet-onedemo/blob/master/manifests/solutions.pp > > Here I have a wrapper for launching vm''s using an OpenNebula resource, > and I handle exporting configurations for each node. > > Its not perfect and lacks time-based orchestration which is the major > hick-up for such a solution. Not to mention that you would have to > have stored configurations scale very well ... but if you keep the > layer you export down to a minimum little information needs to live in > the stored configurations themselves. So largely this is a proof of > concept. > > The app_stuck example actually is meant to build a complete working > ''pastie'' style application - with databases, app servers and load > balancers. > > Generally though - this topic is a hot discussion these days I''ve been > having with quite a few individuals. > > ken. > > On Tue, Oct 11, 2011 at 10:13 AM, Lucian Holland <lucian@symposion.co.uk (mailto:lucian@symposion.co.uk)> wrote: > > Hi All, > > > > I''ve been exploring puppet and mcollective recently and I was > > wondering if people here might be able to point me towards some > > information on a potential use case. It seems like puppet is primarily > > used at the moment as a tool for managing individual machine > > configurations (''What do I need for a web server in production?''). At > > the same time there is work going into imperative, distributed > > solutions for provisioning machines and getting them set up with > > puppet, either based on mcollective or custom built with tools like > > fog (e.g. the new cloud provisioner). I''m wondering, though, has > > anyone gone a step further and used the puppet declarative approach to > > define the structure of a whole network? This would be particularly > > useful for local CI environments when combined with virtualisation. > > > > I''m assuming that a start point would either be exported resources or > > (better, in my opinion) something vaguely like this - > > http://www.devco.net/archives/2010/09/18/puppet_search_engine_with_mcollective.php > > - using fact-based discovery to find appropriate services to link up > > machines together. But the next logical step would be to go beyond > > simply querying the state of existing resources ("what is the ip of > > the database?") and say something more like "Ensure there is a > > postgres 9.1 database running on my local subnet". I accept that this > > could be a bit terrifying for a production environment, but in a local/ > > testing environment I could see it being hugely helpful. Clearly it > > relies on puppet itself being able to control a variety of > > virtualisation options, and perhaps a richer array of network service > > controls as well. > > > > Is this a crazy idea? Has anyone tried it? Or would sticking to e.g. > > TheForeman be a better plan (I have only looked briefly at it, not > > really tried it yet - at first glance it still seems a bit more > > imperative than what I''m suggesting + I get the impression that it > > relies on DNS to glue a lot of things together, which isn''t > > necessarily so appropriate in a heavily virtualised environment. > > > > Lucian > > > > -- > > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > > To post to this group, send email to puppet-users@googlegroups.com (mailto:puppet-users@googlegroups.com). > > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com (mailto:puppet-users+unsubscribe@googlegroups.com). > > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com (mailto:puppet-users@googlegroups.com). > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com (mailto:puppet-users+unsubscribe@googlegroups.com). > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Wednesday, 12 October 2011 at 13:44, Stu Teasdale wrote:> On Tue, Oct 11, 2011 at 02:13:36AM -0700, Lucian Holland wrote: > > I''ve been pondering something not dissimiliar to this recently, a sort > of "config introspection", for some of our live services (e.g. load > balancers). My initial thoughts were geared around collected resources, > but I''m not sure the model works well for them. We already use puppet > manifests to provide control info to mcollective, so feeding mcollective > discovery back into puppet might be an interesting way to do some of > this information gathering. The scope of what we''re thinking about is a > fair bit smaller than the sort of holistic, whole system stuff Lucian is > talking about though. >(Hi Stuart!) Isn''t this essentially what http://www.devco.net/archives/2010/09/18/puppet_search_engine_with_mcollective.php is doing using mcollective registration? Although it''s very cool I wonder about having another fact store at another level of indirection from the original information (machine properties -> puppet -> yaml -> mcollective -> mongodb) - it feels a bit like a cache hierarchy without enough in the way of proper cache management. But I suppose that''s a soluble problem with a bit of work. Lucian -- Lucian Holland Sent with Sparrow (http://www.sparrowmailapp.com/?sig) -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Hi Lucian, If you are interested I''ll be doing a demo in London next week: http://pug-london-oct-2011-eorg.eventbrite.com/ ken. On Wed, Oct 12, 2011 at 3:23 PM, Lucian Holland <lucian@symposion.co.uk> wrote:> Wow, thanks for this - there''s a huge amount in there. It will take me a > while to get unpick it all (I''m still a puppet novice really!) but that > looks like exactly the sort of thing I was talking about. Will have a play > and see how I get on! > Lucian > -- > Lucian Holland > Sent with Sparrow > > On Tuesday, 11 October 2011 at 17:25, Ken Barber wrote: > > I''ve done some work on creating entire networks using defined > resources or 100% exported resources for the nodes you build. The > following is an example environment with classes included: > > https://github.com/kbarber/puppet-onedemo/tree/master/manifests > > If you take a look at the nodes.pp you can see my use of app_* cases. > The resources are defined in the ''apps'' module path ... and they all > use the pattern described in: > > https://github.com/kbarber/puppet-onedemo/blob/master/manifests/solutions.pp > > Here I have a wrapper for launching vm''s using an OpenNebula resource, > and I handle exporting configurations for each node. > > Its not perfect and lacks time-based orchestration which is the major > hick-up for such a solution. Not to mention that you would have to > have stored configurations scale very well ... but if you keep the > layer you export down to a minimum little information needs to live in > the stored configurations themselves. So largely this is a proof of > concept. > > The app_stuck example actually is meant to build a complete working > ''pastie'' style application - with databases, app servers and load > balancers. > > Generally though - this topic is a hot discussion these days I''ve been > having with quite a few individuals. > > ken. > > On Tue, Oct 11, 2011 at 10:13 AM, Lucian Holland <lucian@symposion.co.uk> > wrote: > > Hi All, > > I''ve been exploring puppet and mcollective recently and I was > wondering if people here might be able to point me towards some > information on a potential use case. It seems like puppet is primarily > used at the moment as a tool for managing individual machine > configurations (''What do I need for a web server in production?''). At > the same time there is work going into imperative, distributed > solutions for provisioning machines and getting them set up with > puppet, either based on mcollective or custom built with tools like > fog (e.g. the new cloud provisioner). I''m wondering, though, has > anyone gone a step further and used the puppet declarative approach to > define the structure of a whole network? This would be particularly > useful for local CI environments when combined with virtualisation. > > I''m assuming that a start point would either be exported resources or > (better, in my opinion) something vaguely like this - > http://www.devco.net/archives/2010/09/18/puppet_search_engine_with_mcollective.php > - using fact-based discovery to find appropriate services to link up > machines together. But the next logical step would be to go beyond > simply querying the state of existing resources ("what is the ip of > the database?") and say something more like "Ensure there is a > postgres 9.1 database running on my local subnet". I accept that this > could be a bit terrifying for a production environment, but in a local/ > testing environment I could see it being hugely helpful. Clearly it > relies on puppet itself being able to control a variety of > virtualisation options, and perhaps a richer array of network service > controls as well. > > Is this a crazy idea? Has anyone tried it? Or would sticking to e.g. > TheForeman be a better plan (I have only looked briefly at it, not > really tried it yet - at first glance it still seems a bit more > imperative than what I''m suggesting + I get the impression that it > relies on DNS to glue a lot of things together, which isn''t > necessarily so appropriate in a heavily virtualised environment. > > Lucian > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
----- Original Message -----> > On Wednesday, 12 October 2011 at 13:44, Stu Teasdale wrote: > > > > On Tue, Oct 11, 2011 at 02:13:36AM -0700, Lucian Holland wrote: > > I''ve been pondering something not dissimiliar to this recently, a > sort of "config introspection", for some of our live services (e.g. load > balancers). My initial thoughts were geared around collected > resources, but I''m not sure the model works well for them. We already use puppet > manifests to provide control info to mcollective, so feeding > mcollective discovery back into puppet might be an interesting way to do some of > this information gathering. The scope of what we''re thinking about is > a fair bit smaller than the sort of holistic, whole system stuff Lucian > is talking about though. > > > (Hi Stuart!) Isn''t this essentially what > http://www.devco.net/archives/2010/09/18/puppet_search_engine_with_mcollective.php > is doing using mcollective registration? Although it''s very cool I > wonder about having another fact store at another level of > indirection from the original information (machine properties -> > puppet -> yaml -> mcollective -> mongodb) - it feels a bit like a > cache hierarchy without enough in the way of proper cache > management. But I suppose that''s a soluble problem with a bit of > work.the registration based search only goes a small part of the way, its not nearly far enough. It''s interesting and solves a subset of problems and a quick win but its far from complete. If you need application level awareness where applications are spread over a group of machines and dynamically managed through elastic auto scalers you need a lot more than just that. Luke has done some prototypes around moving to a application view rather than a node view [1][2], not sure what conclusions can be made from the existance of these repos though. You really do need to also include things like resource states (puppet resources) service states (monitoring) and weather information (latency between points) to make the kinds of decisions cloud managing needs.[3] Much of this is solvable by plugging in observers, reading states from monitoring etc and exposing it through some querying interface. But it''s pretty complex especially as there''s so few standards in this space. You may have all the data but no tools that can query it and understand the results. You may have tools but you need a solid expressive way for automated systems to reason out what is the best next step etc. I think we''re some way off from seeing a real generic answer to this problem - but its totally possible to build your own now, the information is there and tools like mcollective allows you to keep that up to date and relevant. [1] https://github.com/lak/Puppet-Platform-Manager [2] https://github.com/lak/puppet-environment-manager [3] http://www.devco.net/archives/2011/05/04/monitor_framework_minimal_configuration.php -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.