Hi, Has anyone given thought to the best way to maintain multiple versions of a manifest in a way that allows a client to specify which version it would like to use? I actually thought that''s what config_version was for until I read the docs...boo hoo. Seems like it would take some hacking. I think it would be nice though, giving more control over when a client would get a newly pushed manifest while allowing puppetd to continue running (instead of using puppet command directly with no client daemon). -scott --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I use environments for that, my environments are build from a selection of modules - e.g.: i have an env called puppetmasters, and the moduledir has the following links: apache2->/etc/puppet/modules/stable/services/apache2_0.2/ collectd->/etc/puppet/modules/stable/services/collectd_0.1/ host-base->/etc/puppet/modules/stable/services/host-base_0.13/ host-puppetmaster->/etc/puppet/modules/stable/host-puppetmaster_0.23/ monit->/etc/puppet/modules/stable/services/monit_0.2/ redhat->/etc/puppet/modules/stable/services/redhat_0.13/ sendmail->/etc/puppet/modules/stable/services/sendmail_0.14/ ssh->/etc/puppet/modules/stable/services/ssh_0.2/ subversion->/etc/puppet/modules/stable/services/subversion_0.11/ sudo->/etc/puppet/modules/stable/services/sudo_0.1/ syslog-ng->/etc/puppet/modules/stable/services/syslog-ng_0.13/ this allow me to reuse the modules in different environments, and have as many environments per different use cases. the version of each module could be changed per environment. Hope this helps, Ohad On Tue, Nov 17, 2009 at 10:47 AM, Scott Smith <scott@ohlol.net> wrote:> > Hi, > > Has anyone given thought to the best way to maintain multiple versions > of a manifest in a way that allows a client to specify which version it > would like to use? I actually thought that''s what config_version was for > until I read the docs...boo hoo. > > Seems like it would take some hacking. I think it would be nice though, > giving more control over when a client would get a newly pushed manifest > while allowing puppetd to continue running (instead of using puppet > command directly with no client daemon). > > -scott > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
2009/11/17 Scott Smith <scott@ohlol.net>:> > Has anyone given thought to the best way to maintain multiple versions > of a manifest in a way that allows a client to specify which version it > would like to use? I actually thought that''s what config_version was for > until I read the docs...boo hoo. > > Seems like it would take some hacking. I think it would be nice though, > giving more control over when a client would get a newly pushed manifest > while allowing puppetd to continue running (instead of using puppet > command directly with no client daemon).I use tags and environments for this: http://reductivelabs.com/trac/puppet/wiki/UsingTags http://reductivelabs.com/trac/puppet/wiki/UsingMultipleEnvironments Also being added in the next major release - currently code-named Rowlf - is module metadata. You can see the proposed code at: http://github.com/lak/puppet/commit/3896288ff56b44ae6f5e1f0c2a8adc995a458f76 This is just an initial cut but should let you see what could be done. Regards James Turnbull -- Author of: * Pro Linux System Administration (http://tinyurl.com/linuxadmin) * Pulling Strings with Puppet (http://tinyurl.com/pupbook) * Pro Nagios 2.0 (http://tinyurl.com/pronagios) * Hardening Linux (http://tinyurl.com/hardeninglinux) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
James Turnbull wrote:> 2009/11/17 Scott Smith <scott@ohlol.net>: >> Has anyone given thought to the best way to maintain multiple versions >> of a manifest in a way that allows a client to specify which version it >> would like to use? I actually thought that''s what config_version was for >> until I read the docs...boo hoo. >> >> Seems like it would take some hacking. I think it would be nice though, >> giving more control over when a client would get a newly pushed manifest >> while allowing puppetd to continue running (instead of using puppet >> command directly with no client daemon). > > I use tags and environments for this: > > http://reductivelabs.com/trac/puppet/wiki/UsingTags > http://reductivelabs.com/trac/puppet/wiki/UsingMultipleEnvironments >Yeah, I considered environments. I''d just rather not have to change my puppetmasters'' puppet.conf when I want to deploy a new manifest version to them. It wouldn''t be *difficult*; I''m just concerned with the puppet.conf becoming unwieldy after a while.> Also being added in the next major release - currently code-named > Rowlf - is module metadata. You can see the proposed code at: > > http://github.com/lak/puppet/commit/3896288ff56b44ae6f5e1f0c2a8adc995a458f76 > > This is just an initial cut but should let you see what could be done. >Hmm, interesting. Thanks, Ohad & James! -scott --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
never change the puppet.conf manually :) Maybe this can help a bit: http://github.com/ohadlevy/puppet-multipuppetmaster cheers, Ohad On Tue, Nov 17, 2009 at 11:20 AM, Scott Smith <scott@ohlol.net> wrote:> > J > Yeah, I considered environments. I''d just rather not have to change my > puppetmasters'' puppet.conf when I want to deploy a new manifest version > to them. It wouldn''t be *difficult*; I''m just concerned with the > puppet.conf becoming unwieldy after a while. > > > Also being added in the next major release - currently code-named > > Rowlf - is module metadata. You can see the proposed code at: > > > > > http://github.com/lak/puppet/commit/3896288ff56b44ae6f5e1f0c2a8adc995a458f76 > > > > This is just an initial cut but should let you see what could be done. > > > > Hmm, interesting. > > Thanks, Ohad & James! > > -scott > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Ohad Levy wrote:> never change the puppet.conf manually :) >Oh, no. We have some pretty cool deployment tools that I use to deploy puppetmasters. I can also use them to update manifests as if they were config updates to any other app. We have something like 4k or so hosts in prod (and another 1k or so in qa/dev) so it''s pretty much a necessity =) It also allows me to enforce a true development lifecycle model for Puppet. The manifests, configs, and everything else really do look like any other app we develop/maintain.> Maybe this can help a bit: > http://github.com/ohadlevy/puppet-multipuppetmaster >I''ll check it out, thanks! -scott --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Tue, Nov 17, 2009 at 11:33 AM, Scott Smith <scott@ohlol.net> wrote:> > > Oh, no. We have some pretty cool deployment tools that I use to deploy > puppetmasters. I can also use them to update manifests as if they were > config updates to any other app. >you are not using puppet to manage your puppetmasters?> > We have something like 4k or so hosts in prod (and another 1k or so in > qa/dev) so it''s pretty much a necessity =) >I know the feeling> > It also allows me to enforce a true development lifecycle model for > Puppet. The manifests, configs, and everything else really do look like > any other app we develop/maintain. >This is the main reason why we''ve started using environments, this was one of the only ways we could ensure that a host manifest will change only if it was planned to release. the testing process was also simplified a lot, as you can just create a test env with the same links (actually puppet does it all for you) but you can just play with the links to point to newer versions of your modules. on a side note, if you are willing, I''m interested in the performance of Foreman (http://theforeman.org) in a large infrastructure (for the Puppet reports, inventory etc) , I''ll appreciate it if you are willing to provide feedback cheers, Ohad --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I have series of module servers that serve modules to the host. I don''t use puppetmasters though. In a nutshell, I have a manifest server called the ''puppeteer'' that generates manifests based on meta data stored in a database. Well, it will be in database soon. For now it''s in XML files. The meta data contains everything from modules and what version of the modules a host should have as well as things like variables, virtual resources, resource defaults and classes. When it''s time for a host to do a puppet run, the puppeteer generates a manifest and puts an entry in to a queue for that host. A service on the host, called ''puppetstring'' polls the queue every few seconds for work. When there''s an entry in the queue for it, it pulls down its data (the manifest and modules it needs). It syncs up the modules and the appropriate version from the local module servers (puppetcache servers) and executes the manifest. This awards me much flexibility. Basically my modules rarely contain data, just structure. The contents of files, and even entire resources themselves can be controlled through a web interface and be deployed on the host, site or global level. This also allows me to do automated rollouts where I have a sequence of hosts do runs and based on their success or failure, I can determine whether or not to move on to the next step. This also allows me to allow certain control over specific machines and resources to end users. My database engineers can, for instance, control the contents of the tnsnames.ora files on all Oracle clients through a web interface. I can also put logic in to my scheduling of runs based on external criteria. For instance, if the cluster has a lot of work to do for the next two hours, don''t do puppet runs every thirty minutes but instead every hour to reduce the CPU load. Or maybe don''t do a run on a cluster node if more than 40% of the cluster nodes are currently doing runs. The web interface has yet to actually be built and I''m really sure whether or not I should extend on The Foreman or role out my own, but that''s another topic. I''ve promised several people a blog post on all this and one day I''ll deliver on that promise :) On Nov 16, 2009, at 9:43 PM, Ohad Levy wrote:> > > On Tue, Nov 17, 2009 at 11:33 AM, Scott Smith <scott@ohlol.net> wrote: > > > Oh, no. We have some pretty cool deployment tools that I use to deploy > puppetmasters. I can also use them to update manifests as if they were > config updates to any other app. > you are not using puppet to manage your puppetmasters? > > > We have something like 4k or so hosts in prod (and another 1k or so in > qa/dev) so it''s pretty much a necessity =) > I know the feeling > > It also allows me to enforce a true development lifecycle model for > Puppet. The manifests, configs, and everything else really do look like > any other app we develop/maintain. > This is the main reason why we''ve started using environments, this was one of the only ways we could ensure that a host manifest will change only if it was planned to release. > the testing process was also simplified a lot, as you can just create a test env with the same links (actually puppet does it all for you) but you can just play with the links to point to newer versions of your modules. > > on a side note, if you are willing, I''m interested in the performance of Foreman (http://theforeman.org) in a large infrastructure (for the Puppet reports, inventory etc) , I''ll appreciate it if you are willing to provide feedback > > cheers, > Ohad > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Ohad Levy wrote:> you are not using puppet to manage your puppetmasters? > >Nah, we have a client/server deployment app that performs two primary functions: - allows us to manage software deployments to specific hosts (or a range) from a shell (obviously could be web based too) - maintains & displays the state of all hosts it knows about (what is deployed to it, if it''s running, etc) I''m using this to actively deploy puppetmasters. So if there is a new version of puppet, facter or w/e, I simply use these tools to update when ready (I''m not doing the same for puppetd right now due to some technical limitations). App packages & configs are indirectly (via gitweb) fetched from git on deploy time by an agent on each host. In addition, in our manifest lifecycle, we will deploy them in the same manner (a new config will point to a new set of manifests that the deploy fetches). Dev -> QA -> Prod will all have their own configs and manifests. So in this sense, there won''t be seperate Puppet environments for each stage. It''s kind of hard for me to explain in this format, but it would make sense if you saw command output =) Actually, this reminds me of something else I was considering - seperate manifests bundled with deployed apps that are run in a post-deploy step. Instead of having to maintain a list of what specific system config changes a host or piece of software needs based on some arbitrary knowledge (usually in someone''s head...you know how that goes!), the engineers would be able to include their own Puppet manifests that get run at deploy time via `puppet''. I know it could easily be included in the host manifests, but we manage a large number of apps (no, really...A LOT!). Keeping track of that matrix is kind of overwhelming. So I''m thinking puppetd manages generic host stuff, and the deploy manifest handles the specifics. I think the two trickiest pieces would be 1) ensuring seperation between deployment manifests and host manifests (we''d review pre-deployment), and 2) reporting changes from a deployed puppet run (since they won''t communicate with puppetmasterd). I could use tags for this, but it could require many more Puppetmaster manifest deployments to prod. I want to keep them as static as possible.> This is the main reason why we''ve started using environments, this was > one of the only ways we could ensure that a host manifest will change > only if it was planned to release. > the testing process was also simplified a lot, as you can just create a > test env with the same links (actually puppet does it all for you) but > you can just play with the links to point to newer versions of your modules. >Cool.> on a side note, if you are willing, I''m interested in the performance of > Foreman (http://theforeman.org) in a large infrastructure (for the > Puppet reports, inventory etc) , I''ll appreciate it if you are willing > to provide feedback >I would love to, and actually showed it to some of my colleagues after seeing your presentation at Puppet Camp. Unfortunately we already have some inventory tools that others here have put a lot of development time into =( -scott --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Hi Carl, One thing that I didn''t understand, does it means that the nodes you have get only their part of the manifest? or do they get all manifest and apply the right parts? the main issue I have with your approach is that a lot of information is available on the client side (even if there is no node specific information). as far for features for foreman, let me know what you had in mind... cheers, Ohad On Tue, Nov 17, 2009 at 12:30 PM, Carl Caum <carl.caum@gmail.com> wrote:> I have series of module servers that serve modules to the host. I don''t > use puppetmasters though. In a nutshell, I have a manifest server called > the ''puppeteer'' that generates manifests based on meta data stored in a > database. Well, it will be in database soon. For now it''s in XML files. > The meta data contains everything from modules and what version of the > modules a host should have as well as things like variables, virtual > resources, resource defaults and classes. When it''s time for a host to do a > puppet run, the puppeteer generates a manifest and puts an entry in to a > queue for that host. A service on the host, called ''puppetstring'' polls the > queue every few seconds for work. When there''s an entry in the queue for > it, it pulls down its data (the manifest and modules it needs). It syncs up > the modules and the appropriate version from the local module servers > (puppetcache servers) and executes the manifest. > > This awards me much flexibility. Basically my modules rarely contain data, > just structure. The contents of files, and even entire resources themselves > can be controlled through a web interface and be deployed on the host, site > or global level. This also allows me to do automated rollouts where I have > a sequence of hosts do runs and based on their success or failure, I can > determine whether or not to move on to the next step. This also allows me > to allow certain control over specific machines and resources to end users. > My database engineers can, for instance, control the contents of the > tnsnames.ora files on all Oracle clients through a web interface. I can > also put logic in to my scheduling of runs based on external criteria. For > instance, if the cluster has a lot of work to do for the next two hours, > don''t do puppet runs every thirty minutes but instead every hour to reduce > the CPU load. Or maybe don''t do a run on a cluster node if more than 40% of > the cluster nodes are currently doing runs. > > The web interface has yet to actually be built and I''m really sure whether > or not I should extend on The Foreman or role out my own, but that''s another > topic. > > I''ve promised several people a blog post on all this and one day I''ll > deliver on that promise :) > > On Nov 16, 2009, at 9:43 PM, Ohad Levy wrote: > > > > On Tue, Nov 17, 2009 at 11:33 AM, Scott Smith <scott@ohlol.net> wrote: > >> >> >> Oh, no. We have some pretty cool deployment tools that I use to deploy >> puppetmasters. I can also use them to update manifests as if they were >> config updates to any other app. >> > you are not using puppet to manage your puppetmasters? > > >> >> We have something like 4k or so hosts in prod (and another 1k or so in >> qa/dev) so it''s pretty much a necessity =) >> > I know the feeling > >> >> It also allows me to enforce a true development lifecycle model for >> Puppet. The manifests, configs, and everything else really do look like >> any other app we develop/maintain. >> > This is the main reason why we''ve started using environments, this was one > of the only ways we could ensure that a host manifest will change only if it > was planned to release. > the testing process was also simplified a lot, as you can just create a > test env with the same links (actually puppet does it all for you) but you > can just play with the links to point to newer versions of your modules. > > on a side note, if you are willing, I''m interested in the performance of > Foreman (http://theforeman.org) in a large infrastructure (for the Puppet > reports, inventory etc) , I''ll appreciate it if you are willing to provide > feedback > > cheers, > Ohad > > > > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
No, the manifest is generated per host on the fly. So each manifest only has
access to the data for it specifically. That''s one advantage of my
approach. I maintain the security model of using puppetmasters, but I
distribute the work load to the nodes themselves.
The manifest that''s delivered to the host looks like this:
variables
resource defaults
virtual resources
node host {
classes
}
On Nov 16, 2009, at 10:58 PM, Ohad Levy wrote:
> Hi Carl,
>
> One thing that I didn''t understand, does it means that the nodes
you have get only their part of the manifest? or do they get all manifest and
apply the right parts?
>
> the main issue I have with your approach is that a lot of information is
available on the client side (even if there is no node specific information).
>
> as far for features for foreman, let me know what you had in mind...
>
> cheers,
> Ohad
>
> On Tue, Nov 17, 2009 at 12:30 PM, Carl Caum <carl.caum@gmail.com>
wrote:
> I have series of module servers that serve modules to the host. I
don''t use puppetmasters though. In a nutshell, I have a manifest
server called the ''puppeteer'' that generates manifests based
on meta data stored in a database. Well, it will be in database soon. For now
it''s in XML files. The meta data contains everything from modules and
what version of the modules a host should have as well as things like variables,
virtual resources, resource defaults and classes. When it''s time for a
host to do a puppet run, the puppeteer generates a manifest and puts an entry in
to a queue for that host. A service on the host, called
''puppetstring'' polls the queue every few seconds for work.
When there''s an entry in the queue for it, it pulls down its data (the
manifest and modules it needs). It syncs up the modules and the appropriate
version from the local module servers (puppetcache servers) and executes the
manifest.
>
> This awards me much flexibility. Basically my modules rarely contain data,
just structure. The contents of files, and even entire resources themselves can
be controlled through a web interface and be deployed on the host, site or
global level. This also allows me to do automated rollouts where I have a
sequence of hosts do runs and based on their success or failure, I can determine
whether or not to move on to the next step. This also allows me to allow
certain control over specific machines and resources to end users. My database
engineers can, for instance, control the contents of the tnsnames.ora files on
all Oracle clients through a web interface. I can also put logic in to my
scheduling of runs based on external criteria. For instance, if the cluster has
a lot of work to do for the next two hours, don''t do puppet runs every
thirty minutes but instead every hour to reduce the CPU load. Or maybe
don''t do a run on a cluster node if more than 40% of the cluster nodes
are currently doing runs.
>
> The web interface has yet to actually be built and I''m really sure
whether or not I should extend on The Foreman or role out my own, but
that''s another topic.
>
> I''ve promised several people a blog post on all this and one day
I''ll deliver on that promise :)
>
> On Nov 16, 2009, at 9:43 PM, Ohad Levy wrote:
>
>>
>>
>> On Tue, Nov 17, 2009 at 11:33 AM, Scott Smith <scott@ohlol.net>
wrote:
>>
>>
>> Oh, no. We have some pretty cool deployment tools that I use to deploy
>> puppetmasters. I can also use them to update manifests as if they were
>> config updates to any other app.
>> you are not using puppet to manage your puppetmasters?
>>
>>
>> We have something like 4k or so hosts in prod (and another 1k or so in
>> qa/dev) so it''s pretty much a necessity =)
>> I know the feeling
>>
>> It also allows me to enforce a true development lifecycle model for
>> Puppet. The manifests, configs, and everything else really do look like
>> any other app we develop/maintain.
>> This is the main reason why we''ve started using environments,
this was one of the only ways we could ensure that a host manifest will change
only if it was planned to release.
>> the testing process was also simplified a lot, as you can just create a
test env with the same links (actually puppet does it all for you) but you can
just play with the links to point to newer versions of your modules.
>>
>> on a side note, if you are willing, I''m interested in the
performance of Foreman (http://theforeman.org) in a large infrastructure (for
the Puppet reports, inventory etc) , I''ll appreciate it if you are
willing to provide feedback
>>
>> cheers,
>> Ohad
>>
>>
>
>
>
>
>
> >
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---
that sounds interesting, how does it work? in a way, you are reproducing the role of the puppetmaser... On Tue, Nov 17, 2009 at 1:20 PM, Carl Caum <carl.caum@gmail.com> wrote:> No, the manifest is generated per host on the fly. So each manifest only > has access to the data for it specifically. That''s one advantage of my > approach. I maintain the security model of using puppetmasters, but I > distribute the work load to the nodes themselves. > > The manifest that''s delivered to the host looks like this: > > variables > resource defaults > virtual resources > node host { > classes > } > > On Nov 16, 2009, at 10:58 PM, Ohad Levy wrote: > > Hi Carl, > > One thing that I didn''t understand, does it means that the nodes you have > get only their part of the manifest? or do they get all manifest and apply > the right parts? > > the main issue I have with your approach is that a lot of information is > available on the client side (even if there is no node specific > information). > > as far for features for foreman, let me know what you had in mind... > > cheers, > Ohad > > On Tue, Nov 17, 2009 at 12:30 PM, Carl Caum <carl.caum@gmail.com> wrote: > >> I have series of module servers that serve modules to the host. I don''t >> use puppetmasters though. In a nutshell, I have a manifest server called >> the ''puppeteer'' that generates manifests based on meta data stored in a >> database. Well, it will be in database soon. For now it''s in XML files. >> The meta data contains everything from modules and what version of the >> modules a host should have as well as things like variables, virtual >> resources, resource defaults and classes. When it''s time for a host to do a >> puppet run, the puppeteer generates a manifest and puts an entry in to a >> queue for that host. A service on the host, called ''puppetstring'' polls the >> queue every few seconds for work. When there''s an entry in the queue for >> it, it pulls down its data (the manifest and modules it needs). It syncs up >> the modules and the appropriate version from the local module servers >> (puppetcache servers) and executes the manifest. >> >> This awards me much flexibility. Basically my modules rarely contain >> data, just structure. The contents of files, and even entire resources >> themselves can be controlled through a web interface and be deployed on the >> host, site or global level. This also allows me to do automated rollouts >> where I have a sequence of hosts do runs and based on their success or >> failure, I can determine whether or not to move on to the next step. This >> also allows me to allow certain control over specific machines and resources >> to end users. My database engineers can, for instance, control the contents >> of the tnsnames.ora files on all Oracle clients through a web interface. I >> can also put logic in to my scheduling of runs based on external criteria. >> For instance, if the cluster has a lot of work to do for the next two >> hours, don''t do puppet runs every thirty minutes but instead every hour to >> reduce the CPU load. Or maybe don''t do a run on a cluster node if more than >> 40% of the cluster nodes are currently doing runs. >> >> The web interface has yet to actually be built and I''m really sure whether >> or not I should extend on The Foreman or role out my own, but that''s another >> topic. >> >> I''ve promised several people a blog post on all this and one day I''ll >> deliver on that promise :) >> >> On Nov 16, 2009, at 9:43 PM, Ohad Levy wrote: >> >> >> >> On Tue, Nov 17, 2009 at 11:33 AM, Scott Smith <scott@ohlol.net> wrote: >> >>> >>> >>> Oh, no. We have some pretty cool deployment tools that I use to deploy >>> puppetmasters. I can also use them to update manifests as if they were >>> config updates to any other app. >>> >> you are not using puppet to manage your puppetmasters? >> >> >>> >>> We have something like 4k or so hosts in prod (and another 1k or so in >>> qa/dev) so it''s pretty much a necessity =) >>> >> I know the feeling >> >>> >>> It also allows me to enforce a true development lifecycle model for >>> Puppet. The manifests, configs, and everything else really do look like >>> any other app we develop/maintain. >>> >> This is the main reason why we''ve started using environments, this was one >> of the only ways we could ensure that a host manifest will change only if it >> was planned to release. >> the testing process was also simplified a lot, as you can just create a >> test env with the same links (actually puppet does it all for you) but you >> can just play with the links to point to newer versions of your modules. >> >> on a side note, if you are willing, I''m interested in the performance of >> Foreman (http://theforeman.org) in a large infrastructure (for the Puppet >> reports, inventory etc) , I''ll appreciate it if you are willing to provide >> feedback >> >> cheers, >> Ohad >> >> >> >> >> >> > > > > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I am reproducing the role of the puppetmaster, but I have more overall control and I''m pushing the work of compiling the catalogue off to the nodes. Currently I use a series of XML files until I''m ready to move them to a database. My directory structure looks like this: /configuration.xml /os/OS/configuration.xml /sites/SITECODE/configuration.xml /sites/SITECODE/hosts/HOST/configuration.xml When the puppeteer determines it''s time to do a run on a host, it reads in what site code the host is in from a database. It then takes the site code and passes it to the manifest generator function to parse the appropriate XML files. As I read in a host, I convert the XML entries into a hash and merge them with right side precedence in the following order: global->os->site->host So a host only gets the appropriate data for that specific host, which maintains security levels. Also, any global resource that''s defined (virtual resource, resource defaults, variables) can be overwritten at the os, site and host levels. I can also disable virtual resources and resource defaults at a lower level than the level it was defined in. This allows my actual puppet code to be more generic and contain far less data. This way I feel safe with the code living on the hosts instead of the puppetmaster. Of course this would never work in some industries like banking, but it works great for us. It''s going to allow us to role out changes and puppetize existing boxes much faster. On Nov 17, 2009, at 12:16 AM, Ohad Levy wrote:> that sounds interesting, how does it work? in a way, you are reproducing the role of the puppetmaser... > > > On Tue, Nov 17, 2009 at 1:20 PM, Carl Caum <carl.caum@gmail.com> wrote: > No, the manifest is generated per host on the fly. So each manifest only has access to the data for it specifically. That''s one advantage of my approach. I maintain the security model of using puppetmasters, but I distribute the work load to the nodes themselves. > > The manifest that''s delivered to the host looks like this: > > variables > resource defaults > virtual resources > node host { > classes > } > > On Nov 16, 2009, at 10:58 PM, Ohad Levy wrote: > >> Hi Carl, >> >> One thing that I didn''t understand, does it means that the nodes you have get only their part of the manifest? or do they get all manifest and apply the right parts? >> >> the main issue I have with your approach is that a lot of information is available on the client side (even if there is no node specific information). >> >> as far for features for foreman, let me know what you had in mind... >> >> cheers, >> Ohad >> >> On Tue, Nov 17, 2009 at 12:30 PM, Carl Caum <carl.caum@gmail.com> wrote: >> I have series of module servers that serve modules to the host. I don''t use puppetmasters though. In a nutshell, I have a manifest server called the ''puppeteer'' that generates manifests based on meta data stored in a database. Well, it will be in database soon. For now it''s in XML files. The meta data contains everything from modules and what version of the modules a host should have as well as things like variables, virtual resources, resource defaults and classes. When it''s time for a host to do a puppet run, the puppeteer generates a manifest and puts an entry in to a queue for that host. A service on the host, called ''puppetstring'' polls the queue every few seconds for work. When there''s an entry in the queue for it, it pulls down its data (the manifest and modules it needs). It syncs up the modules and the appropriate version from the local module servers (puppetcache servers) and executes the manifest. >> >> This awards me much flexibility. Basically my modules rarely contain data, just structure. The contents of files, and even entire resources themselves can be controlled through a web interface and be deployed on the host, site or global level. This also allows me to do automated rollouts where I have a sequence of hosts do runs and based on their success or failure, I can determine whether or not to move on to the next step. This also allows me to allow certain control over specific machines and resources to end users. My database engineers can, for instance, control the contents of the tnsnames.ora files on all Oracle clients through a web interface. I can also put logic in to my scheduling of runs based on external criteria. For instance, if the cluster has a lot of work to do for the next two hours, don''t do puppet runs every thirty minutes but instead every hour to reduce the CPU load. Or maybe don''t do a run on a cluster node if more than 40% of the cluster nodes are currently doing runs. >> >> The web interface has yet to actually be built and I''m really sure whether or not I should extend on The Foreman or role out my own, but that''s another topic. >> >> I''ve promised several people a blog post on all this and one day I''ll deliver on that promise :) >> >> On Nov 16, 2009, at 9:43 PM, Ohad Levy wrote: >> >>> >>> >>> On Tue, Nov 17, 2009 at 11:33 AM, Scott Smith <scott@ohlol.net> wrote: >>> >>> >>> Oh, no. We have some pretty cool deployment tools that I use to deploy >>> puppetmasters. I can also use them to update manifests as if they were >>> config updates to any other app. >>> you are not using puppet to manage your puppetmasters? >>> >>> >>> We have something like 4k or so hosts in prod (and another 1k or so in >>> qa/dev) so it''s pretty much a necessity =) >>> I know the feeling >>> >>> It also allows me to enforce a true development lifecycle model for >>> Puppet. The manifests, configs, and everything else really do look like >>> any other app we develop/maintain. >>> This is the main reason why we''ve started using environments, this was one of the only ways we could ensure that a host manifest will change only if it was planned to release. >>> the testing process was also simplified a lot, as you can just create a test env with the same links (actually puppet does it all for you) but you can just play with the links to point to newer versions of your modules. >>> >>> on a side note, if you are willing, I''m interested in the performance of Foreman (http://theforeman.org) in a large infrastructure (for the Puppet reports, inventory etc) , I''ll appreciate it if you are willing to provide feedback >>> >>> cheers, >>> Ohad >>> >>> >> >> >> >> >> >> > > > > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Nov 16, 2009, at 10:20 PM, Scott Smith wrote:> Yeah, I considered environments. I''d just rather not have to change my > puppetmasters'' puppet.conf when I want to deploy a new manifest version > to them. It wouldn''t be *difficult*; I''m just concerned with the > puppet.conf becoming unwieldy after a while.I use environments and never have to touch the puppet.conf on the clients. All of the nodes are stored in LDAP with an attribute called "environment". The puppetmaster will use that in choosing which manifests to apply. -- Rob McBroom <http://www.skurfer.com/> It''s not that I think guns, drugs, prostitution, swimming, eating and reading should be legal. It''s just that no one on Earth has the authority to make them illegal. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Rob McBroom wrote:> On Nov 16, 2009, at 10:20 PM, Scott Smith wrote: > >> Yeah, I considered environments. I''d just rather not have to change my >> puppetmasters'' puppet.conf when I want to deploy a new manifest version >> to them. It wouldn''t be *difficult*; I''m just concerned with the >> puppet.conf becoming unwieldy after a while. > > I use environments and never have to touch the puppet.conf on the clients. All of the nodes are stored in LDAP with an attribute called "environment". The puppetmaster will use that in choosing which manifests to apply. >Usng environments as a "version" of a manifest will require adding an environment to *Puppetmaster* every time I bump the manifest version. Not the client, but Puppetmaster. :) -scott -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=.
On Nov 17, 2009, at 6:01 PM, Scott Smith wrote:> Usng environments as a "version" of a manifest will require adding an > environment to *Puppetmaster* every time I bump the manifest version. > Not the client, but Puppetmaster. :)Well, true. I just have 3 environments. The "experimental" for testing manifests after changes, the "test" which gets applied to all dev/test/staging boxes, and the "prod" that gets applied to production servers. They''re all working copies of the same repo and are identical most of the time, except when testing something new. Sounds like you want something more complicated. -- Rob McBroom <http://www.skurfer.com/> Because it screws up the order in which people normally read text. Original message:> Why is it bad to top-post your reply?-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=.
On Wed, Nov 18, 2009 at 7:01 AM, Scott Smith <scott@ohlol.net> wrote:> > Usng environments as a "version" of a manifest will require adding an > environment to *Puppetmaster* every time I bump the manifest version. > Not the client, but Puppetmaster. :) >Generating it on the puppetmaster is not a big task, just a loop over the environment names :) - at least thats what I do. note that there is a bug with 0.24.x that if you change the modules in your env and you use passenger, than you must restart your puppetmaster instances (as it wont reload the manifests) Ohad -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=.
On Nov 18, 2009, at 12:23 AM, Ohad Levy wrote:> note that there is a bug with 0.24.x that if you change the modules in your env and you use passenger, than you must restart your puppetmaster instances (as it wont reload the manifests)I haven''t noticed any problems, but I''m not using modules at all. Does it only happen when you use them? -- Rob McBroom <http://www.skurfer.com/> It''s not that I think guns, drugs, prostitution, swimming, eating and reading should be legal. It''s just that no one on Earth has the authority to make them illegal. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=.
On Tue, Nov 17, 2009 at 11:23 PM, Ohad Levy <ohadlevy@gmail.com> wrote:> On Wed, Nov 18, 2009 at 7:01 AM, Scott Smith <scott@ohlol.net> wrote: > >> >> Usng environments as a "version" of a manifest will require adding an >> environment to *Puppetmaster* every time I bump the manifest version. >> Not the client, but Puppetmaster. :) >> > > Generating it on the puppetmaster is not a big task, just a loop over the > environment names :) - at least thats what I do. >I have an XML file that describes the relationship between filesystem paths and environment names for my VCS puppet repository. My puppetmasters sync to VCS at regular intervals, and generate the puppet.conf file environments subsection on the fly based upon the XML file and the filesystem. This way we can just add a new environment by creating the relevant path in VCS.> > note that there is a bug with 0.24.x that if you change the modules in your > env and you use passenger, than you must restart your puppetmaster instances > (as it wont reload the manifests) > > Ohad > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=. >-- nigel -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=.
On Wed, Nov 18, 2009 at 11:15 PM, Nigel Kersten <nigelk@google.com> wrote:> > Generating it on the puppetmaster is not a big task, just a loop over the >> environment names :) - at least thats what I do. >> > > I have an XML file that describes the relationship between filesystem paths > and environment names for my VCS puppet repository. > > My puppetmasters sync to VCS at regular intervals, and generate the > puppet.conf file environments subsection on the fly based upon the XML file > and the filesystem. > > This way we can just add a new environment by creating the relevant path in > VCS. >Yeah, we have something similar too, I didn''t mean that I keep a long string with all environments (they are auto generated via another process too). Ohad -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=.