Douglas Garstang
2011-Sep-21 20:20 UTC
[Puppet Users] Variable number of variables in external node output...
All, I have a situation where I need to get some fairly complex configuration files onto systems, and I''m wondering if puppet can even do this. Lets say that my external node script will go and source all the data it needs from an external database, and dump out all variables that the node will need. The relevant puppet module(s) will then have to inject these variables into templates to be deployed to the systems. Now, what if the number of variables that get dumped by the external node script is variable? And, what if these variables are broken into chunks that need to be split into different files on the remote system? How would I do such a complex thing in puppet? A lack of any type of looping constructs in puppet would seem to make this rather difficult. Doug. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Brian Gupta
2011-Sep-21 21:38 UTC
Re: [Puppet Users] Variable number of variables in external node output...
Puppet can do this. I would propose the following solution, but there may be other ways to do it. 1) Use a template. Templates have embedded ruby code and run on the clients. 2) Link to some library that allows you to query your database (You can use one or more require statements in your ruby code) 3) Embed ruby code to parse and format your variables into configuration See http://docs.puppetlabs.com/guides/templating.html for a bit more about templates. -Brian On Wed, Sep 21, 2011 at 4:20 PM, Douglas Garstang <doug.garstang@gmail.com>wrote:> All, > > I have a situation where I need to get some fairly complex > configuration files onto systems, and I''m wondering if puppet can even > do this. Lets say that my external node script will go and source all > the data it needs from an external database, and dump out all > variables that the node will need. The relevant puppet module(s) will > then have to inject these variables into templates to be deployed to > the systems. > > Now, what if the number of variables that get dumped by the external > node script is variable? And, what if these variables are broken into > chunks that need to be split into different files on the remote > system? How would I do such a complex thing in puppet? A lack of any > type of looping constructs in puppet would seem to make this rather > difficult. > > Doug. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- <http://aws.amazon.com/solutions/solution-providers/brandorr/> -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Aaron Grewell
2011-Sep-21 21:48 UTC
Re: [Puppet Users] Variable number of variables in external node output...
Remember too that if you can do it once (say with a define) Puppet will do it as many times as you like just by putting the data in an array. On Sep 21, 2011 2:39 PM, "Brian Gupta" <brian.gupta@brandorr.com> wrote:> Puppet can do this. I would propose the following solution, but there maybe> other ways to do it. > > 1) Use a template. Templates have embedded ruby code and run on theclients.> 2) Link to some library that allows you to query your database (You canuse> one or more require statements in your ruby code) > 3) Embed ruby code to parse and format your variables into configuration > > See http://docs.puppetlabs.com/guides/templating.html for a bit more about > templates. > > -Brian > > On Wed, Sep 21, 2011 at 4:20 PM, Douglas Garstang > <doug.garstang@gmail.com>wrote: > >> All, >> >> I have a situation where I need to get some fairly complex >> configuration files onto systems, and I''m wondering if puppet can even >> do this. Lets say that my external node script will go and source all >> the data it needs from an external database, and dump out all >> variables that the node will need. The relevant puppet module(s) will >> then have to inject these variables into templates to be deployed to >> the systems. >> >> Now, what if the number of variables that get dumped by the external >> node script is variable? And, what if these variables are broken into >> chunks that need to be split into different files on the remote >> system? How would I do such a complex thing in puppet? A lack of any >> type of looping constructs in puppet would seem to make this rather >> difficult. >> >> Doug. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> >> > > > -- > <http://aws.amazon.com/solutions/solution-providers/brandorr/> > > -- > You received this message because you are subscribed to the Google Groups"Puppet Users" group.> To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email topuppet-users+unsubscribe@googlegroups.com.> For more options, visit this group athttp://groups.google.com/group/puppet-users?hl=en.>-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Douglas Garstang
2011-Sep-21 23:26 UTC
Re: [Puppet Users] Variable number of variables in external node output...
Thanks, but I''m not sure templates will quite cut it. There is no way that I am aware of to iterate over all the variables obtained through an external node script, and put chunks into separate files. This falls outside the bounds of what embedded ruby can do. As for using define(), well, I am trying to separate the data from the manifest, and putting N number of defines in the manifests and passing values to it, breaks that. Doug. On Wed, Sep 21, 2011 at 2:48 PM, Aaron Grewell <aaron.grewell@gmail.com> wrote:> Remember too that if you can do it once (say with a define) Puppet will do > it as many times as you like just by putting the data in an array. > > On Sep 21, 2011 2:39 PM, "Brian Gupta" <brian.gupta@brandorr.com> wrote: >> Puppet can do this. I would propose the following solution, but there may >> be >> other ways to do it. >> >> 1) Use a template. Templates have embedded ruby code and run on the >> clients. >> 2) Link to some library that allows you to query your database (You can >> use >> one or more require statements in your ruby code) >> 3) Embed ruby code to parse and format your variables into configuration >> >> See http://docs.puppetlabs.com/guides/templating.html for a bit more about >> templates. >> >> -Brian >> >> On Wed, Sep 21, 2011 at 4:20 PM, Douglas Garstang >> <doug.garstang@gmail.com>wrote: >> >>> All, >>> >>> I have a situation where I need to get some fairly complex >>> configuration files onto systems, and I''m wondering if puppet can even >>> do this. Lets say that my external node script will go and source all >>> the data it needs from an external database, and dump out all >>> variables that the node will need. The relevant puppet module(s) will >>> then have to inject these variables into templates to be deployed to >>> the systems. >>> >>> Now, what if the number of variables that get dumped by the external >>> node script is variable? And, what if these variables are broken into >>> chunks that need to be split into different files on the remote >>> system? How would I do such a complex thing in puppet? A lack of any >>> type of looping constructs in puppet would seem to make this rather >>> difficult. >>> >>> Doug. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Puppet Users" group. >>> To post to this group, send email to puppet-users@googlegroups.com. >>> To unsubscribe from this group, send email to >>> puppet-users+unsubscribe@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/puppet-users?hl=en. >>> >>> >> >> >> -- >> <http://aws.amazon.com/solutions/solution-providers/brandorr/> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- Regards, Douglas Garstang http://www.linkedin.com/in/garstang Email: doug.garstang@gmail.com Cell: +1-805-340-5627 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Nigel Kersten
2011-Sep-21 23:34 UTC
Re: [Puppet Users] Variable number of variables in external node output...
On Wed, Sep 21, 2011 at 1:20 PM, Douglas Garstang <doug.garstang@gmail.com> wrote:> All, > > I have a situation where I need to get some fairly complex > configuration files onto systems, and I''m wondering if puppet can even > do this. Lets say that my external node script will go and source all > the data it needs from an external database, and dump out all > variables that the node will need. The relevant puppet module(s) will > then have to inject these variables into templates to be deployed to > the systems.Can you provide an example of a chunk of actual data and the desired end result to see if there''s a better alternative Doug? Do you have full control over the external data source and external node script? Are there other systems in place that already rely upon your existing data formats in the external source?> Now, what if the number of variables that get dumped by the external > node script is variable? And, what if these variables are broken into > chunks that need to be split into different files on the remote > system? How would I do such a complex thing in puppet? A lack of any > type of looping constructs in puppet would seem to make this rather > difficult. > > Doug. > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > >-- Nigel Kersten Product Manager, Puppet Labs Join us for PuppetConf <http://www.bit.ly/puppetconfsig> Sept 22/23 Portland, Oregon, USA. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Peter Meier
2011-Sep-22 06:19 UTC
Re: [Puppet Users] Variable number of variables in external node output...
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1> 1) Use a template. Templates have embedded ruby code and run on the clients.template() is a function and as any other function it will run on the server. ~pete -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk56028ACgkQbwltcAfKi3/e+wCfVcovUxo7vjzG4BD7XOeZLU/6 XlIAn231meTEH9REpxBBWxtcvSSedhMz =di4d -----END PGP SIGNATURE----- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
jcbollinger
2011-Sep-22 12:57 UTC
[Puppet Users] Re: Variable number of variables in external node output...
On Sep 21, 6:34 pm, Nigel Kersten <ni...@puppetlabs.com> wrote:> On Wed, Sep 21, 2011 at 1:20 PM, Douglas Garstang > > <doug.garst...@gmail.com> wrote: > > All, > > > I have a situation where I need to get some fairly complex > > configuration files onto systems, and I''m wondering if puppet can even > > do this. Lets say that my external node script will go and source all > > the data it needs from an external database, and dump out all > > variables that the node will need. The relevant puppet module(s) will > > then have to inject these variables into templates to be deployed to > > the systems. > > Can you provide an example of a chunk of actual data and the desired > end result to see if there''s a better alternative Doug?I second that request. The scenario presented is more a design concept than a problem description. Although I am confident that Puppet indeed can handle the scenario as presented, it may be that there are alternative designs that would accomplish the same objective in a simpler way.> Do you have full control over the external data source and external node script? > Are there other systems in place that already rely upon your existing > data formats in the external source?John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Douglas Garstang
2011-Sep-22 16:05 UTC
Re: [Puppet Users] Re: Variable number of variables in external node output...
On Thu, Sep 22, 2011 at 5:57 AM, jcbollinger <John.Bollinger@stjude.org> wrote:> > > On Sep 21, 6:34 pm, Nigel Kersten <ni...@puppetlabs.com> wrote: >> On Wed, Sep 21, 2011 at 1:20 PM, Douglas Garstang >> >> <doug.garst...@gmail.com> wrote: >> > All, >> >> > I have a situation where I need to get some fairly complex >> > configuration files onto systems, and I''m wondering if puppet can even >> > do this. Lets say that my external node script will go and source all >> > the data it needs from an external database, and dump out all >> > variables that the node will need. The relevant puppet module(s) will >> > then have to inject these variables into templates to be deployed to >> > the systems. >> >> Can you provide an example of a chunk of actual data and the desired >> end result to see if there''s a better alternative Doug? > > > I second that request. The scenario presented is more a design > concept than a problem description. Although I am confident that > Puppet indeed can handle the scenario as presented, it may be that > there are alternative designs that would accomplish the same objective > in a simpler way.Well, speaking somewhat generically still, we have an application that will need to run multiple times on a single system, and each instance of that running application will have it''s own config file, each with a variable number of items (they''re disk volumes). If the external node script returned YAML data like this: appX_inst1_vol1_name: vol1 appX_inst1_vol1_size: 1G appX_inst1_vol1_active: true appX_inst2_vol1_name: vol2 appX_inst2_vol1_size: 2G appX_inst2_vol1_active: false ... and so on. All the volumes for inst1 need to go into one config file, all the volumes for inst2 in another config file and so on. I don''t see a way that puppet can iterate over a variable number of items and split the data into multiple files. Doug. $vol_host1_inst1_ \ -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Christopher Wood
2011-Sep-22 16:14 UTC
Re: [Puppet Users] Re: Variable number of variables in external node output...
On Thu, Sep 22, 2011 at 09:05:32AM -0700, Douglas Garstang wrote:> On Thu, Sep 22, 2011 at 5:57 AM, jcbollinger <John.Bollinger@stjude.org> wrote: > > > > > > On Sep 21, 6:34 pm, Nigel Kersten <ni...@puppetlabs.com> wrote: > >> On Wed, Sep 21, 2011 at 1:20 PM, Douglas Garstang > >> > >> <doug.garst...@gmail.com> wrote: > >> > All, > >> > >> > I have a situation where I need to get some fairly complex > >> > configuration files onto systems, and I''m wondering if puppet can even > >> > do this. Lets say that my external node script will go and source all > >> > the data it needs from an external database, and dump out all > >> > variables that the node will need. The relevant puppet module(s) will > >> > then have to inject these variables into templates to be deployed to > >> > the systems. > >> > >> Can you provide an example of a chunk of actual data and the desired > >> end result to see if there''s a better alternative Doug? > > > > > > I second that request. The scenario presented is more a design > > concept than a problem description. Although I am confident that > > Puppet indeed can handle the scenario as presented, it may be that > > there are alternative designs that would accomplish the same objective > > in a simpler way. > > Well, speaking somewhat generically still, we have an application that > will need to run multiple times on a single system, and each instance > of that running application will have it''s own config file, each with > a variable number of items (they''re disk volumes). > > If the external node script returned YAML data like this: > > appX_inst1_vol1_name: vol1 > appX_inst1_vol1_size: 1G > appX_inst1_vol1_active: true > appX_inst2_vol1_name: vol2 > appX_inst2_vol1_size: 2G > appX_inst2_vol1_active: false > > ... and so on. All the volumes for inst1 need to go into one config > file, all the volumes for inst2 in another config file and so on. I > don''t see a way that puppet can iterate over a variable number of > items and split the data into multiple files.At the risk of sounding dim, if you have variable files full of variable data, why don''t you handle that logic in the ruby stanzas in your template? You can template the main file, and the ruby portions can write other files as needed. Otherwise, if you really have variable files full of variable data, I''m curious about just what application you''re attempting to configure. Obviously> Doug. > > > > > > $vol_host1_inst1_ > \ > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Douglas Garstang
2011-Sep-22 16:23 UTC
Re: [Puppet Users] Re: Variable number of variables in external node output...
On Thu, Sep 22, 2011 at 9:14 AM, Christopher Wood <christopher_wood@pobox.com> wrote:> On Thu, Sep 22, 2011 at 09:05:32AM -0700, Douglas Garstang wrote: >> On Thu, Sep 22, 2011 at 5:57 AM, jcbollinger <John.Bollinger@stjude.org> wrote: >> > >> > >> > On Sep 21, 6:34 pm, Nigel Kersten <ni...@puppetlabs.com> wrote: >> >> On Wed, Sep 21, 2011 at 1:20 PM, Douglas Garstang >> >> >> >> <doug.garst...@gmail.com> wrote: >> >> > All, >> >> >> >> > I have a situation where I need to get some fairly complex >> >> > configuration files onto systems, and I''m wondering if puppet can even >> >> > do this. Lets say that my external node script will go and source all >> >> > the data it needs from an external database, and dump out all >> >> > variables that the node will need. The relevant puppet module(s) will >> >> > then have to inject these variables into templates to be deployed to >> >> > the systems. >> >> >> >> Can you provide an example of a chunk of actual data and the desired >> >> end result to see if there''s a better alternative Doug? >> > >> > >> > I second that request. The scenario presented is more a design >> > concept than a problem description. Although I am confident that >> > Puppet indeed can handle the scenario as presented, it may be that >> > there are alternative designs that would accomplish the same objective >> > in a simpler way. >> >> Well, speaking somewhat generically still, we have an application that >> will need to run multiple times on a single system, and each instance >> of that running application will have it''s own config file, each with >> a variable number of items (they''re disk volumes). >> >> If the external node script returned YAML data like this: >> >> appX_inst1_vol1_name: vol1 >> appX_inst1_vol1_size: 1G >> appX_inst1_vol1_active: true >> appX_inst2_vol1_name: vol2 >> appX_inst2_vol1_size: 2G >> appX_inst2_vol1_active: false >> >> ... and so on. All the volumes for inst1 need to go into one config >> file, all the volumes for inst2 in another config file and so on. I >> don''t see a way that puppet can iterate over a variable number of >> items and split the data into multiple files. > > At the risk of sounding dim, if you have variable files full of variable data, why don''t you handle that logic in the ruby stanzas in your template? You can template the main file, and the ruby portions can write other files as needed.I wasn''t quite sure, but is the embedded ruby in the templates fully functional ruby? ie it can write files etc?> > Otherwise, if you really have variable files full of variable data, I''m curious about just what application you''re attempting to configure. >It''s proprietary software. Every service company has proprietary software that may (or may not in our case) be designed very well. Doug. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Douglas Garstang
2011-Sep-22 16:59 UTC
Re: [Puppet Users] Re: Variable number of variables in external node output...
On Thu, Sep 22, 2011 at 9:23 AM, Douglas Garstang <doug.garstang@gmail.com> wrote:> On Thu, Sep 22, 2011 at 9:14 AM, Christopher Wood > <christopher_wood@pobox.com> wrote: >> On Thu, Sep 22, 2011 at 09:05:32AM -0700, Douglas Garstang wrote: >>> On Thu, Sep 22, 2011 at 5:57 AM, jcbollinger <John.Bollinger@stjude.org> wrote: >>> > >>> > >>> > On Sep 21, 6:34 pm, Nigel Kersten <ni...@puppetlabs.com> wrote: >>> >> On Wed, Sep 21, 2011 at 1:20 PM, Douglas Garstang >>> >> >>> >> <doug.garst...@gmail.com> wrote: >>> >> > All, >>> >> >>> >> > I have a situation where I need to get some fairly complex >>> >> > configuration files onto systems, and I''m wondering if puppet can even >>> >> > do this. Lets say that my external node script will go and source all >>> >> > the data it needs from an external database, and dump out all >>> >> > variables that the node will need. The relevant puppet module(s) will >>> >> > then have to inject these variables into templates to be deployed to >>> >> > the systems. >>> >> >>> >> Can you provide an example of a chunk of actual data and the desired >>> >> end result to see if there''s a better alternative Doug? >>> > >>> > >>> > I second that request. The scenario presented is more a design >>> > concept than a problem description. Although I am confident that >>> > Puppet indeed can handle the scenario as presented, it may be that >>> > there are alternative designs that would accomplish the same objective >>> > in a simpler way. >>> >>> Well, speaking somewhat generically still, we have an application that >>> will need to run multiple times on a single system, and each instance >>> of that running application will have it''s own config file, each with >>> a variable number of items (they''re disk volumes). >>> >>> If the external node script returned YAML data like this: >>> >>> appX_inst1_vol1_name: vol1 >>> appX_inst1_vol1_size: 1G >>> appX_inst1_vol1_active: true >>> appX_inst2_vol1_name: vol2 >>> appX_inst2_vol1_size: 2G >>> appX_inst2_vol1_active: false >>> >>> ... and so on. All the volumes for inst1 need to go into one config >>> file, all the volumes for inst2 in another config file and so on. I >>> don''t see a way that puppet can iterate over a variable number of >>> items and split the data into multiple files. >> >> At the risk of sounding dim, if you have variable files full of variable data, why don''t you handle that logic in the ruby stanzas in your template? You can template the main file, and the ruby portions can write other files as needed. > > I wasn''t quite sure, but is the embedded ruby in the templates fully > functional ruby? ie it can write files etc?Actually, that doesn''t work. Putting a file write operation into the template works, but it gets executed on the server, which is useless. Doug. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Christopher Wood
2011-Sep-22 17:08 UTC
Re: [Puppet Users] Re: Variable number of variables in external node output...
On Thu, Sep 22, 2011 at 09:23:56AM -0700, Douglas Garstang wrote:> On Thu, Sep 22, 2011 at 9:14 AM, Christopher Wood > <christopher_wood@pobox.com> wrote: > > On Thu, Sep 22, 2011 at 09:05:32AM -0700, Douglas Garstang wrote: > >> On Thu, Sep 22, 2011 at 5:57 AM, jcbollinger <John.Bollinger@stjude.org> wrote: > >> > > >> > > >> > On Sep 21, 6:34 pm, Nigel Kersten <ni...@puppetlabs.com> wrote: > >> >> On Wed, Sep 21, 2011 at 1:20 PM, Douglas Garstang > >> >> > >> >> <doug.garst...@gmail.com> wrote: > >> >> > All, > >> >> > >> >> > I have a situation where I need to get some fairly complex > >> >> > configuration files onto systems, and I''m wondering if puppet can even > >> >> > do this. Lets say that my external node script will go and source all > >> >> > the data it needs from an external database, and dump out all > >> >> > variables that the node will need. The relevant puppet module(s) will > >> >> > then have to inject these variables into templates to be deployed to > >> >> > the systems. > >> >> > >> >> Can you provide an example of a chunk of actual data and the desired > >> >> end result to see if there''s a better alternative Doug? > >> > > >> > > >> > I second that request. The scenario presented is more a design > >> > concept than a problem description. Although I am confident that > >> > Puppet indeed can handle the scenario as presented, it may be that > >> > there are alternative designs that would accomplish the same objective > >> > in a simpler way. > >> > >> Well, speaking somewhat generically still, we have an application that > >> will need to run multiple times on a single system, and each instance > >> of that running application will have it''s own config file, each with > >> a variable number of items (they''re disk volumes). > >> > >> If the external node script returned YAML data like this: > >> > >> appX_inst1_vol1_name: vol1 > >> appX_inst1_vol1_size: 1G > >> appX_inst1_vol1_active: true > >> appX_inst2_vol1_name: vol2 > >> appX_inst2_vol1_size: 2G > >> appX_inst2_vol1_active: false > >> > >> ... and so on. All the volumes for inst1 need to go into one config > >> file, all the volumes for inst2 in another config file and so on. I > >> don''t see a way that puppet can iterate over a variable number of > >> items and split the data into multiple files. > > > > At the risk of sounding dim, if you have variable files full of variable data, why don''t you handle that logic in the ruby stanzas in your template? You can template the main file, and the ruby portions can write other files as needed. > > I wasn''t quite sure, but is the embedded ruby in the templates fully > functional ruby? ie it can write files etc?Yes and no. Templates are sorted out on the server, so files are written there. I''m using this capacity to create /etc/network/interfaces in Debian/Ubuntu, still thinking about how to handle the same thing in CentOS/RedHat. I suspect I''ll end up templating a yaml file and having an agent-side script create the files themselves. The script exec can subscribe to the yaml file and kick off the networking reload itself.> > Otherwise, if you really have variable files full of variable data, I''m curious about just what application you''re attempting to configure. > > > > It''s proprietary software. Every service company has proprietary > software that may (or may not in our case) be designed very well.Tell them that the internet people had you name your config handling module vendorname__workarounds.> Doug. > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Craig White
2011-Sep-22 17:22 UTC
Re: [Puppet Users] Variable number of variables in external node output...
Isn''t this the itch that create_resources is designed to scratch? Craig On Sep 21, 2011, at 4:26 PM, Douglas Garstang wrote:> Thanks, but I''m not sure templates will quite cut it. There is no way > that I am aware of to iterate over all the variables obtained through > an external node script, and put chunks into separate files. This > falls outside the bounds of what embedded ruby can do. > > As for using define(), well, I am trying to separate the data from the > manifest, and putting N number of defines in the manifests and passing > values to it, breaks that. > > Doug. > > On Wed, Sep 21, 2011 at 2:48 PM, Aaron Grewell <aaron.grewell@gmail.com> wrote: >> Remember too that if you can do it once (say with a define) Puppet will do >> it as many times as you like just by putting the data in an array. >> >> On Sep 21, 2011 2:39 PM, "Brian Gupta" <brian.gupta@brandorr.com> wrote: >>> Puppet can do this. I would propose the following solution, but there may >>> be >>> other ways to do it. >>> >>> 1) Use a template. Templates have embedded ruby code and run on the >>> clients. >>> 2) Link to some library that allows you to query your database (You can >>> use >>> one or more require statements in your ruby code) >>> 3) Embed ruby code to parse and format your variables into configuration >>> >>> See http://docs.puppetlabs.com/guides/templating.html for a bit more about >>> templates. >>> >>> -Brian >>> >>> On Wed, Sep 21, 2011 at 4:20 PM, Douglas Garstang >>> <doug.garstang@gmail.com>wrote: >>> >>>> All, >>>> >>>> I have a situation where I need to get some fairly complex >>>> configuration files onto systems, and I''m wondering if puppet can even >>>> do this. Lets say that my external node script will go and source all >>>> the data it needs from an external database, and dump out all >>>> variables that the node will need. The relevant puppet module(s) will >>>> then have to inject these variables into templates to be deployed to >>>> the systems. >>>> >>>> Now, what if the number of variables that get dumped by the external >>>> node script is variable? And, what if these variables are broken into >>>> chunks that need to be split into different files on the remote >>>> system? How would I do such a complex thing in puppet? A lack of any >>>> type of looping constructs in puppet would seem to make this rather >>>> difficult. >>>> >>>> Doug. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Puppet Users" group. >>>> To post to this group, send email to puppet-users@googlegroups.com. >>>> To unsubscribe from this group, send email to >>>> puppet-users+unsubscribe@googlegroups.com. >>>> For more options, visit this group at >>>> http://groups.google.com/group/puppet-users?hl=en. >>>> >>>> >>> >>> >>> -- >>> <http://aws.amazon.com/solutions/solution-providers/brandorr/> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Puppet Users" group. >>> To post to this group, send email to puppet-users@googlegroups.com. >>> To unsubscribe from this group, send email to >>> puppet-users+unsubscribe@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/puppet-users?hl=en. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> > > > > -- > Regards, > > Douglas Garstang > http://www.linkedin.com/in/garstang > Email: doug.garstang@gmail.com > Cell: +1-805-340-5627 > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >-- Craig White ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ craig.white@ttiltd.com 1.800.869.6908 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ www.ttiassessments.com Need help communicating between generations at work to achieve your desired success? Let us help! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Aaron Grewell
2011-Sep-22 17:48 UTC
Re: [Puppet Users] Variable number of variables in external node output...
Yes, but if he doesn''t like defines then create_resources is probably out as well. On Thu, Sep 22, 2011 at 10:22 AM, Craig White <craig.white@ttiltd.com>wrote:> Isn''t this the itch that create_resources is designed to scratch? > > Craig > > On Sep 21, 2011, at 4:26 PM, Douglas Garstang wrote: > > > Thanks, but I''m not sure templates will quite cut it. There is no way > > that I am aware of to iterate over all the variables obtained through > > an external node script, and put chunks into separate files. This > > falls outside the bounds of what embedded ruby can do. > > > > As for using define(), well, I am trying to separate the data from the > > manifest, and putting N number of defines in the manifests and passing > > values to it, breaks that. > > > > Doug. > > > > On Wed, Sep 21, 2011 at 2:48 PM, Aaron Grewell <aaron.grewell@gmail.com> > wrote: > >> Remember too that if you can do it once (say with a define) Puppet will > do > >> it as many times as you like just by putting the data in an array. > >> > >> On Sep 21, 2011 2:39 PM, "Brian Gupta" <brian.gupta@brandorr.com> > wrote: > >>> Puppet can do this. I would propose the following solution, but there > may > >>> be > >>> other ways to do it. > >>> > >>> 1) Use a template. Templates have embedded ruby code and run on the > >>> clients. > >>> 2) Link to some library that allows you to query your database (You can > >>> use > >>> one or more require statements in your ruby code) > >>> 3) Embed ruby code to parse and format your variables into > configuration > >>> > >>> See http://docs.puppetlabs.com/guides/templating.html for a bit more > about > >>> templates. > >>> > >>> -Brian > >>> > >>> On Wed, Sep 21, 2011 at 4:20 PM, Douglas Garstang > >>> <doug.garstang@gmail.com>wrote: > >>> > >>>> All, > >>>> > >>>> I have a situation where I need to get some fairly complex > >>>> configuration files onto systems, and I''m wondering if puppet can even > >>>> do this. Lets say that my external node script will go and source all > >>>> the data it needs from an external database, and dump out all > >>>> variables that the node will need. The relevant puppet module(s) will > >>>> then have to inject these variables into templates to be deployed to > >>>> the systems. > >>>> > >>>> Now, what if the number of variables that get dumped by the external > >>>> node script is variable? And, what if these variables are broken into > >>>> chunks that need to be split into different files on the remote > >>>> system? How would I do such a complex thing in puppet? A lack of any > >>>> type of looping constructs in puppet would seem to make this rather > >>>> difficult. > >>>> > >>>> Doug. > >>>> > >>>> -- > >>>> You received this message because you are subscribed to the Google > Groups > >>>> "Puppet Users" group. > >>>> To post to this group, send email to puppet-users@googlegroups.com. > >>>> To unsubscribe from this group, send email to > >>>> puppet-users+unsubscribe@googlegroups.com. > >>>> For more options, visit this group at > >>>> http://groups.google.com/group/puppet-users?hl=en. > >>>> > >>>> > >>> > >>> > >>> -- > >>> <http://aws.amazon.com/solutions/solution-providers/brandorr/> > >>> > >>> -- > >>> You received this message because you are subscribed to the Google > Groups > >>> "Puppet Users" group. > >>> To post to this group, send email to puppet-users@googlegroups.com. > >>> To unsubscribe from this group, send email to > >>> puppet-users+unsubscribe@googlegroups.com. > >>> For more options, visit this group at > >>> http://groups.google.com/group/puppet-users?hl=en. > >>> > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Puppet Users" group. > >> To post to this group, send email to puppet-users@googlegroups.com. > >> To unsubscribe from this group, send email to > >> puppet-users+unsubscribe@googlegroups.com. > >> For more options, visit this group at > >> http://groups.google.com/group/puppet-users?hl=en. > >> > > > > > > > > -- > > Regards, > > > > Douglas Garstang > > http://www.linkedin.com/in/garstang > > Email: doug.garstang@gmail.com > > Cell: +1-805-340-5627 > > > > -- > > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > > To post to this group, send email to puppet-users@googlegroups.com. > > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > > > -- > Craig White ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > craig.white@ttiltd.com > 1.800.869.6908 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > www.ttiassessments.com > > Need help communicating between generations at work to achieve your desired > success? Let us help! > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
jcbollinger
2011-Sep-22 20:56 UTC
[Puppet Users] Re: Variable number of variables in external node output...
On Sep 22, 11:05 am, Douglas Garstang <doug.garst...@gmail.com> wrote:> On Thu, Sep 22, 2011 at 5:57 AM, jcbollinger <John.Bollin...@stjude.org> wrote: > > > On Sep 21, 6:34 pm, Nigel Kersten <ni...@puppetlabs.com> wrote: > >> On Wed, Sep 21, 2011 at 1:20 PM, Douglas Garstang > > >> <doug.garst...@gmail.com> wrote: > >> > All, > > >> > I have a situation where I need to get some fairly complex > >> > configuration files onto systems, and I''m wondering if puppet can even > >> > do this. Lets say that my external node script will go and source all > >> > the data it needs from an external database, and dump out all > >> > variables that the node will need. The relevant puppet module(s) will > >> > then have to inject these variables into templates to be deployed to > >> > the systems. > > >> Can you provide an example of a chunk of actual data and the desired > >> end result to see if there''s a better alternative Doug? > > > I second that request. The scenario presented is more a design > > concept than a problem description. Although I am confident that > > Puppet indeed can handle the scenario as presented, it may be that > > there are alternative designs that would accomplish the same objective > > in a simpler way. > > Well, speaking somewhat generically still, we have an application that > will need to run multiple times on a single system, and each instance > of that running application will have it''s own config file, each with > a variable number of items (they''re disk volumes). > > If the external node script returned YAML data like this: > > appX_inst1_vol1_name: vol1 > appX_inst1_vol1_size: 1G > appX_inst1_vol1_active: true > appX_inst2_vol1_name: vol2 > appX_inst2_vol1_size: 2G > appX_inst2_vol1_active: false > > ... and so on. All the volumes for inst1 need to go into one config > file, all the volumes for inst2 in another config file and so on. I > don''t see a way that puppet can iterate over a variable number of > items and split the data into multiple files.Part of the question is why the ENC would structure the data that way in the first place. I don''t use one, so perhaps I am displaying my ignorance, but can''t ENCs assign hash and array values to variables? As any Perl hand will know, arrays and hashes are plenty to build data structures of arbitrary complexity. On the Puppet side, array resource titles, defined types, and create_resources() have enough power to pull that data back out, at least if you structure it with a bit of care. I think hiera could help go even further. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Douglas Garstang
2011-Sep-22 21:11 UTC
Re: [Puppet Users] Re: Variable number of variables in external node output...
On Thu, Sep 22, 2011 at 1:56 PM, jcbollinger <John.Bollinger@stjude.org> wrote:> > > On Sep 22, 11:05 am, Douglas Garstang <doug.garst...@gmail.com> wrote: >> On Thu, Sep 22, 2011 at 5:57 AM, jcbollinger <John.Bollin...@stjude.org> wrote: >> >> > On Sep 21, 6:34 pm, Nigel Kersten <ni...@puppetlabs.com> wrote: >> >> On Wed, Sep 21, 2011 at 1:20 PM, Douglas Garstang >> >> >> <doug.garst...@gmail.com> wrote: >> >> > All, >> >> >> > I have a situation where I need to get some fairly complex >> >> > configuration files onto systems, and I''m wondering if puppet can even >> >> > do this. Lets say that my external node script will go and source all >> >> > the data it needs from an external database, and dump out all >> >> > variables that the node will need. The relevant puppet module(s) will >> >> > then have to inject these variables into templates to be deployed to >> >> > the systems. >> >> >> Can you provide an example of a chunk of actual data and the desired >> >> end result to see if there''s a better alternative Doug? >> >> > I second that request. The scenario presented is more a design >> > concept than a problem description. Although I am confident that >> > Puppet indeed can handle the scenario as presented, it may be that >> > there are alternative designs that would accomplish the same objective >> > in a simpler way. >> >> Well, speaking somewhat generically still, we have an application that >> will need to run multiple times on a single system, and each instance >> of that running application will have it''s own config file, each with >> a variable number of items (they''re disk volumes). >> >> If the external node script returned YAML data like this: >> >> appX_inst1_vol1_name: vol1 >> appX_inst1_vol1_size: 1G >> appX_inst1_vol1_active: true >> appX_inst2_vol1_name: vol2 >> appX_inst2_vol1_size: 2G >> appX_inst2_vol1_active: false >> >> ... and so on. All the volumes for inst1 need to go into one config >> file, all the volumes for inst2 in another config file and so on. I >> don''t see a way that puppet can iterate over a variable number of >> items and split the data into multiple files. > > > Part of the question is why the ENC would structure the data that way > in the first place. I don''t use one, so perhaps I am displaying my > ignorance, but can''t ENCs assign hash and array values to variables? > As any Perl hand will know, arrays and hashes are plenty to build data > structures of arbitrary complexity.The ENC needs to structure the data that way in order to allow puppet to reformat the data in a way that the software recognises. I don''t think ENC''s can do anything more complex than assign key/value pairs under ''parameters''. I''ve tried to put more complex data under parameters, but I can''t access it in the manifest. I also need to break the data into multiple files, which erb can''t do, since it can only write files to the server, not the client.> > On the Puppet side, array resource titles, defined types, and > create_resources() have enough power to pull that data back out, at > least if you structure it with a bit of care. I think hiera could > help go even further.See my post just now about passing an array to a definition. It looks like the $name field has a size limit which stops me from passing all the data I need. Not sure what create_resource is .... documentation? Doug. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
jcbollinger
2011-Sep-23 13:24 UTC
[Puppet Users] Re: Variable number of variables in external node output...
On Sep 22, 4:11 pm, Douglas Garstang <doug.garst...@gmail.com> wrote:> On Thu, Sep 22, 2011 at 1:56 PM, jcbollinger <John.Bollin...@stjude.org> wrote: > > > On Sep 22, 11:05 am, Douglas Garstang <doug.garst...@gmail.com> wrote: > >> On Thu, Sep 22, 2011 at 5:57 AM, jcbollinger <John.Bollin...@stjude.org> wrote: > > >> > On Sep 21, 6:34 pm, Nigel Kersten <ni...@puppetlabs.com> wrote: > >> >> On Wed, Sep 21, 2011 at 1:20 PM, Douglas Garstang > > >> >> <doug.garst...@gmail.com> wrote: > >> >> > All, > > >> >> > I have a situation where I need to get some fairly complex > >> >> > configuration files onto systems, and I''m wondering if puppet can even > >> >> > do this. Lets say that my external node script will go and source all > >> >> > the data it needs from an external database, and dump out all > >> >> > variables that the node will need. The relevant puppet module(s) will > >> >> > then have to inject these variables into templates to be deployed to > >> >> > the systems. > > >> >> Can you provide an example of a chunk of actual data and the desired > >> >> end result to see if there''s a better alternative Doug? > > >> > I second that request. The scenario presented is more a design > >> > concept than a problem description. Although I am confident that > >> > Puppet indeed can handle the scenario as presented, it may be that > >> > there are alternative designs that would accomplish the same objective > >> > in a simpler way. > > >> Well, speaking somewhat generically still, we have an application that > >> will need to run multiple times on a single system, and each instance > >> of that running application will have it''s own config file, each with > >> a variable number of items (they''re disk volumes). > > >> If the external node script returned YAML data like this: > > >> appX_inst1_vol1_name: vol1 > >> appX_inst1_vol1_size: 1G > >> appX_inst1_vol1_active: true > >> appX_inst2_vol1_name: vol2 > >> appX_inst2_vol1_size: 2G > >> appX_inst2_vol1_active: false > > >> ... and so on. All the volumes for inst1 need to go into one config > >> file, all the volumes for inst2 in another config file and so on. I > >> don''t see a way that puppet can iterate over a variable number of > >> items and split the data into multiple files. > > > Part of the question is why the ENC would structure the data that way > > in the first place. I don''t use one, so perhaps I am displaying my > > ignorance, but can''t ENCs assign hash and array values to variables? > > As any Perl hand will know, arrays and hashes are plenty to build data > > structures of arbitrary complexity. > > The ENC needs to structure the data that way in order to allow puppet > to reformat the data in a way that the software recognises. I don''t > think ENC''s can do anything more complex than assign key/value pairs > under ''parameters''. I''ve tried to put more complex data under > parameters, but I can''t access it in the manifest.It looks like this would be http://projects.puppetlabs.com/issues/3154. Sorry for that red herring.> I also need to > break the data into multiple files, which erb can''t do, since it can > only write files to the server, not the client.As I see it, you have two main problems: 1) Obtaining or building a sufficiently structured form of the data defining the needed config files 2) Building the files from the data It is a shame that (1) cannot currently be accomplished directly from your ENC, but that does not mean it cannot be accomplished at all. If you are free to obtain the data from a separate data source, then a custom function can in principle do whatever you want. For this job, however, Hiera looks like it could already do the job nicely. Perhaps extlookup() could also, but I''m not sure off-hand whether it can provide data of complex types. Even with the data coming from the ENC largely as you are currently trying to do, you should still be able to analyze it successfully in Puppet. In particular, it looks like one of your key problems is determining which variables are present for your node to use, but you should be able to use an inline template to find these (see http://docs.puppetlabs.com/guides/templating.html#access-to-variables-and-puppet-functions-with-the-scope-object). As for problem (2), I reiterate that Puppet DSL gives you enough to accomplish your objective. Nevertheless, you also have the option of writing pure-Ruby manifests, which provides all the capabilities of a general-purpose programming language.> > On the Puppet side, array resource titles, defined types, and > > create_resources() have enough power to pull that data back out, at > > least if you structure it with a bit of care. I think hiera could > > help go even further. > > See my post just now about passing an array to a definition. It looks > like the $name field has a size limit which stops me from passing all > the data I need.As Ken Barber already replied, in that case you are attempting to pass a hash as the resource title. Puppet does the best it can with that (in particular, it flattens it to a string), but that is extremely unlikely to be what you want.> Not sure what create_resource is .... documentation?http://docs.puppetlabs.com/references/stable/function.html#createresources It is for declaring a resource based on a hash of its properties, which appears to be what you were looking for when you tried passing a hash as a resource title. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.