Gavin Williams
2013-Jan-15 10:34 UTC
[Puppet Users] Puppet, Hiera and complex data structures...
Morning all... Firstly, apologies for the length of this post, however I thought it would be beneficial to fully explain what I''m trying to achieve. I''ve started using Hiera with Puppet 3.0, and must say I''m very impressed soo far. It''s handled the bulk of what I''ve thrown at it so far without any issues... However I''m now starting to think about some of the more complex challenges we''re hoping to solve with Puppet... One of those is Oracle Database configuration management... Under this falls the following components: - NetApp Filer volume configuration - Active site and DR site, with SnapMirror relationship between the 2. - Oracle Database server configuration - Active site and DR site. As you can see, that''s effectively 4 separate components, spread across 2 separate sites, but which are closely related... Currently, all our Hiera configuration is driven from the physical view, that is a host, or an environment, or a data-centre... However I''d like to try and turn this configuration on it''s head and go logical... That is each Database instance has it''s own configuration data, which contains all the relevant information such as what it''s Active and DR NetApp filer is, what it''s volume sizes are, what Active database server it should live on, where it should be in DR mode... Now this all seems quite easy to define in Hiera Yaml... Something like the following should cover it: --- database_sid: ''ORACLE01'' active_filer: ''site1-nactl01'' replica_filer: ''site2-nactl01'' active_database_server: ''site1-db01'' replicate_database_server: ''site2-db01'' oracle_version: ''11.2.0.3'' volumes: ORACLE01_oractrl: ensure: present size: ''1g'' ORACLE01_oradata: ensure: present size: ''150g'' ORACLE01_oraredo: ensure: present size: ''10g'' ORACLE01_oratemp: ensure: present size: ''10g'' ORACLE01_oraarch: ensure: present size: ''20g'' snapschedule: minutes: 0 hours: 36 days: 0 weeks: 0 snapmirror: ... However the challenge I''m currently struggling to get my head around is how I can turn that logical view into the various physical views that are needed for each component... It also gets further complicated in that there is going to be multiple database instances, with some commonality between them, and some variation... So if I''m running puppet on *site1-db01*, I want to process all databases that have either an ''*active_database_server*'' or a ''* replicate_database_server*'' value matching the hostname. If running puppet against *site1-nactl01*, then need to process all databases that have an ''*active_filer*'' matching the hostname... Any ideas on where I could begin? Or is there a better way for me to achieve this? Thank-you in advance for any replies. Regards Gavin -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/H3tYf7ngL5wJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
jcbollinger
2013-Jan-15 19:05 UTC
[Puppet Users] Re: Puppet, Hiera and complex data structures...
On Tuesday, January 15, 2013 4:34:19 AM UTC-6, Gavin Williams wrote:> > Morning all... > > Firstly, apologies for the length of this post, however I thought it would > be beneficial to fully explain what I''m trying to achieve. > > I''ve started using Hiera with Puppet 3.0, and must say I''m very impressed > soo far. It''s handled the bulk of what I''ve thrown at it so far without any > issues... > > However I''m now starting to think about some of the more complex > challenges we''re hoping to solve with Puppet... > One of those is Oracle Database configuration management... > > Under this falls the following components: > > - NetApp Filer volume configuration - Active site and DR site, with > SnapMirror relationship between the 2. > - Oracle Database server configuration - Active site and DR site. > > As you can see, that''s effectively 4 separate components, spread across 2 > separate sites, but which are closely related... > > Currently, all our Hiera configuration is driven from the physical view, > that is a host, or an environment, or a data-centre... > However I''d like to try and turn this configuration on it''s head and go > logical... That is each Database instance has it''s own configuration data, > which contains all the relevant information such as what it''s Active and DR > NetApp filer is, what it''s volume sizes are, what Active database server it > should live on, where it should be in DR mode... > > Now this all seems quite easy to define in Hiera Yaml... Something like > the following should cover it: > --- > database_sid: ''ORACLE01'' > active_filer: ''site1-nactl01'' > replica_filer: ''site2-nactl01'' > active_database_server: ''site1-db01'' > replicate_database_server: ''site2-db01'' > oracle_version: ''11.2.0.3'' > volumes: > ORACLE01_oractrl: > ensure: present > size: ''1g'' > ORACLE01_oradata: > ensure: present > size: ''150g'' > ORACLE01_oraredo: > ensure: present > size: ''10g'' > ORACLE01_oratemp: > ensure: present > size: ''10g'' > ORACLE01_oraarch: > ensure: present > size: ''20g'' > snapschedule: > minutes: 0 > hours: 36 > days: 0 > weeks: 0 > snapmirror: > ... > > However the challenge I''m currently struggling to get my head around is > how I can turn that logical view into the various physical views that are > needed for each component... > It also gets further complicated in that there is going to be multiple > database instances, with some commonality between them, and some > variation... > > So if I''m running puppet on *site1-db01*, I want to process all databases > that have either an ''*active_database_server*'' or a ''* > replicate_database_server*'' value matching the hostname. > If running puppet against *site1-nactl01*, then need to process all > databases that have an ''*active_filer*'' matching the hostname... > > Any ideas on where I could begin? > Or is there a better way for me to achieve this? >Arranging for each node to draw on (only) the correct data is normally a question of how you structure your data and organize your hierarchy. Hiera has no built-in mechanism for applying the kind of selection predicates you describe, so you would layout your data such that nodes only get the data for the instances that should be configured on them. Failing (or instead of) that, you can enhance selectivity by declaring all the relevant resources virtually, and then using collections with appropriate predicates to realize only the ones that are wanted for the current target. If that''s insufficient then you could consider creating custom functions to sift the wanted data from the available data. You might even be able to make it generic enough to be reusable. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/8fYMI6vvWUIJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Gavin Williams
2013-Jan-16 15:23 UTC
[Puppet Users] Re: Puppet, Hiera and complex data structures...
John Cheers for that... Could you expand on *"enhance selectivity by declaring all the relevant resources virtually, and then using collections with appropriate predicates to realize only the ones that are wanted for the current target."*? Have used virtual resources for other bits and pieces, but not familiar with ''collections''... Cheers Gavin On Tuesday, 15 January 2013 19:05:33 UTC, jcbollinger wrote:> > > > On Tuesday, January 15, 2013 4:34:19 AM UTC-6, Gavin Williams wrote: >> >> Morning all... >> >> Firstly, apologies for the length of this post, however I thought it >> would be beneficial to fully explain what I''m trying to achieve. >> >> I''ve started using Hiera with Puppet 3.0, and must say I''m very impressed >> soo far. It''s handled the bulk of what I''ve thrown at it so far without any >> issues... >> >> However I''m now starting to think about some of the more complex >> challenges we''re hoping to solve with Puppet... >> One of those is Oracle Database configuration management... >> >> Under this falls the following components: >> >> - NetApp Filer volume configuration - Active site and DR site, with >> SnapMirror relationship between the 2. >> - Oracle Database server configuration - Active site and DR site. >> >> As you can see, that''s effectively 4 separate components, spread across 2 >> separate sites, but which are closely related... >> >> Currently, all our Hiera configuration is driven from the physical view, >> that is a host, or an environment, or a data-centre... >> However I''d like to try and turn this configuration on it''s head and go >> logical... That is each Database instance has it''s own configuration data, >> which contains all the relevant information such as what it''s Active and DR >> NetApp filer is, what it''s volume sizes are, what Active database server it >> should live on, where it should be in DR mode... >> >> Now this all seems quite easy to define in Hiera Yaml... Something like >> the following should cover it: >> --- >> database_sid: ''ORACLE01'' >> active_filer: ''site1-nactl01'' >> replica_filer: ''site2-nactl01'' >> active_database_server: ''site1-db01'' >> replicate_database_server: ''site2-db01'' >> oracle_version: ''11.2.0.3'' >> volumes: >> ORACLE01_oractrl: >> ensure: present >> size: ''1g'' >> ORACLE01_oradata: >> ensure: present >> size: ''150g'' >> ORACLE01_oraredo: >> ensure: present >> size: ''10g'' >> ORACLE01_oratemp: >> ensure: present >> size: ''10g'' >> ORACLE01_oraarch: >> ensure: present >> size: ''20g'' >> snapschedule: >> minutes: 0 >> hours: 36 >> days: 0 >> weeks: 0 >> snapmirror: >> ... >> >> However the challenge I''m currently struggling to get my head around is >> how I can turn that logical view into the various physical views that are >> needed for each component... >> It also gets further complicated in that there is going to be multiple >> database instances, with some commonality between them, and some >> variation... >> >> So if I''m running puppet on *site1-db01*, I want to process all >> databases that have either an ''*active_database_server*'' or a ''* >> replicate_database_server*'' value matching the hostname. >> If running puppet against *site1-nactl01*, then need to process all >> databases that have an ''*active_filer*'' matching the hostname... >> >> Any ideas on where I could begin? >> Or is there a better way for me to achieve this? >> > > > Arranging for each node to draw on (only) the correct data is normally a > question of how you structure your data and organize your hierarchy. Hiera > has no built-in mechanism for applying the kind of selection predicates you > describe, so you would layout your data such that nodes only get the data > for the instances that should be configured on them. > > Failing (or instead of) that, you can enhance selectivity by declaring all > the relevant resources virtually, and then using collections with > appropriate predicates to realize only the ones that are wanted for the > current target. > > If that''s insufficient then you could consider creating custom functions > to sift the wanted data from the available data. You might even be able to > make it generic enough to be reusable. > > > John > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/0_lYJVhBtlcJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
jcbollinger
2013-Jan-16 20:57 UTC
[Puppet Users] Re: Puppet, Hiera and complex data structures...
On Wednesday, January 16, 2013 9:23:42 AM UTC-6, Gavin Williams wrote:> > John > > Cheers for that... > > Could you expand on *"enhance selectivity by declaring all the relevant > resources virtually, and then using collections with appropriate predicates > to realize only the ones that are wanted for the current target."*? > Have used virtual resources for other bits and pieces, but not familiar > with ''collections''... > >Simplified example: @user { "alice": tag => ''user''; "bob": tag => ''admin''; } # Realize users tagged as admins, but not other virtual users User<| tag == ''admin'' |> Selection predicates may use other resource properties, too, not just tags. See http://docs.puppetlabs.com/puppet/3/reference/lang_virtual.html for (a bit) more information. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/GSYyBJFMBhMJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.