Luke Bigum
2012-May-11 15:42 UTC
[Puppet Users] how best to combine business data (Hiera) and machine data (Facts)
Hi all, I''ve been improving our sysctl module and come across an interesting design problem I''d like feedback on. I approached the re-factor with Hiera in mind - I would put all our sysctl data in Hiera hash and pull that into a hiera_hash, merging the hierarchy of data and allowing higher priority sysctl settings to override the baseline defaults. I then use create_resources to write sysctl.conf. Works great to start with, but now I come across more and more cases where the sysctl data is dependent on machine logic (virtual vs physical, types of hardware, etc) that doesn''t seem right to put into Hiera as I''d have a complex hierarchy for a bunch of edge case Facts. I seem to need to make decisions on two sources: business logic in Hiera hierarchy (that''s easy with merging hashes) as well as considering what Facts or Classes applies to a node (machine logic). That''s not trivial to do, especially with a potentially large set of data like sysctl.conf keys. Does anyone have any thoughts or tips on how they might be managing a similar situation? Thanks, -Luke -- Luke Bigum Information Systems Ph: +44 (0) 20 3192 2520 luke.bigum@lmax.com | http://www.lmax.com LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN FX and CFDs are leveraged products that can result in losses exceeding your deposit. They are not suitable for everyone so please ensure you fully understand the risks involved. The information in this email is not directed at residents of the United States of America or any other jurisdiction where trading in CFDs and/or FX is restricted or prohibited by local laws or regulations. The information in this email and any attachment is confidential and is intended only for the named recipient(s). The email may not be disclosed or used by any person other than the addressee, nor may it be copied in any way. If you are not the intended recipient please notify the sender immediately and delete any copies of this message. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden. LMAX operates a multilateral trading facility. Authorised and regulated by the Financial Services Authority (firm registration number 509778) and is registered in England and Wales (number 06505809). Our registered address is Yellow Building, 1A Nicholas Road, London, W11 4AN. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Gary Larizza
2012-May-11 15:53 UTC
Re: [Puppet Users] how best to combine business data (Hiera) and machine data (Facts)
I see this with people looking to move to the hierarchical system that Hiera brings. It essentially boils down to "How do I do this without having a ton of hierarchy levels?". Usually we tend to recommend using the hierarchy to hit the 80% mark for the data you need in your modules. Anything that''s module-specific-data should then be broken out to a data.pp or params.pp file with conditional logic there. I tend to ask people: "Is this something others are going to hit when they try to use the module too?", as in - "Are there path differences between operating systems?", or "Are there important changes to the data between RHEL 5 and 6?". If the answer to these is yes, then I tend to favor putting that data into a module''s data class so that it''s exposed for ANYONE who wants to use the module. Why would you want to hide these differences in the hierarchy - especially if others might run into them? Does this sound similar to the problems you''re facing? Or is this a case where you have custom facts that are specific to your organization that determine how you manage sysctl? On Fri, May 11, 2012 at 8:42 AM, Luke Bigum <Luke.Bigum@lmax.com> wrote:> Hi all, > > I''ve been improving our sysctl module and come across an interesting > design problem I''d like feedback on. > > I approached the re-factor with Hiera in mind - I would put all our sysctl > data in Hiera hash and pull that into a hiera_hash, merging the hierarchy > of data and allowing higher priority sysctl settings to override the > baseline defaults. I then use create_resources to write sysctl.conf. Works > great to start with, but now I come across more and more cases where the > sysctl data is dependent on machine logic (virtual vs physical, types of > hardware, etc) that doesn''t seem right to put into Hiera as I''d have a > complex hierarchy for a bunch of edge case Facts. > > I seem to need to make decisions on two sources: business logic in Hiera > hierarchy (that''s easy with merging hashes) as well as considering what > Facts or Classes applies to a node (machine logic). That''s not trivial to > do, especially with a potentially large set of data like sysctl.conf keys. > > Does anyone have any thoughts or tips on how they might be managing a > similar situation? > > Thanks, > > -Luke > > -- > Luke Bigum > > Information Systems > Ph: +44 (0) 20 3192 2520 > luke.bigum@lmax.com | http://www.lmax.com > LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN > > > FX and CFDs are leveraged products that can result in losses exceeding > your deposit. They are not suitable for everyone so please ensure you > fully understand the risks involved. The information in this email is not > directed at residents of the United States of America or any other > jurisdiction where trading in CFDs and/or FX is restricted or prohibited > by local laws or regulations. > > The information in this email and any attachment is confidential and is > intended only for the named recipient(s). The email may not be disclosed > or used by any person other than the addressee, nor may it be copied in > any way. If you are not the intended recipient please notify the sender > immediately and delete any copies of this message. Any unauthorised > copying, disclosure or distribution of the material in this e-mail is > strictly forbidden. > > LMAX operates a multilateral trading facility. Authorised and regulated > by the Financial Services Authority (firm registration number 509778) and > is registered in England and Wales (number 06505809). Our registered > address is Yellow Building, 1A Nicholas Road, London, W11 > 4AN. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@** > googlegroups.com <puppet-users%2Bunsubscribe@googlegroups.com>. > For more options, visit this group at http://groups.google.com/** > group/puppet-users?hl=en<http://groups.google.com/group/puppet-users?hl=en> > . > >-- Gary Larizza Professional Services Engineer Puppet Labs -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Luke Bigum
2012-May-11 16:47 UTC
Re: [Puppet Users] how best to combine business data (Hiera) and machine data (Facts)
Hi Gary,
Not quite... Let me go into more detail.
I''m trying to handle sysctl "perfectly" which is probably my
real
problem. Hiera''s ability to merge hashes together makes it perfect for
arriving at one set of sysctl options for a server based on "business
logic" (my hierarchy). For Hiera data on ''someserver''
below which has
''some_role'', calling hiera_hash in a Puppet manifest will give
me IP
forwarding set and rp_filter set, which is what I want:
--------- some_role.json ------------
{
"sysctl" : {
"net.ipv4.ip_forward" : {
"comment" : "Controls IP packet forwarding",
"value" : "1"
}
}
--------------------------------------------
--------- common.json ----------
{
"sysctl" : {
"net.ipv4.ip_forward" : {
"comment" : "Controls IP packet forwarding",
"value" : "0"
},
"net.ipv4.conf.default.rp_filter" : {
"comment" : "Controls source route verification",
"value" : "1"
}
...
...
}
-------------------------------------
Where it becomes difficult is trying to then incorporate pure Fact data
to influence or modify these decisions.
Lets say that I actually get back 20 keys of sysctl data, one of those
is ''vm.swappiness''. Most of my nodes have a value of
''10'', but lets say
hypothetically that I have a small set of nodes that require a different
value because of the amount of RAM available in the machine (a decision
needs to be made based on hardware, not business logic). This is purely
a Fact. Introducing another level of hierarchy for Fact
''memorytotal'' is
a bit silly in this case.
The sysctl class looks roughly like this:
------- sysctl.pp ----------
class sysctl {
$sysctl_hash = hiera_hash(''sysctl'')
create_resources(''sysctl'', $sysctl_hash)
}
------------------------------
I love that simplicity, however it''s difficult to introduce edge cases
that modify the data retrieved from Hiera based on Facts. Class
inheritance won''t work because create_resources() seems to insert into
the catalog in an uninheritable way - bug report or fixable with Ruby
DSL perhaps? Filling this class full of "if ($fact) modify hash" to
munge the data pulled from Hiera seems dirty too.
There may be no elegant solution and as you say, 80-90% may have to do.
-Luke
On 11/05/12 16:53, Gary Larizza wrote:> I see this with people looking to move to the hierarchical system that
> Hiera brings. It essentially boils down to "How do I do this without
> having a ton of hierarchy levels?". Usually we tend to recommend
> using the hierarchy to hit the 80% mark for the data you need in your
> modules. Anything that''s module-specific-data should then be
broken
> out to a data.pp or params.pp file with conditional logic there. I
> tend to ask people: "Is this something others are going to hit when
> they try to use the module too?", as in - "Are there path
differences
> between operating systems?", or "Are there important changes to
the
> data between RHEL 5 and 6?". If the answer to these is yes, then I
> tend to favor putting that data into a module''s data class so that
> it''s exposed for ANYONE who wants to use the module. Why would
you
> want to hide these differences in the hierarchy - especially if others
> might run into them?
>
> Does this sound similar to the problems you''re facing? Or is this
a
> case where you have custom facts that are specific to your
> organization that determine how you manage sysctl?
>
>
> On Fri, May 11, 2012 at 8:42 AM, Luke Bigum <Luke.Bigum@lmax.com
> <mailto:Luke.Bigum@lmax.com>> wrote:
>
> Hi all,
>
> I''ve been improving our sysctl module and come across an
> interesting design problem I''d like feedback on.
>
> I approached the re-factor with Hiera in mind - I would put all
> our sysctl data in Hiera hash and pull that into a hiera_hash,
> merging the hierarchy of data and allowing higher priority sysctl
> settings to override the baseline defaults. I then use
> create_resources to write sysctl.conf. Works great to start with,
> but now I come across more and more cases where the sysctl data is
> dependent on machine logic (virtual vs physical, types of
> hardware, etc) that doesn''t seem right to put into Hiera as
I''d
> have a complex hierarchy for a bunch of edge case Facts.
>
> I seem to need to make decisions on two sources: business logic in
> Hiera hierarchy (that''s easy with merging hashes) as well as
> considering what Facts or Classes applies to a node (machine
> logic). That''s not trivial to do, especially with a
potentially
> large set of data like sysctl.conf keys.
>
> Does anyone have any thoughts or tips on how they might be
> managing a similar situation?
>
> Thanks,
>
> -Luke
>
> --
> Luke Bigum
>
> Information Systems
> Ph: +44 (0) 20 3192 2520 <tel:%2B44%20%280%29%2020%203192%202520>
> luke.bigum@lmax.com <mailto:luke.bigum@lmax.com> |
http://www.lmax.com
> LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN
>
>
> FX and CFDs are leveraged products that can result in losses exceeding
> your deposit. They are not suitable for everyone so please ensure you
> fully understand the risks involved. The information in this
> email is not
> directed at residents of the United States of America or any other
> jurisdiction where trading in CFDs and/or FX is restricted or
> prohibited
> by local laws or regulations.
>
> The information in this email and any attachment is confidential
> and is
> intended only for the named recipient(s). The email may not be
> disclosed
> or used by any person other than the addressee, nor may it be
> copied in
> any way. If you are not the intended recipient please notify the
> sender
> immediately and delete any copies of this message. Any unauthorised
> copying, disclosure or distribution of the material in this e-mail is
> strictly forbidden.
>
> LMAX operates a multilateral trading facility. Authorised and
> regulated by the Financial Services Authority (firm registration
> number 509778) and
> is registered in England and Wales (number 06505809). Our
> registered address is Yellow Building, 1A Nicholas Road, London, W11
> 4AN.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com
> <mailto:puppet-users@googlegroups.com>.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com
> <mailto:puppet-users%2Bunsubscribe@googlegroups.com>.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
>
>
> --
>
> Gary Larizza
> Professional Services Engineer
> Puppet Labs
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
--
Luke Bigum
Information Systems
Ph: +44 (0) 20 3192 2520
luke.bigum@lmax.com | http://www.lmax.com
LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN
FX and CFDs are leveraged products that can result in losses exceeding
your deposit. They are not suitable for everyone so please ensure you
fully understand the risks involved. The information in this email is not
directed at residents of the United States of America or any other
jurisdiction where trading in CFDs and/or FX is restricted or prohibited
by local laws or regulations.
The information in this email and any attachment is confidential and is
intended only for the named recipient(s). The email may not be disclosed
or used by any person other than the addressee, nor may it be copied in
any way. If you are not the intended recipient please notify the sender
immediately and delete any copies of this message. Any unauthorised
copying, disclosure or distribution of the material in this e-mail is
strictly forbidden.
LMAX operates a multilateral trading facility. Authorised and regulated
by the Financial Services Authority (firm registration number 509778) and
is registered in England and Wales (number 06505809).
Our registered address is Yellow Building, 1A Nicholas Road, London, W11
4AN.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Aaron Grewell
2012-May-11 16:56 UTC
Re: [Puppet Users] how best to combine business data (Hiera) and machine data (Facts)
I suppose you could create a separate class for the entries that will be fact-driven versus Hiera-driven. You wouldn''t be able to use a single template, but either augeas or concat should work. I wouldn''t call it elegant, but the code might be less ugly. On Fri, May 11, 2012 at 9:47 AM, Luke Bigum <Luke.Bigum@lmax.com> wrote:> Hi Gary, > > Not quite... Let me go into more detail. > > I''m trying to handle sysctl "perfectly" which is probably my real problem. > Hiera''s ability to merge hashes together makes it perfect for arriving at > one set of sysctl options for a server based on "business logic" (my > hierarchy). For Hiera data on ''someserver'' below which has ''some_role'', > calling hiera_hash in a Puppet manifest will give me IP forwarding set and > rp_filter set, which is what I want: > > --------- some_role.json ------------ > { > "sysctl" : { > "net.ipv4.ip_forward" : { > "comment" : "Controls IP packet forwarding", > "value" : "1" > } > } > -------------------------------------------- > --------- common.json ---------- > { > "sysctl" : { > "net.ipv4.ip_forward" : { > "comment" : "Controls IP packet forwarding", > "value" : "0" > }, > "net.ipv4.conf.default.rp_filter" : { > "comment" : "Controls source route verification", > "value" : "1" > } > ... > ... > } > ------------------------------------- > > Where it becomes difficult is trying to then incorporate pure Fact data to > influence or modify these decisions. > > Lets say that I actually get back 20 keys of sysctl data, one of those is > ''vm.swappiness''. Most of my nodes have a value of ''10'', but lets say > hypothetically that I have a small set of nodes that require a different > value because of the amount of RAM available in the machine (a decision > needs to be made based on hardware, not business logic). This is purely a > Fact. Introducing another level of hierarchy for Fact ''memorytotal'' is a bit > silly in this case. > > The sysctl class looks roughly like this: > > ------- sysctl.pp ---------- > class sysctl { > $sysctl_hash = hiera_hash(''sysctl'') > create_resources(''sysctl'', $sysctl_hash) > } > ------------------------------ > > I love that simplicity, however it''s difficult to introduce edge cases that > modify the data retrieved from Hiera based on Facts. Class inheritance won''t > work because create_resources() seems to insert into the catalog in an > uninheritable way - bug report or fixable with Ruby DSL perhaps? Filling > this class full of "if ($fact) modify hash" to munge the data pulled from > Hiera seems dirty too. > > There may be no elegant solution and as you say, 80-90% may have to do. > > -Luke > > > On 11/05/12 16:53, Gary Larizza wrote: > > I see this with people looking to move to the hierarchical system that Hiera > brings. It essentially boils down to "How do I do this without having a ton > of hierarchy levels?". Usually we tend to recommend using the hierarchy to > hit the 80% mark for the data you need in your modules. Anything that''s > module-specific-data should then be broken out to a data.pp or params.pp > file with conditional logic there. I tend to ask people: "Is this something > others are going to hit when they try to use the module too?", as in - "Are > there path differences between operating systems?", or "Are there important > changes to the data between RHEL 5 and 6?". If the answer to these is yes, > then I tend to favor putting that data into a module''s data class so that > it''s exposed for ANYONE who wants to use the module. Why would you want to > hide these differences in the hierarchy - especially if others might run > into them? > > Does this sound similar to the problems you''re facing? Or is this a case > where you have custom facts that are specific to your organization that > determine how you manage sysctl? > > > On Fri, May 11, 2012 at 8:42 AM, Luke Bigum <Luke.Bigum@lmax.com> wrote: >> >> Hi all, >> >> I''ve been improving our sysctl module and come across an interesting >> design problem I''d like feedback on. >> >> I approached the re-factor with Hiera in mind - I would put all our sysctl >> data in Hiera hash and pull that into a hiera_hash, merging the hierarchy of >> data and allowing higher priority sysctl settings to override the baseline >> defaults. I then use create_resources to write sysctl.conf. Works great to >> start with, but now I come across more and more cases where the sysctl data >> is dependent on machine logic (virtual vs physical, types of hardware, etc) >> that doesn''t seem right to put into Hiera as I''d have a complex hierarchy >> for a bunch of edge case Facts. >> >> I seem to need to make decisions on two sources: business logic in Hiera >> hierarchy (that''s easy with merging hashes) as well as considering what >> Facts or Classes applies to a node (machine logic). That''s not trivial to >> do, especially with a potentially large set of data like sysctl.conf keys. >> >> Does anyone have any thoughts or tips on how they might be managing a >> similar situation? >> >> Thanks, >> >> -Luke >> >> -- >> Luke Bigum >> >> Information Systems >> Ph: +44 (0) 20 3192 2520 >> luke.bigum@lmax.com | http://www.lmax.com >> LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN >> >> >> FX and CFDs are leveraged products that can result in losses exceeding >> your deposit. They are not suitable for everyone so please ensure you >> fully understand the risks involved. The information in this email is not >> directed at residents of the United States of America or any other >> jurisdiction where trading in CFDs and/or FX is restricted or prohibited >> by local laws or regulations. >> >> The information in this email and any attachment is confidential and is >> intended only for the named recipient(s). The email may not be disclosed >> or used by any person other than the addressee, nor may it be copied in >> any way. If you are not the intended recipient please notify the sender >> immediately and delete any copies of this message. Any unauthorised >> copying, disclosure or distribution of the material in this e-mail is >> strictly forbidden. >> >> LMAX operates a multilateral trading facility. Authorised and regulated >> by the Financial Services Authority (firm registration number 509778) and >> is registered in England and Wales (number 06505809). Our registered >> address is Yellow Building, 1A Nicholas Road, London, W11 >> 4AN. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> > > > > -- > > Gary Larizza > Professional Services Engineer > Puppet Labs > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > > > -- > Luke Bigum > > Information Systems > Ph: +44 (0) 20 3192 2520 > luke.bigum@lmax.com | http://www.lmax.com > LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN > > > FX and CFDs are leveraged products that can result in losses exceeding > your deposit. They are not suitable for everyone so please ensure you > fully understand the risks involved. The information in this email is not > directed at residents of the United States of America or any other > jurisdiction where trading in CFDs and/or FX is restricted or prohibited > by local laws or regulations. > > The information in this email and any attachment is confidential and is > intended only for the named recipient(s). The email may not be disclosed > or used by any person other than the addressee, nor may it be copied in > any way. If you are not the intended recipient please notify the sender > immediately and delete any copies of this message. Any unauthorised > copying, disclosure or distribution of the material in this e-mail is > strictly forbidden. > > LMAX operates a multilateral trading facility. Authorised and regulated > by the Financial Services Authority (firm registration number 509778) and > is registered in England and Wales (number 06505809). > Our registered address is Yellow Building, 1A Nicholas Road, London, W11 > 4AN. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Gary Larizza
2012-May-11 17:19 UTC
Re: [Puppet Users] how best to combine business data (Hiera) and machine data (Facts)
I see (and I saw your post on create_resources - I tested it out and also noticed that inheritance didn''t work as expected for me either). My initial thought would be to try and break down the individual sysctl entries as individual resources that could be declared with a defined type (as Aaron mentioned) - then you can use hiera to declare them with create_resources() AS WELL as individually declaring additional resources. On Fri, May 11, 2012 at 9:47 AM, Luke Bigum <Luke.Bigum@lmax.com> wrote:> Hi Gary, > > Not quite... Let me go into more detail. > > I''m trying to handle sysctl "perfectly" which is probably my real problem. > Hiera''s ability to merge hashes together makes it perfect for arriving at > one set of sysctl options for a server based on "business logic" (my > hierarchy). For Hiera data on ''someserver'' below which has ''some_role'', > calling hiera_hash in a Puppet manifest will give me IP forwarding set and > rp_filter set, which is what I want: > > --------- some_role.json ------------ > { > "sysctl" : { > "net.ipv4.ip_forward" : { > "comment" : "Controls IP packet forwarding", > "value" : "1" > } > } > -------------------------------------------- > --------- common.json ---------- > { > "sysctl" : { > "net.ipv4.ip_forward" : { > "comment" : "Controls IP packet forwarding", > "value" : "0" > }, > "net.ipv4.conf.default.rp_filter" : { > "comment" : "Controls source route verification", > "value" : "1" > } > ... > ... > } > ------------------------------------- > > Where it becomes difficult is trying to then incorporate pure Fact data to > influence or modify these decisions. > > Lets say that I actually get back 20 keys of sysctl data, one of those is > ''vm.swappiness''. Most of my nodes have a value of ''10'', but lets say > hypothetically that I have a small set of nodes that require a different > value because of the amount of RAM available in the machine (a decision > needs to be made based on hardware, not business logic). This is purely a > Fact. Introducing another level of hierarchy for Fact ''memorytotal'' is a > bit silly in this case. > > The sysctl class looks roughly like this: > > ------- sysctl.pp ---------- > class sysctl { > $sysctl_hash = hiera_hash(''sysctl'') > create_resources(''sysctl'', $sysctl_hash) > } > ------------------------------ > > I love that simplicity, however it''s difficult to introduce edge cases > that modify the data retrieved from Hiera based on Facts. Class inheritance > won''t work because create_resources() seems to insert into the catalog in > an uninheritable way - bug report or fixable with Ruby DSL perhaps? Filling > this class full of "if ($fact) modify hash" to munge the data pulled from > Hiera seems dirty too. > > There may be no elegant solution and as you say, 80-90% may have to do. > > -Luke > > > On 11/05/12 16:53, Gary Larizza wrote: > > I see this with people looking to move to the hierarchical system that > Hiera brings. It essentially boils down to "How do I do this without > having a ton of hierarchy levels?". Usually we tend to recommend using the > hierarchy to hit the 80% mark for the data you need in your modules. > Anything that''s module-specific-data should then be broken out to a > data.pp or params.pp file with conditional logic there. I tend to ask > people: "Is this something others are going to hit when they try to use the > module too?", as in - "Are there path differences between operating > systems?", or "Are there important changes to the data between RHEL 5 and > 6?". If the answer to these is yes, then I tend to favor putting that data > into a module''s data class so that it''s exposed for ANYONE who wants to use > the module. Why would you want to hide these differences in the hierarchy > - especially if others might run into them? > > Does this sound similar to the problems you''re facing? Or is this a > case where you have custom facts that are specific to your organization > that determine how you manage sysctl? > > > On Fri, May 11, 2012 at 8:42 AM, Luke Bigum <Luke.Bigum@lmax.com> wrote: > >> Hi all, >> >> I''ve been improving our sysctl module and come across an interesting >> design problem I''d like feedback on. >> >> I approached the re-factor with Hiera in mind - I would put all our >> sysctl data in Hiera hash and pull that into a hiera_hash, merging the >> hierarchy of data and allowing higher priority sysctl settings to override >> the baseline defaults. I then use create_resources to write sysctl.conf. >> Works great to start with, but now I come across more and more cases where >> the sysctl data is dependent on machine logic (virtual vs physical, types >> of hardware, etc) that doesn''t seem right to put into Hiera as I''d have a >> complex hierarchy for a bunch of edge case Facts. >> >> I seem to need to make decisions on two sources: business logic in Hiera >> hierarchy (that''s easy with merging hashes) as well as considering what >> Facts or Classes applies to a node (machine logic). That''s not trivial to >> do, especially with a potentially large set of data like sysctl.conf keys. >> >> Does anyone have any thoughts or tips on how they might be managing a >> similar situation? >> >> Thanks, >> >> -Luke >> >> -- >> Luke Bigum >> >> Information Systems >> Ph: +44 (0) 20 3192 2520 >> luke.bigum@lmax.com | http://www.lmax.com >> LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN >> >> >> FX and CFDs are leveraged products that can result in losses exceeding >> your deposit. They are not suitable for everyone so please ensure you >> fully understand the risks involved. The information in this email is not >> directed at residents of the United States of America or any other >> jurisdiction where trading in CFDs and/or FX is restricted or prohibited >> by local laws or regulations. >> >> The information in this email and any attachment is confidential and is >> intended only for the named recipient(s). The email may not be disclosed >> or used by any person other than the addressee, nor may it be copied in >> any way. If you are not the intended recipient please notify the sender >> immediately and delete any copies of this message. Any unauthorised >> copying, disclosure or distribution of the material in this e-mail is >> strictly forbidden. >> >> LMAX operates a multilateral trading facility. Authorised and regulated >> by the Financial Services Authority (firm registration number 509778) and >> is registered in England and Wales (number 06505809). Our registered >> address is Yellow Building, 1A Nicholas Road, London, W11 >> 4AN. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> >> > > > -- > > Gary Larizza > Professional Services Engineer > Puppet Labs > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > > > -- > Luke Bigum > > Information Systems > Ph: +44 (0) 20 3192 2520luke.bigum@lmax.com | http://www.lmax.com > LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN > > > FX and CFDs are leveraged products that can result in losses exceeding > your deposit. They are not suitable for everyone so please ensure you > fully understand the risks involved. The information in this email is not > directed at residents of the United States of America or any other > jurisdiction where trading in CFDs and/or FX is restricted or prohibited > by local laws or regulations. > > The information in this email and any attachment is confidential and is > intended only for the named recipient(s). The email may not be disclosed > or used by any person other than the addressee, nor may it be copied in > any way. If you are not the intended recipient please notify the sender > immediately and delete any copies of this message. Any unauthorised > copying, disclosure or distribution of the material in this e-mail is > strictly forbidden. > > LMAX operates a multilateral trading facility. Authorised and regulated > by the Financial Services Authority (firm registration number 509778) and > is registered in England and Wales (number 06505809). > Our registered address is Yellow Building, 1A Nicholas Road, London, W11 > 4AN. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- Gary Larizza Professional Services Engineer Puppet Labs -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Jeff
2012-May-11 17:45 UTC
Re: [Puppet Users] how best to combine business data (Hiera) and machine data (Facts)
I have a similar issue with Hiera when decisions depend on more than facts alone... thus exluded from the hierarchy. In example, the hostname can tell a node is a database server (fact: role) but only a custom group will tell it should run Oracle 10g or 11g (product) and the proper sysctl to apply specifically. I can''t build a fact about something that ain''t installed yet and I don''t want a local file to define group membership (giving up control). The node to group membership has to be in a CMDB somewhere. So far I think using an ENC with Hiera at the same time is the solution to cover all cases.... unless you guys know of a better way. jean On Friday, 11 May 2012 13:19:33 UTC-4, Gary Larizza wrote:> I see (and I saw your post on create_resources - I tested it out and also > noticed that inheritance didn''t work as expected for me either). My > initial thought would be to try and break down the individual sysctl > entries as individual resources that could be declared with a defined type > (as Aaron mentioned) - then you can use hiera to declare them with > create_resources() AS WELL as individually declaring additional resources. > > > > On Fri, May 11, 2012 at 9:47 AM, Luke Bigum <Luke.Bigum@lmax.com> wrote: > >> Hi Gary, >> >> Not quite... Let me go into more detail. >> >> I''m trying to handle sysctl "perfectly" which is probably my real >> problem. Hiera''s ability to merge hashes together makes it perfect for >> arriving at one set of sysctl options for a server based on "business >> logic" (my hierarchy). For Hiera data on ''someserver'' below which has >> ''some_role'', calling hiera_hash in a Puppet manifest will give me IP >> forwarding set and rp_filter set, which is what I want: >> >> --------- some_role.json ------------ >> { >> "sysctl" : { >> "net.ipv4.ip_forward" : { >> "comment" : "Controls IP packet forwarding", >> "value" : "1" >> } >> } >> -------------------------------------------- >> --------- common.json ---------- >> { >> "sysctl" : { >> "net.ipv4.ip_forward" : { >> "comment" : "Controls IP packet forwarding", >> "value" : "0" >> }, >> "net.ipv4.conf.default.rp_filter" : { >> "comment" : "Controls source route verification", >> "value" : "1" >> } >> ... >> ... >> } >> ------------------------------------- >> >> Where it becomes difficult is trying to then incorporate pure Fact data >> to influence or modify these decisions. >> >> Lets say that I actually get back 20 keys of sysctl data, one of those is >> ''vm.swappiness''. Most of my nodes have a value of ''10'', but lets say >> hypothetically that I have a small set of nodes that require a different >> value because of the amount of RAM available in the machine (a decision >> needs to be made based on hardware, not business logic). This is purely a >> Fact. Introducing another level of hierarchy for Fact ''memorytotal'' is a >> bit silly in this case. >> >> The sysctl class looks roughly like this: >> >> ------- sysctl.pp ---------- >> class sysctl { >> $sysctl_hash = hiera_hash(''sysctl'') >> create_resources(''sysctl'', $sysctl_hash) >> } >> ------------------------------ >> >> I love that simplicity, however it''s difficult to introduce edge cases >> that modify the data retrieved from Hiera based on Facts. Class inheritance >> won''t work because create_resources() seems to insert into the catalog in >> an uninheritable way - bug report or fixable with Ruby DSL perhaps? Filling >> this class full of "if ($fact) modify hash" to munge the data pulled from >> Hiera seems dirty too. >> >> There may be no elegant solution and as you say, 80-90% may have to do. >> >> -Luke >> >> >> On 11/05/12 16:53, Gary Larizza wrote: >> >> I see this with people looking to move to the hierarchical system that >> Hiera brings. It essentially boils down to "How do I do this without >> having a ton of hierarchy levels?". Usually we tend to recommend using the >> hierarchy to hit the 80% mark for the data you need in your modules. >> Anything that''s module-specific-data should then be broken out to a >> data.pp or params.pp file with conditional logic there. I tend to ask >> people: "Is this something others are going to hit when they try to use the >> module too?", as in - "Are there path differences between operating >> systems?", or "Are there important changes to the data between RHEL 5 and >> 6?". If the answer to these is yes, then I tend to favor putting that data >> into a module''s data class so that it''s exposed for ANYONE who wants to use >> the module. Why would you want to hide these differences in the hierarchy >> - especially if others might run into them? >> >> Does this sound similar to the problems you''re facing? Or is this a >> case where you have custom facts that are specific to your organization >> that determine how you manage sysctl? >> >> >> On Fri, May 11, 2012 at 8:42 AM, Luke Bigum <Luke.Bigum@lmax.com> wrote: >> >>> Hi all, >>> >>> I''ve been improving our sysctl module and come across an interesting >>> design problem I''d like feedback on. >>> >>> I approached the re-factor with Hiera in mind - I would put all our >>> sysctl data in Hiera hash and pull that into a hiera_hash, merging the >>> hierarchy of data and allowing higher priority sysctl settings to override >>> the baseline defaults. I then use create_resources to write sysctl.conf. >>> Works great to start with, but now I come across more and more cases where >>> the sysctl data is dependent on machine logic (virtual vs physical, types >>> of hardware, etc) that doesn''t seem right to put into Hiera as I''d have a >>> complex hierarchy for a bunch of edge case Facts. >>> >>> I seem to need to make decisions on two sources: business logic in Hiera >>> hierarchy (that''s easy with merging hashes) as well as considering what >>> Facts or Classes applies to a node (machine logic). That''s not trivial to >>> do, especially with a potentially large set of data like sysctl.conf keys. >>> >>> Does anyone have any thoughts or tips on how they might be managing a >>> similar situation? >>> >>> Thanks, >>> >>> -Luke >>> >>> -- >>> Luke Bigum >>> >>> Information Systems >>> Ph: +44 (0) 20 3192 2520 >>> luke.bigum@lmax.com | http://www.lmax.com >>> LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN >>> >>> >>> FX and CFDs are leveraged products that can result in losses exceeding >>> your deposit. They are not suitable for everyone so please ensure you >>> fully understand the risks involved. The information in this email is >>> not >>> directed at residents of the United States of America or any other >>> jurisdiction where trading in CFDs and/or FX is restricted or prohibited >>> by local laws or regulations. >>> >>> The information in this email and any attachment is confidential and is >>> intended only for the named recipient(s). The email may not be disclosed >>> or used by any person other than the addressee, nor may it be copied in >>> any way. If you are not the intended recipient please notify the sender >>> immediately and delete any copies of this message. Any unauthorised >>> copying, disclosure or distribution of the material in this e-mail is >>> strictly forbidden. >>> >>> LMAX operates a multilateral trading facility. Authorised and regulated >>> by the Financial Services Authority (firm registration number 509778) and >>> is registered in England and Wales (number 06505809). Our registered >>> address is Yellow Building, 1A Nicholas Road, London, W11 >>> 4AN. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> To post to this group, send email to puppet-users@googlegroups.com. >>> To unsubscribe from this group, send email to >>> puppet-users+unsubscribe@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/puppet-users?hl=en. >>> >>> >> >> >> -- >> >> Gary Larizza >> Professional Services Engineer >> Puppet Labs >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> >> >> >> -- >> Luke Bigum >> >> Information Systems >> Ph: +44 (0) 20 3192 2520luke.bigum@lmax.com | http://www.lmax.com >> LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN >> >> >> FX and CFDs are leveraged products that can result in losses exceeding >> your deposit. They are not suitable for everyone so please ensure you >> fully understand the risks involved. The information in this email is not >> directed at residents of the United States of America or any other >> jurisdiction where trading in CFDs and/or FX is restricted or prohibited >> by local laws or regulations. >> >> The information in this email and any attachment is confidential and is >> intended only for the named recipient(s). The email may not be disclosed >> or used by any person other than the addressee, nor may it be copied in >> any way. If you are not the intended recipient please notify the sender >> immediately and delete any copies of this message. Any unauthorised >> copying, disclosure or distribution of the material in this e-mail is >> strictly forbidden. >> >> LMAX operates a multilateral trading facility. Authorised and regulated >> by the Financial Services Authority (firm registration number 509778) and >> is registered in England and Wales (number 06505809). >> Our registered address is Yellow Building, 1A Nicholas Road, London, W11 >> 4AN. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> > > > > -- > > Gary Larizza > Professional Services Engineer > Puppet Labs > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/0NSu9kWGZggJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Wolf Noble
2012-May-14 18:44 UTC
Re: [Puppet Users] how best to combine business data (Hiera) and machine data (Facts)
Hi Luke,
Would adding fqdn to the bottom of your hierarchy give you the ability to
override the values on the outliers that you can''t automagically set
while retaining elegance in your class? Or am I missing something?
Wolf Noble
Datapipe Managed IT Services
On May 11, 2012, at 12:45 PM, Jeff wrote:
I have a similar issue with Hiera when decisions depend on more than facts
alone... thus exluded from the hierarchy.
In example, the hostname can tell a node is a database server (fact: role) but
only a custom group will tell it should run Oracle 10g or 11g (product) and the
proper sysctl to apply specifically. I can''t build a fact about
something that ain''t installed yet and I don''t want a local
file to define group membership (giving up control). The node to group
membership has to be in a CMDB somewhere.
So far I think using an ENC with Hiera at the same time is the solution to cover
all cases.... unless you guys know of a better way.
jean
On Friday, 11 May 2012 13:19:33 UTC-4, Gary Larizza wrote:
I see (and I saw your post on create_resources - I tested it out and also
noticed that inheritance didn''t work as expected for me either). My
initial thought would be to try and break down the individual sysctl entries as
individual resources that could be declared with a defined type (as Aaron
mentioned) - then you can use hiera to declare them with create_resources() AS
WELL as individually declaring additional resources.
On Fri, May 11, 2012 at 9:47 AM, Luke Bigum
<Luke.Bigum@lmax.com<mailto:Luke.Bigum@lmax.com>> wrote:
Hi Gary,
Not quite... Let me go into more detail.
I''m trying to handle sysctl "perfectly" which is probably my
real problem. Hiera''s ability to merge hashes together makes it perfect
for arriving at one set of sysctl options for a server based on "business
logic" (my hierarchy). For Hiera data on ''someserver''
below which has ''some_role'', calling hiera_hash in a Puppet
manifest will give me IP forwarding set and rp_filter set, which is what I want:
--------- some_role.json ------------
{
"sysctl" : {
"net.ipv4.ip_forward" : {
"comment" : "Controls IP packet forwarding",
"value" : "1"
}
}
--------------------------------------------
--------- common.json ----------
{
"sysctl" : {
"net.ipv4.ip_forward" : {
"comment" : "Controls IP packet forwarding",
"value" : "0"
},
"net.ipv4.conf.default.rp_filter" : {
"comment" : "Controls source route verification",
"value" : "1"
}
...
...
}
-------------------------------------
Where it becomes difficult is trying to then incorporate pure Fact data to
influence or modify these decisions.
Lets say that I actually get back 20 keys of sysctl data, one of those is
''vm.swappiness''. Most of my nodes have a value of
''10'', but lets say hypothetically that I have a small set of
nodes that require a different value because of the amount of RAM available in
the machine (a decision needs to be made based on hardware, not business logic).
This is purely a Fact. Introducing another level of hierarchy for Fact
''memorytotal'' is a bit silly in this case.
The sysctl class looks roughly like this:
------- sysctl.pp ----------
class sysctl {
$sysctl_hash = hiera_hash(''sysctl'')
create_resources(''sysctl'', $sysctl_hash)
}
------------------------------
I love that simplicity, however it''s difficult to introduce edge cases
that modify the data retrieved from Hiera based on Facts. Class inheritance
won''t work because create_resources() seems to insert into the catalog
in an uninheritable way - bug report or fixable with Ruby DSL perhaps? Filling
this class full of "if ($fact) modify hash" to munge the data pulled
from Hiera seems dirty too.
There may be no elegant solution and as you say, 80-90% may have to do.
-Luke
On 11/05/12 16:53, Gary Larizza wrote:
I see this with people looking to move to the hierarchical system that Hiera
brings. It essentially boils down to "How do I do this without having a
ton of hierarchy levels?". Usually we tend to recommend using the
hierarchy to hit the 80% mark for the data you need in your modules. Anything
that''s module-specific-data should then be broken out to a data.pp or
params.pp file with conditional logic there. I tend to ask people: "Is
this something others are going to hit when they try to use the module
too?", as in - "Are there path differences between operating
systems?", or "Are there important changes to the data between RHEL 5
and 6?". If the answer to these is yes, then I tend to favor putting that
data into a module''s data class so that it''s exposed for
ANYONE who wants to use the module. Why would you want to hide these
differences in the hierarchy - especially if others might run into them?
Does this sound similar to the problems you''re facing? Or is this a
case where you have custom facts that are specific to your organization that
determine how you manage sysctl?
On Fri, May 11, 2012 at 8:42 AM, Luke Bigum
<Luke.Bigum@lmax.com<mailto:Luke.Bigum@lmax.com>> wrote:
Hi all,
I''ve been improving our sysctl module and come across an interesting
design problem I''d like feedback on.
I approached the re-factor with Hiera in mind - I would put all our sysctl data
in Hiera hash and pull that into a hiera_hash, merging the hierarchy of data and
allowing higher priority sysctl settings to override the baseline defaults. I
then use create_resources to write sysctl.conf. Works great to start with, but
now I come across more and more cases where the sysctl data is dependent on
machine logic (virtual vs physical, types of hardware, etc) that
doesn''t seem right to put into Hiera as I''d have a complex
hierarchy for a bunch of edge case Facts.
I seem to need to make decisions on two sources: business logic in Hiera
hierarchy (that''s easy with merging hashes) as well as considering what
Facts or Classes applies to a node (machine logic). That''s not trivial
to do, especially with a potentially large set of data like sysctl.conf keys.
Does anyone have any thoughts or tips on how they might be managing a similar
situation?
Thanks,
-Luke
--
Luke Bigum
Information Systems
Ph: +44 (0) 20 3192 2520
luke.bigum@lmax.com<mailto:luke.bigum@lmax.com> |
http://www.lmax.com<http://www.lmax.com/>
LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN
FX and CFDs are leveraged products that can result in losses exceeding
your deposit. They are not suitable for everyone so please ensure you
fully understand the risks involved. The information in this email is not
directed at residents of the United States of America or any other
jurisdiction where trading in CFDs and/or FX is restricted or prohibited
by local laws or regulations.
The information in this email and any attachment is confidential and is
intended only for the named recipient(s). The email may not be disclosed
or used by any person other than the addressee, nor may it be copied in
any way. If you are not the intended recipient please notify the sender
immediately and delete any copies of this message. Any unauthorised
copying, disclosure or distribution of the material in this e-mail is
strictly forbidden.
LMAX operates a multilateral trading facility. Authorised and regulated by the
Financial Services Authority (firm registration number 509778) and
is registered in England and Wales (number 06505809). Our registered address is
Yellow Building, 1A Nicholas Road, London, W11
4AN.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to
puppet-users@googlegroups.com<mailto:puppet-users@googlegroups.com>.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com<mailto:puppet-users%2Bunsubscribe@googlegroups.com>.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
--
Gary Larizza
Professional Services Engineer
Puppet Labs
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to
puppet-users@googlegroups.com<mailto:puppet-users@googlegroups.com>.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com<mailto:puppet-users+unsubscribe@googlegroups.com>.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
--
Luke Bigum
Information Systems
Ph: +44 (0) 20 3192 2520
luke.bigum@lmax.com<mailto:luke.bigum@lmax.com> |
http://www.lmax.com<http://www.lmax.com/>
LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN
FX and CFDs are leveraged products that can result in losses exceeding
your deposit. They are not suitable for everyone so please ensure you
fully understand the risks involved. The information in this email is not
directed at residents of the United States of America or any other
jurisdiction where trading in CFDs and/or FX is restricted or prohibited
by local laws or regulations.
The information in this email and any attachment is confidential and is
intended only for the named recipient(s). The email may not be disclosed
or used by any person other than the addressee, nor may it be copied in
any way. If you are not the intended recipient please notify the sender
immediately and delete any copies of this message. Any unauthorised
copying, disclosure or distribution of the material in this e-mail is
strictly forbidden.
LMAX operates a multilateral trading facility. Authorised and regulated
by the Financial Services Authority (firm registration number 509778) and
is registered in England and Wales (number 06505809).
Our registered address is Yellow Building, 1A Nicholas Road, London, W11
4AN.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to
puppet-users@googlegroups.com<mailto:puppet-users@googlegroups.com>.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com<mailto:puppet-users%2Bunsubscribe@googlegroups.com>.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
--
Gary Larizza
Professional Services Engineer
Puppet Labs
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/0NSu9kWGZggJ.
To post to this group, send email to
puppet-users@googlegroups.com<mailto:puppet-users@googlegroups.com>.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com<mailto:puppet-users+unsubscribe@googlegroups.com>.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
________________________________
This message may contain confidential or privileged information. If you are not
the intended recipient, please advise us immediately and delete this message.
See http://www.datapipe.com/legal/email_disclaimer/ for further information on
confidentiality and the risks of non-secure electronic communication. If you
cannot access these links, please notify us by reply message and we will send
the contents to you.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Luke Bigum
2012-May-15 10:57 UTC
Re: [Puppet Users] how best to combine business data (Hiera) and machine data (Facts)
Hi Wolf,
At the top of the hierarchy, yes (the hash merge is done top to bottom,
where the top is the highest of the hierarchy, see mine below). I have
this in place already and you''re right it will work elegantly, however
it moves the ''decision'' of what to do into Hiera and you have
to "know"
that certain nodes require certain vm.swappiness
values depending on what hardware they are on:
---
:backends: - json
:hierarchy: - %{fqdn}
- %{lmax_role}_role
- %{lmax_env}_server
- %{pop}.tradefair
- common
:json:
:datadir: - /etc/puppet/private/
- /etc/puppet/environments/%{environment}/hiera_data_store/
- /etc/puppet/environments/%{environment}/rebirth_data_store/
-
/etc/puppet/environments/%{environment}/satellite_system_groups/
Thank you for everyone else who replied too. Re Gary, I already use a
slightly modified version of puppetlabs-sysctl''s custom type and
provider, so I can define individual sysctl lines as well as with
create_resources, it''s the combination of the two I''ve got
problems
with. My next crazy idea is to try wrap them all in a class and use
inline_template to dynamically generate class names - I hope I can
override classes that are created with create_resources if resources
cannot be - but that could get very messy very quickly, I''ll let you
know how I go.
Thanks,
-Luke
On 14/05/12 19:44, Wolf Noble wrote:> Hi Luke,
>
> Would adding fqdn to the bottom of your hierarchy give you the ability to
override the values on the outliers that you can''t automagically set
while retaining elegance in your class? Or am I missing something?
>
>
>
> Wolf Noble
> Datapipe Managed IT Services
>
>
>
> On May 11, 2012, at 12:45 PM, Jeff wrote:
>
> I have a similar issue with Hiera when decisions depend on more than facts
alone... thus exluded from the hierarchy.
>
> In example, the hostname can tell a node is a database server (fact: role)
but only a custom group will tell it should run Oracle 10g or 11g (product) and
the proper sysctl to apply specifically. I can''t build a fact about
something that ain''t installed yet and I don''t want a local
file to define group membership (giving up control). The node to group
membership has to be in a CMDB somewhere.
>
> So far I think using an ENC with Hiera at the same time is the solution to
cover all cases.... unless you guys know of a better way.
>
> jean
>
> On Friday, 11 May 2012 13:19:33 UTC-4, Gary Larizza wrote:
> I see (and I saw your post on create_resources - I tested it out and also
noticed that inheritance didn''t work as expected for me either). My
initial thought would be to try and break down the individual sysctl entries as
individual resources that could be declared with a defined type (as Aaron
mentioned) - then you can use hiera to declare them with create_resources() AS
WELL as individually declaring additional resources.
>
>
>
> On Fri, May 11, 2012 at 9:47 AM, Luke
Bigum<Luke.Bigum@lmax.com<mailto:Luke.Bigum@lmax.com>> wrote:
> Hi Gary,
>
> Not quite... Let me go into more detail.
>
> I''m trying to handle sysctl "perfectly" which is
probably my real problem. Hiera''s ability to merge hashes together
makes it perfect for arriving at one set of sysctl options for a server based on
"business logic" (my hierarchy). For Hiera data on
''someserver'' below which has ''some_role'',
calling hiera_hash in a Puppet manifest will give me IP forwarding set and
rp_filter set, which is what I want:
>
> --------- some_role.json ------------
> {
> "sysctl" : {
> "net.ipv4.ip_forward" : {
> "comment" : "Controls IP packet forwarding",
> "value" : "1"
> }
> }
> --------------------------------------------
> --------- common.json ----------
> {
> "sysctl" : {
> "net.ipv4.ip_forward" : {
> "comment" : "Controls IP packet forwarding",
> "value" : "0"
> },
> "net.ipv4.conf.default.rp_filter" : {
> "comment" : "Controls source route
verification",
> "value" : "1"
> }
> ...
> ...
> }
> -------------------------------------
>
> Where it becomes difficult is trying to then incorporate pure Fact data to
influence or modify these decisions.
>
> Lets say that I actually get back 20 keys of sysctl data, one of those is
''vm.swappiness''. Most of my nodes have a value of
''10'', but lets say hypothetically that I have a small set of
nodes that require a different value because of the amount of RAM available in
the machine (a decision needs to be made based on hardware, not business logic).
This is purely a Fact. Introducing another level of hierarchy for Fact
''memorytotal'' is a bit silly in this case.
>
> The sysctl class looks roughly like this:
>
> ------- sysctl.pp ----------
> class sysctl {
> $sysctl_hash = hiera_hash(''sysctl'')
> create_resources(''sysctl'', $sysctl_hash)
> }
> ------------------------------
>
> I love that simplicity, however it''s difficult to introduce edge
cases that modify the data retrieved from Hiera based on Facts. Class
inheritance won''t work because create_resources() seems to insert into
the catalog in an uninheritable way - bug report or fixable with Ruby DSL
perhaps? Filling this class full of "if ($fact) modify hash" to munge
the data pulled from Hiera seems dirty too.
>
> There may be no elegant solution and as you say, 80-90% may have to do.
>
> -Luke
>
>
> On 11/05/12 16:53, Gary Larizza wrote:
> I see this with people looking to move to the hierarchical system that
Hiera brings. It essentially boils down to "How do I do this without
having a ton of hierarchy levels?". Usually we tend to recommend using the
hierarchy to hit the 80% mark for the data you need in your modules. Anything
that''s module-specific-data should then be broken out to a data.pp or
params.pp file with conditional logic there. I tend to ask people: "Is
this something others are going to hit when they try to use the module
too?", as in - "Are there path differences between operating
systems?", or "Are there important changes to the data between RHEL 5
and 6?". If the answer to these is yes, then I tend to favor putting that
data into a module''s data class so that it''s exposed for
ANYONE who wants to use the module. Why would you want to hide these
differences in the hierarchy - especially if others might run into them?
>
> Does this sound similar to the problems you''re facing? Or is this
a case where you have custom facts that are specific to your organization that
determine how you manage sysctl?
>
>
> On Fri, May 11, 2012 at 8:42 AM, Luke
Bigum<Luke.Bigum@lmax.com<mailto:Luke.Bigum@lmax.com>> wrote:
> Hi all,
>
> I''ve been improving our sysctl module and come across an
interesting design problem I''d like feedback on.
>
> I approached the re-factor with Hiera in mind - I would put all our sysctl
data in Hiera hash and pull that into a hiera_hash, merging the hierarchy of
data and allowing higher priority sysctl settings to override the baseline
defaults. I then use create_resources to write sysctl.conf. Works great to start
with, but now I come across more and more cases where the sysctl data is
dependent on machine logic (virtual vs physical, types of hardware, etc) that
doesn''t seem right to put into Hiera as I''d have a complex
hierarchy for a bunch of edge case Facts.
>
> I seem to need to make decisions on two sources: business logic in Hiera
hierarchy (that''s easy with merging hashes) as well as considering what
Facts or Classes applies to a node (machine logic). That''s not trivial
to do, especially with a potentially large set of data like sysctl.conf keys.
>
> Does anyone have any thoughts or tips on how they might be managing a
similar situation?
>
> Thanks,
>
> -Luke
>
> --
> Luke Bigum
>
> Information Systems
> Ph: +44 (0) 20 3192 2520
> luke.bigum@lmax.com<mailto:luke.bigum@lmax.com> |
http://www.lmax.com<http://www.lmax.com/>
> LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN
>
>
> FX and CFDs are leveraged products that can result in losses exceeding
> your deposit. They are not suitable for everyone so please ensure you
> fully understand the risks involved. The information in this email is not
> directed at residents of the United States of America or any other
> jurisdiction where trading in CFDs and/or FX is restricted or prohibited
> by local laws or regulations.
>
> The information in this email and any attachment is confidential and is
> intended only for the named recipient(s). The email may not be disclosed
> or used by any person other than the addressee, nor may it be copied in
> any way. If you are not the intended recipient please notify the sender
> immediately and delete any copies of this message. Any unauthorised
> copying, disclosure or distribution of the material in this e-mail is
> strictly forbidden.
>
> LMAX operates a multilateral trading facility. Authorised and regulated by
the Financial Services Authority (firm registration number 509778) and
> is registered in England and Wales (number 06505809). Our registered
address is Yellow Building, 1A Nicholas Road, London, W11
> 4AN.
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to
puppet-users@googlegroups.com<mailto:puppet-users@googlegroups.com>.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com<mailto:puppet-users%2Bunsubscribe@googlegroups.com>.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
>
>
>
> --
>
> Gary Larizza
> Professional Services Engineer
> Puppet Labs
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to
puppet-users@googlegroups.com<mailto:puppet-users@googlegroups.com>.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com<mailto:puppet-users+unsubscribe@googlegroups.com>.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
>
>
> --
> Luke Bigum
>
> Information Systems
> Ph: +44 (0) 20 3192 2520
> luke.bigum@lmax.com<mailto:luke.bigum@lmax.com> |
http://www.lmax.com<http://www.lmax.com/>
> LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN
>
> FX and CFDs are leveraged products that can result in losses exceeding
> your deposit. They are not suitable for everyone so please ensure you
> fully understand the risks involved. The information in this email is not
> directed at residents of the United States of America or any other
> jurisdiction where trading in CFDs and/or FX is restricted or prohibited
> by local laws or regulations.
>
> The information in this email and any attachment is confidential and is
> intended only for the named recipient(s). The email may not be disclosed
> or used by any person other than the addressee, nor may it be copied in
> any way. If you are not the intended recipient please notify the sender
> immediately and delete any copies of this message. Any unauthorised
> copying, disclosure or distribution of the material in this e-mail is
> strictly forbidden.
>
> LMAX operates a multilateral trading facility. Authorised and regulated
> by the Financial Services Authority (firm registration number 509778) and
> is registered in England and Wales (number 06505809).
> Our registered address is Yellow Building, 1A Nicholas Road, London, W11
> 4AN.
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to
puppet-users@googlegroups.com<mailto:puppet-users@googlegroups.com>.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com<mailto:puppet-users%2Bunsubscribe@googlegroups.com>.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
>
>
> --
>
> Gary Larizza
> Professional Services Engineer
> Puppet Labs
>
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/0NSu9kWGZggJ.
> To post to this group, send email to
puppet-users@googlegroups.com<mailto:puppet-users@googlegroups.com>.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com<mailto:puppet-users+unsubscribe@googlegroups.com>.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
>
> ________________________________
>
> This message may contain confidential or privileged information. If you are
not the intended recipient, please advise us immediately and delete this
message. See http://www.datapipe.com/legal/email_disclaimer/ for further
information on confidentiality and the risks of non-secure electronic
communication. If you cannot access these links, please notify us by reply
message and we will send the contents to you.
>
--
Luke Bigum
Information Systems
Ph: +44 (0) 20 3192 2520
luke.bigum@lmax.com | http://www.lmax.com
LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN
FX and CFDs are leveraged products that can result in losses exceeding
your deposit. They are not suitable for everyone so please ensure you
fully understand the risks involved. The information in this email is not
directed at residents of the United States of America or any other
jurisdiction where trading in CFDs and/or FX is restricted or prohibited
by local laws or regulations.
The information in this email and any attachment is confidential and is
intended only for the named recipient(s). The email may not be disclosed
or used by any person other than the addressee, nor may it be copied in
any way. If you are not the intended recipient please notify the sender
immediately and delete any copies of this message. Any unauthorised
copying, disclosure or distribution of the material in this e-mail is
strictly forbidden.
LMAX operates a multilateral trading facility. Authorised and regulated
by the Financial Services Authority (firm registration number 509778) and
is registered in England and Wales (number 06505809).
Our registered address is Yellow Building, 1A Nicholas Road, London, W11
4AN.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.