So, I was following the thread "how to conditionally add users to a virtualized group?" and had a bit of a realization that I''m not quite sure why Hiera is a better backend than LDAP. Hiera: - Stores hierarchical data locally on your system - Uses YAML - Integrates with puppet LDAP - Stores hierarchical data across potentially multiple systems (think puppet master scaling and data sync) - Uses LDIFs - Needs some glue code written However, both are hierarchical databases based on ''read often/write rarely'' principals. Besides the glue code to make LDAP do Hiera-like things, what are the issues? It also seems that using a well known and supported system, such as LDAP, would foster greater enterprise support (except in those places where you have to spawn your own due to insane directory admins). And, yes, I know that a hiera back-end could be written to support LDAP but that would just be an unnecessary data transference if I''m reading it right. If you wanted local "fast" copies of the data on all of your puppet masters (and you do) then a simple LDAP slave would be spawned on each master. Thanks, Trevor -- Trevor Vaughan Vice President, Onyx Point, Inc (410) 541-6699 tvaughan@onyxpoint.com -- This account not approved for unencrypted proprietary information -- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
(inline, verbosely rhubarbing for the audience not the poster) On Wed, Jul 18, 2012 at 05:09:42PM -0400, Trevor Vaughan wrote:> So, I was following the thread "how to conditionally add users to a > virtualized group?" and had a bit of a realization that I''m not quite > sure why Hiera is a better backend than LDAP. > > Hiera: > > - Stores hierarchical data locally on your system > - Uses YAMLYAML is... something that nearly everything can read/write. People who don''t often write YAML (like me) write our YAML in our favourite scripting languages and then export via one of the many export modules. YAML is a simple text file, or four.> - Integrates with puppet > > LDAP > > - Stores hierarchical data across potentially multiple systems (think > puppet master scaling and data sync) > - Uses LDIFs > - Needs some glue code writtens/some/much/ The skills to deal with LDAP are much less commonplace than the skills to deal with YAML. YAML is much more standardized than all the different LDAP implementations out there (I have used several).> However, both are hierarchical databases based on ''read often/write > rarely'' principals. > > Besides the glue code to make LDAP do Hiera-like things, what are the > issues? It also seems that using a well known and supported system, > such as LDAP, would foster greater enterprise support (except in those > places where you have to spawn your own due to insane directory > admins).LDAP is quite verbose. Compare: operations: bind: ssh HOST "hostname;ps -ef | grep named" date: ssh HOST "echo \`hostname\` \`date\`" df: ssh HOST "df -h -l" Versus pseudo-ldif: dn: dc=operations,dc=c,dc=scripts,dc=mycompany,dc=com dc: operations objectclass: yamlthing item:: YmluZDogc3NoIEhPU1QgImhvc3RuYW1lO3BzIC1lZiB8IGdyZXAgbmFtZWQi item:: ZGF0ZTogc3NoIEhPU1QgImVjaG8gXGBob3N0bmFtZVxgIFxgZGF0ZVxgIg=item:: PW4gZGY6IHNzaCBIT1NUICJkZiAtaCAtbCIK One big difference between LDAP and YAML: You can have ordered attributes (lists) in YAML. There is no guaranteed attribute ordering in LDAP; attributes may be returned in any order, on an implementation-specific basis. (To the best of my knowledge.) Depending on your access method (and likely implementation details), you may have to implement client-side logic to get your heirarchical values working just right. For instance, which order will the following entries be returned in? dc=three dc=two,dc=three dc=two,dc=two,dc=three dc=one,dc=two,dc=three It might be the above order, as I''ve seen from OpenLDAP. Then the entries'' attributes will be returned in any order. Happy parsing. LDAP also requires another client/server infrastructure, more daemons to patch and maintain, et cetera.> And, yes, I know that a hiera back-end could be written to support > LDAP but that would just be an unnecessary data transference if I''m > reading it right. > > If you wanted local "fast" copies of the data on all of your puppet > masters (and you do) then a simple LDAP slave would be spawned on each > master.Replicating between ldap instances isn''t simple to set up. (Are you using single-master, hub-spoke, multimaster, etc.?) Some light reading: http://www.openldap.org/doc/admin24/replication.html http://directory.fedoraproject.org/wiki/Howto:MultiMasterReplication Of course, I don''t know the point at which the ldap advantages outweigh the setup costs, or how these compare to YAML.> Thanks, > > Trevor > > -- > Trevor Vaughan > Vice President, Onyx Point, Inc > (410) 541-6699 > tvaughan@onyxpoint.com > > -- This account not approved for unencrypted proprietary information -- > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Wed, Jul 18, 2012 at 2:09 PM, Trevor Vaughan <tvaughan@onyxpoint.com> wrote:> So, I was following the thread "how to conditionally add users to a > virtualized group?" and had a bit of a realization that I''m not quite > sure why Hiera is a better backend than LDAP. >In our environment at least, messing around with the LDAP schema is a non-starter. I can change my Hiera setup any time. That alone makes it better for me. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On 07/18/2012 05:42 PM, Aaron Grewell wrote:> On Wed, Jul 18, 2012 at 2:09 PM, Trevor Vaughan <tvaughan@onyxpoint.com> wrote: >> So, I was following the thread "how to conditionally add users to a >> virtualized group?" and had a bit of a realization that I''m not quite >> sure why Hiera is a better backend than LDAP. >> > In our environment at least, messing around with the LDAP schema is a > non-starter. I can change my Hiera setup any time. That alone makes > it better for me. >Along that line, it would be more flexible to write an LDAP backend for hiera. That way, you could use company-wide attributes and overwirte them or augment them in hiera, as needed. Jason -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Thanks for the post, it was quite interesting. For the attributes, I''ve seen some reasonable text about storing serialized JSON in LDAP attributes which should work quite well for preserving ordering, etc... In terms of infrastructure, if you''ve got any significant number of users, you''re probably already using LDAP, whether or not it''s bound with Kerberos so this shouldn''t be something that is rocket surgery for an enterprise scenario. Yes, I *completely* agree that for smaller infrastructures, Hiera makes a lot of sense but I''m starting to feel people pushing at the edges in such a way that we''re going to start reinventing the wheel in terms of needing multi-master replication, synchronization, localized offline caching, etc.... LDAP is certainly verbose, but there are enough libraries to be able to wrap it in pretty much any data structure that you like so that shouldn''t be too bad overall. Interesting conversation. Thanks, Trevor On Wed, Jul 18, 2012 at 5:39 PM, Christopher Wood <christopher_wood@pobox.com> wrote:> (inline, verbosely rhubarbing for the audience not the poster) > > On Wed, Jul 18, 2012 at 05:09:42PM -0400, Trevor Vaughan wrote: >> So, I was following the thread "how to conditionally add users to a >> virtualized group?" and had a bit of a realization that I''m not quite >> sure why Hiera is a better backend than LDAP. >> >> Hiera: >> >> - Stores hierarchical data locally on your system >> - Uses YAML > > YAML is... something that nearly everything can read/write. People who don''t often write YAML (like me) write our YAML in our favourite scripting languages and then export via one of the many export modules. YAML is a simple text file, or four. > >> - Integrates with puppet >> >> LDAP >> >> - Stores hierarchical data across potentially multiple systems (think >> puppet master scaling and data sync) >> - Uses LDIFs >> - Needs some glue code written > > s/some/much/ > > The skills to deal with LDAP are much less commonplace than the skills to deal with YAML. YAML is much more standardized than all the different LDAP implementations out there (I have used several). > >> However, both are hierarchical databases based on ''read often/write >> rarely'' principals. >> >> Besides the glue code to make LDAP do Hiera-like things, what are the >> issues? It also seems that using a well known and supported system, >> such as LDAP, would foster greater enterprise support (except in those >> places where you have to spawn your own due to insane directory >> admins). > > LDAP is quite verbose. Compare: > > operations: > bind: ssh HOST "hostname;ps -ef | grep named" > date: ssh HOST "echo \`hostname\` \`date\`" > df: ssh HOST "df -h -l" > > Versus pseudo-ldif: > > dn: dc=operations,dc=c,dc=scripts,dc=mycompany,dc=com > dc: operations > objectclass: yamlthing > item:: YmluZDogc3NoIEhPU1QgImhvc3RuYW1lO3BzIC1lZiB8IGdyZXAgbmFtZWQi > item:: ZGF0ZTogc3NoIEhPU1QgImVjaG8gXGBob3N0bmFtZVxgIFxgZGF0ZVxgIg=> item:: PW4gZGY6IHNzaCBIT1NUICJkZiAtaCAtbCIK > > One big difference between LDAP and YAML: You can have ordered attributes (lists) in YAML. There is no guaranteed attribute ordering in LDAP; attributes may be returned in any order, on an implementation-specific basis. (To the best of my knowledge.) > > Depending on your access method (and likely implementation details), you may have to implement client-side logic to get your heirarchical values working just right. For instance, which order will the following entries be returned in? > > dc=three > dc=two,dc=three > dc=two,dc=two,dc=three > dc=one,dc=two,dc=three > > It might be the above order, as I''ve seen from OpenLDAP. Then the entries'' attributes will be returned in any order. Happy parsing. > > LDAP also requires another client/server infrastructure, more daemons to patch and maintain, et cetera. > >> And, yes, I know that a hiera back-end could be written to support >> LDAP but that would just be an unnecessary data transference if I''m >> reading it right. >> >> If you wanted local "fast" copies of the data on all of your puppet >> masters (and you do) then a simple LDAP slave would be spawned on each >> master. > > Replicating between ldap instances isn''t simple to set up. (Are you using single-master, hub-spoke, multimaster, etc.?) Some light reading: > > http://www.openldap.org/doc/admin24/replication.html > > http://directory.fedoraproject.org/wiki/Howto:MultiMasterReplication > > Of course, I don''t know the point at which the ldap advantages outweigh the setup costs, or how these compare to YAML. > >> Thanks, >> >> Trevor >> >> -- >> Trevor Vaughan >> Vice President, Onyx Point, Inc >> (410) 541-6699 >> tvaughan@onyxpoint.com >> >> -- This account not approved for unencrypted proprietary information -- >> >> -- >> You received this message because you are subscribed to the Google Groups "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >> >> > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >-- Trevor Vaughan Vice President, Onyx Point, Inc (410) 541-6699 tvaughan@onyxpoint.com -- This account not approved for unencrypted proprietary information -- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Wednesday, July 18, 2012 4:09:42 PM UTC-5, Trevor Vaughan wrote:> > So, I was following the thread "how to conditionally add users to a > virtualized group?" and had a bit of a realization that I''m not quite > sure why Hiera is a better backend than LDAP. > > Hiera: > > - Stores hierarchical data locally on your system > - Uses YAML > - Integrates with puppet >That''s not actually a very good description of Hiera. It describes the combination of the built-in data backend shipped with hiera, whose use is optional, with the Puppet integration layer, which is packaged separately. None of it describes the hiera core, which I would describe as a framework for selecting and combining data from various sources based on a user-defined priority hierarchy.> > LDAP > > - Stores hierarchical data across potentially multiple systems (think > puppet master scaling and data sync) > - Uses LDIFs > - Needs some glue code written > > However, both are hierarchical databases based on ''read often/write > rarely'' principals. >No, I don''t think they''re actually very comparable. The Hiera core is an hierarchically-oriented data access framework, compatible (in principle) with a wide variety of databases. The two are not on the same level. Indeed, as you acknowledged, it would be feasible to write a hiera back-end that accessed data from an LDAP directory. That would be comparable to hiera''s YAML back-end, but not to the hiera core.> > Besides the glue code to make LDAP do Hiera-like things, what are the > issues? It also seems that using a well known and supported system, > such as LDAP, would foster greater enterprise support (except in those > places where you have to spawn your own due to insane directory > admins). >I''m not saying that LDAP wouldn''t be a good data repository for Puppet. You''re quite right that it is well understood and widely deployed. I am certain that there are LDAP backends out there for extlookup(), and perhaps also for Hiera. There are probably also be custom data access functions for LDAP in some shops. I don''t think there are any fundamental issues blocking its use with Puppet. On the other hand, LDAP is far too heavyweight to be viable as the singular primary data source for Puppet, and Puppet needs such a thing to make an ecosystem of modules viable.> > And, yes, I know that a hiera back-end could be written to support > LDAP but that would just be an unnecessary data transference if I''m > reading it right. >I''m not sure what you mean by that, but the advantages of accessing LDAP via hiera include: 1. Abstraction. Module authors don''t need to care where data comes from. 2. Pluggability. The LDAP back-end could be swapped out for a different one. That''s useful for testing, or for non-LDAP shops to use the same modules as LDAP shops. 3. Combining and overriding data. Data packaged with modules or added / overridden by admins don''t need to be recorded in LDAP. Depending on how you do your accounting, there may not even be any more layers in an hiera-based stack. Any way around, getting the data into puppet means going through a Puppet custom function that wraps an LDAP client that gets data from your directory. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/uzrZymh0MHQJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
----- Original Message -----> From: "jcbollinger" <John.Bollinger@stJude.org> > To: puppet-users@googlegroups.com > Sent: Thursday, July 19, 2012 2:06:43 PM > Subject: [Puppet Users] Re: Hiera vs LDAP > > >> And, yes, I know that a hiera back-end could be written to support >> LDAP but that would just be an unnecessary data transference if I''m >> reading it right. > > > I''m not sure what you mean by that, but the advantages of accessing > LDAP via hiera include: > > 1. Abstraction. Module authors don''t need to care where data > comes from. > 2. Pluggability. The LDAP back-end could be swapped out for a > different one. That''s useful for testing, or for non-LDAP shops > to use the same modules as LDAP shops. > 3. Combining and overriding data. Data packaged with modules or > added / overridden by admins don''t need to be recorded in LDAP. > > Depending on how you do your accounting, there may not even be any > more layers in an hiera-based stack. Any way around, getting the > data into puppet means going through a Puppet custom function that > wraps an LDAP client that gets data from your directory.And as of puppet 3 you will get built in integration of hiera into parameterized classes - without the need for calling any custom functions or assigning local variables. This means all parameterized class written to date will automatically become hiera enabled. And by extension become LDAP enabled if you just wrote an LDAP backend for hiera. The cost of an additional abstraction really just doesnt feature when considering this and Johns excellent points. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
All, Thanks for the responses, I''m certainly seeing the benefits of Hiera over a direct LDAP interface but I do think that an LDAP interface and schema should be on the roadmap. Also, you might want to summarize some of this and add it to the Hiera Wiki. Thanks! Trevor On Thu, Jul 19, 2012 at 9:12 AM, R.I.Pienaar <rip@devco.net> wrote:> > > ----- Original Message ----- >> From: "jcbollinger" <John.Bollinger@stJude.org> >> To: puppet-users@googlegroups.com >> Sent: Thursday, July 19, 2012 2:06:43 PM >> Subject: [Puppet Users] Re: Hiera vs LDAP >> >> >>> And, yes, I know that a hiera back-end could be written to support >>> LDAP but that would just be an unnecessary data transference if I''m >>> reading it right. >> >> >> I''m not sure what you mean by that, but the advantages of accessing >> LDAP via hiera include: >> >> 1. Abstraction. Module authors don''t need to care where data >> comes from. >> 2. Pluggability. The LDAP back-end could be swapped out for a >> different one. That''s useful for testing, or for non-LDAP shops >> to use the same modules as LDAP shops. >> 3. Combining and overriding data. Data packaged with modules or >> added / overridden by admins don''t need to be recorded in LDAP. >> >> Depending on how you do your accounting, there may not even be any >> more layers in an hiera-based stack. Any way around, getting the >> data into puppet means going through a Puppet custom function that >> wraps an LDAP client that gets data from your directory. > > And as of puppet 3 you will get built in integration of hiera into > parameterized classes - without the need for calling any custom > functions or assigning local variables. This means all parameterized > class written to date will automatically become hiera enabled. > > And by extension become LDAP enabled if you just wrote an LDAP backend > for hiera. The cost of an additional abstraction really just doesnt > feature when considering this and Johns excellent points. > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >-- Trevor Vaughan Vice President, Onyx Point, Inc (410) 541-6699 tvaughan@onyxpoint.com -- This account not approved for unencrypted proprietary information -- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.