[Puppet Users] Re: Storing puppet info in a database

S H
Fri, 04 Sep 2009 17:31:29 -0000

On Fri, Sep 4, 2009 at 3:44 AM, Kenneth Holter
<kenneho.ndu@gmail.com>wrote:

>
> Thanks for all the replies - this looks very promising wrt gaining more
> control over our node definitions (and inventory). It seems like having the
> puppetmaster fetch client info from a database is a far better approach
than
> having the puppetmaster push info into one.
>
> If I understand correctly, stored configs simply store the puppet
client''s
> configuration in a database, while external nodes classification allows for
> generating configurations based on contents in for example a database.
> In our case, we have a Satellite server that store information in an Oracle
> database, so incoorporating puppet info into that database sound very
> elegant. This way we''ll have one database that contains pretty
much
> everything.
>
> As far as I can see, two questions still remain:
>
>    1. What''s the most practical way of updating external
node''s database,
>    i.e. how to I create new nodes? I guess having a script that queries us
for
>    misc info about new nodes, and updates the database accordingly, seems
like
>    the most neat way of doing this. How are others doing this? Does such a
>    script allready exist?
>    2. Can I have the puppet clients store info (for example hardware info
>    and such) in the same database? Perhaps this is where exported resources
>    comes in handy.
>
> Your external nodes database can really contain anything you want - Puppet
calls a script and expects a YAML response with classes and parameters
defined. What happens in between is none of Puppet''s business.

For example, I use Active Directory as the backend for my external nodes.
This works well for me because all of my hosts need to be in my ADS
configuration anyhow, and this allows me to manage their Puppet classes with
ADS groups. A detailed description of my implementation is at
http://projects.reductivelabs.com/issues/2381.

One last note on #2 - exported resources are mostly a way of saying
"Compile
this resource on node X, but actually apply it on node Y."
(there''s a lot
more to it, but that''s the main use case I''ve found). They
aren''t really
appropriate for long-term information storage.

-Shawn


>
>    1.
>
> By the way, what are the arguments to choose stored configs over external
> nodes classification?
>
> Best regards,
> Kenneth
>
>
> On 9/2/09, Joel Krauska <jkrauska@gmail.com> wrote:
>>
>>
>> Puppet already stores a lot of this data in YAML.
>>
>> Take a look at
>> /var/lib/puppet/yaml/facts/
>>
>> Here''s a quick and dirty perl tool.
>> (it needs the YAML perl module to work)
>>
>> #!/usr/bin/perl
>> use YAML;
>>
>> $facts =''/var/lib/puppet/yaml/facts/*.yaml'';
>>
>> foreach my $file ( glob $facts ) {
>>    my $data = YAML::LoadFile($file);
>>    my $values = $data->{values};
>>    my $name = $data->{name};
>>    $kernellist{$values->{kernelrelease}}++;
>>    $memlist{$values->{memorysize}}++;
>> }
>>
>> print "---------------------------------------------\n";
>> print "Kernel Counts\n";
>> print "---------------------------------------------\n";
>> foreach my $item (sort keys %kernellist) {
>>    print "$kernellist{$item} \t $item\n";
>> }
>>
>>
>> print "\n\n";
>> print "---------------------------------------------\n";
>> print "RAM Counts\n";
>> print "---------------------------------------------\n";
>> foreach my $item (sort keys %memlist) {
>>
>>    print "$memlist{$item} \t $item\n";
>> }
>>
>>
>>
>>
>> example output:
>> ---------------------------------------------
>> Kernel Counts
>> ---------------------------------------------
>> 1        2.6.16.27-0.9-smp
>> 37       2.6.16.53-0.16-smp
>> 9        2.6.16.53-0.8-smp
>> 12       2.6.16.54-0.2.5-smp
>> 3        2.6.20-17-server
>> 8        2.6.24-16-server
>> 12       2.6.24-19-server
>>
>> etc...
>>
>> You can create custom facts to grab the type of information you want
>> and puppet will store that in YAML too.
>>
>> Happy puppeting.
>>
>> --joel
>>
>>
>>
>> On Tue, Sep 1, 2009 at 6:28 PM, Ohad Levy<ohadlevy@gmail.com>
wrote:
>> > and, I forgot, you are not forced to use store configs just for
the
>> > inventory.
>> >
>> > On Wed, Sep 2, 2009 at 9:27 AM, Ohad Levy
<ohadlevy@gmail.com> wrote:
>> >>
>> >> I also recommend using external nodes and facts as inventory.
>> >>
>> >> if you are looking for an active project (unlike puppetshow
and
>> iclassiy)
>> >> you might consider gni.
>> >> it already does what you want, and more :)
>> >> http://wiki.github.com/ohadlevy/gni
>> >>
>> >> cheers,
>> >> Ohad
>> >>
>> >> On Wed, Sep 2, 2009 at 1:00 AM, Kenneth Holter
<kenneho.ndu@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi all.
>> >>> We have a bunch of RHEL servers running Puppet. They are
also
>> connected
>> >>> to our Red Hat Satellite server.
>> >>> Currently we don''t have any master documentation
system that stores
>> all
>> >>> relevant information (i.e. type of server, hardware info,
linux
>> >>> configuration, etc) about the servers. So what
I''d like to do is
>> implement
>> >>> some sort of system that can hold all this information.
There are a
>> lot of
>> >>> asset management products that I guess could get us a long
way (and
>> our
>> >>> organization is probably going to buy one soon), but
I''m not sure how
>> >>> flexible these would be with regards to storing custom
information and
>> such.
>> >>> The asset management product will be administred by
someone else
>> within the
>> >>> organisation, and having that person implement linux
specific info
>> (such as
>> >>> puppet) and stuff may not be very realistic.
>> >>> First of all, how to other people handle this sort of
thing? Storing
>> info
>> >>> in excel-files is no good, since the information will be
outdated by
>> the
>> >>> time I press "save". We need a more or less
automatic way of doing
>> this, but
>> >>> is asset management products the way to go? Currently, our
node
>> definition
>> >>> in Puppet holds the most current infomation when it comes
to
>> confuguration,
>> >>> while Satellite server holds a lot of other information.
I''m hoping to
>> >>> integrate all our server info (including puppet) into a
database
>> (preferably
>> >>> the Satellite server database on Oracle), so that we can
query it for
>> info.
>> >>> I''ve been playing around with two ideas for
automating system
>> >>> documentation:
>> >>> 1) Parse the puppet manifests (and node definitions) and
organize the
>> >>> info in a database
>> >>> 2) Include a logging functionality in the puppet
manifests, so that
>> when
>> >>> the clients implement the configuration it logs a specific
text to
>> syslog.
>> >>> The syslog message is sent over to our log host, which can
parse it
>> and
>> >>> store the information in a database.
>> >>> Maybe the first option is allready implemented? Anyway,
have anyone
>> >>> implemented such mechanisms?
>> >>> Best regards,
>> >>> Kenneth Holter
>> >>>
>> >>
>> >
>> >
>> > >
>> >
>> >>
>>