Puppet users - Jul 2009 - Generating a file from a set of fragments on the puppetmaster

Hi,

I''m trying to create a squid url_regex ACL source file for various
different sites.  Each site needs a slightly different configuration, so
my plan was to create the a bunch of files on the server, then drag them
down and concatenate them into a single file on the client.

I found http://reductivelabs.com/trac/puppet/wiki/CompleteConfiguration
and i''ve been trying to understand its approach to constructing a file
out of fragments.  Am i right in thinking that i need to distribute an
entire directory from the server, then use concatenated_file to combine
those files into one file on the puppet agent?

I''d rather not distribute the entire directory from the server, since
it
contains custom content for each node.  Is there a way i can do this
with templates that include other files?  (Or templates that are plain
text rather than .erb?)  I''d really like to find a technique that
doesn''t require separately copying the file fragments to the client
also...

I''ve had 3 or 4 tries at getting the right approach and am still no
closer to a working solution.  Attached is my non-working attempt at a
class to do this - what am i doing wrong?  (Comments about what''s not
working can be found after the @@@ comments.)  I thought this was a
fairly simple problem, but i''ve been banging my head against it all day
without any success.

Paul


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Paul Gear wrote:
> Hi,
> 
> I''m trying to create a squid url_regex ACL source file for various
> different sites.  Each site needs a slightly different configuration, so
> my plan was to create the a bunch of files on the server, then drag them
> down and concatenate them into a single file on the client.
> 
> I found http://reductivelabs.com/trac/puppet/wiki/CompleteConfiguration
> and i''ve been trying to understand its approach to constructing a
file
> out of fragments.  Am i right in thinking that i need to distribute an
> entire directory from the server, then use concatenated_file to combine
> those files into one file on the puppet agent?
> 
> I''d rather not distribute the entire directory from the server,
since it
> contains custom content for each node.  Is there a way i can do this
> with templates that include other files?  (Or templates that are plain
> text rather than .erb?)  I''d really like to find a technique that
> doesn''t require separately copying the file fragments to the
client also...
> 
> I''ve had 3 or 4 tries at getting the right approach and am still
no
> closer to a working solution.  Attached is my non-working attempt at a
> class to do this - what am i doing wrong?  (Comments about what''s
not
> working can be found after the @@@ comments.)  I thought this was a
> fairly simple problem, but i''ve been banging my head against it
all day
> without any success.
My approach in the CompleteConfiguration repos is to use the 
concatenated_file_part define to put file parts onto the client and then 
locally concatenated them all on the client with the concatenated_file 
define which does some shell magic internally.

You seem to try to use a server side function ( file() ) to create a 
similar effect. You just have to be  aware that functions always run on 
the server.


Regards, DavidS

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

David Schmitt wrote:
> ...
>> I''d rather not distribute the entire directory from the
server, since it
>> contains custom content for each node.  Is there a way i can do this
>> with templates that include other files?  (Or templates that are plain
>> text rather than .erb?)  I''d really like to find a technique
that
>> doesn''t require separately copying the file fragments to the
client also...
>>
>> ...
> 
> My approach in the CompleteConfiguration repos is to use the 
> concatenated_file_part define to put file parts onto the client and then 
> locally concatenated them all on the client with the concatenated_file 
> define which does some shell magic internally.
> 
> You seem to try to use a server side function ( file() ) to create a 
> similar effect. You just have to be  aware that functions always run on 
> the server.
I knew i must be overcomplicating it somehow.  :-)  So if i give the
file() function the full pathname of the directory on the server, i
should be able to achieve what i''m wanting without needing to
distribute
a local copy of the files?

Are there any drawbacks of doing this?  (Such as having to pull down the
content every time to check its MD5 sum, perhaps?)

Paul


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Paul Gear wrote:
> David Schmitt wrote:
>> ...
>>> I''d rather not distribute the entire directory from the
server, since it
>>> contains custom content for each node.  Is there a way i can do
this
>>> with templates that include other files?  (Or templates that are
plain
>>> text rather than .erb?)  I''d really like to find a
technique that
>>> doesn''t require separately copying the file fragments to
the client also...
>>>
>>> ...
>> My approach in the CompleteConfiguration repos is to use the 
>> concatenated_file_part define to put file parts onto the client and
then
>> locally concatenated them all on the client with the concatenated_file 
>> define which does some shell magic internally.
>>
>> You seem to try to use a server side function ( file() ) to create a 
>> similar effect. You just have to be  aware that functions always run on
>> the server.
> 
> I knew i must be overcomplicating it somehow.  :-)  So if i give the
> file() function the full pathname of the directory on the server, i
> should be able to achieve what i''m wanting without needing to
distribute
> a local copy of the files?
 From http://reductivelabs.com/trac/puppet/wiki/FunctionReference#file
:
> Return the contents of a file. Multiple files can be passed, and the first
file that exists will be read in.
You probably should use template() (which concatenates its parameters; 
be aware of ERB expansion though) or create a local concat_file() 
function that works as intended or use generate() to call a server-side 
script.
> Are there any drawbacks of doing this?  (Such as having to pull down the
> content every time to check its MD5 sum, perhaps?)
Exactly. content=> is compiled into the configuration which is sent to 
the client.

Also, it is hard to add to these files from other clients, like with 
export/collect: you need a component on the puppetmaster and a third 
puppetd run.


Regards, DavidS

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

My understanding is that there are a series of files that exist on the
server. Each contains information that is needed by various nodes. All
nodes need some of the files; but not all nodes need all of the files.

Based on what I think you are trying to get done, I would probably use
generate...

For example:

class squid {
     # Generate should return the results of the command...
     $file_content = generate
("cat","header","$hostname.file1","$hostname.file2","isp.
$isp","footer")

     file { "/path/to/squid_file":
          content => $file_content,
     }
}

You could set the ISP variable per node, or maybe create a new fact in
facter; depends on your setup really. To keep from constantly pushing
the configuration, you could look at an initialization file which can
be done a couple ways. The easiest would be to push a file that has
the date in it or something of that nature. Subscribe to that file so
that whenever it is updated, the squid file is also updated. So when
you make changes to the config files, you just update the date in your
init file.

Anyway, just a thought...


On Jul 3, 3:54 am, David Schmitt <da...@dasz.at>
wrote:
> Paul Gear wrote:
> > David Schmitt wrote:
> >> ...
> >>> I''d rather not distribute the entire directory from
the server, since it
> >>> contains custom content for each node.  Is there a way i can
do this
> >>> with templates that include other files?  (Or templates that
are plain
> >>> text rather than .erb?)  I''d really like to find a
technique that
> >>> doesn''t require separately copying the file fragments
to the client also...
>
> >>> ...
> >> My approach in the CompleteConfiguration repos is to use the
> >> concatenated_file_part define to put file parts onto the client
and then
> >> locally concatenated them all on the client with the
concatenated_file
> >> define which does some shell magic internally.
>
> >> You seem to try to use a server side function ( file() ) to create
a
> >> similar effect. You just have to be  aware that functions always
run on
> >> the server.
>
> > I knew i must be overcomplicating it somehow.  :-)  So if i give the
> > file() function the full pathname of the directory on the server, i
> > should be able to achieve what i''m wanting without needing to
distribute
> > a local copy of the files?
>
>  Fromhttp://reductivelabs.com/trac/puppet/wiki/FunctionReference#file:
>
> > Return the contents of a file. Multiple files can be passed, and the
first file that exists will be read in.
>
> You probably should use template() (which concatenates its parameters;
> be aware of ERB expansion though) or create a local concat_file()
> function that works as intended or use generate() to call a server-side
> script.
>
> > Are there any drawbacks of doing this?  (Such as having to pull down
the
> > content every time to check its MD5 sum, perhaps?)
>
> Exactly. content=> is compiled into the configuration which is sent to
> the client.
>
> Also, it is hard to add to these files from other clients, like with
> export/collect: you need a component on the puppetmaster and a third
> puppetd run.
>
> Regards, DavidS- Hide quoted text -
>
> - Show quoted text -
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Paul Gear wrote:
> Hi,
> 
> I''m trying to create a squid url_regex ACL source file for various
> different sites.  Each site needs a slightly different configuration, so
> my plan was to create the a bunch of files on the server, then drag them
> down and concatenate them into a single file on the client.
> 
> I found http://reductivelabs.com/trac/puppet/wiki/CompleteConfiguration
> and i''ve been trying to understand its approach to constructing a
file
> out of fragments.  Am i right in thinking that i need to distribute an
> entire directory from the server, then use concatenated_file to combine
> those files into one file on the puppet agent?
> 
> I''d rather not distribute the entire directory from the server,
since it
> contains custom content for each node.  Is there a way i can do this
> with templates that include other files?  (Or templates that are plain
> text rather than .erb?)  I''d really like to find a technique that
> doesn''t require separately copying the file fragments to the
client also...
> ...
Thanks for the suggestions on this.  I ended up going with a templated
solution that uses a bit of Ruby to intelligently grab files in on the
server.  I''ve documented my efforts here:

http://reductivelabs.com/trac/puppet/wiki/Recipes/SquidConfiguration

Any suggestions for improvement gratefully accepted.

Paul


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Paul Gear wrote:
> ...
> 
> Thanks for the suggestions on this.  I ended up going with a templated
> solution that uses a bit of Ruby to intelligently grab files in on the
> server.  I''ve documented my efforts here:
> 
> http://reductivelabs.com/trac/puppet/wiki/Recipes/SquidConfiguration
> 
> Any suggestions for improvement gratefully accepted.
Hi folks,

Any feedback on whether i could do this better?  I''d definitely like to
get that check_file method out of the top of every file and put it in a
common place.

Regards,
Paul


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Puppeteers,

It''s not just Paul who''d like an answer, if anyone has one to
proffer.
He''s been kind enough to document his work on the wiki.  If
it''s not
exemplary of best practice, then do us all a favor and provide a better
solution.

Thanks,

Peter 
> -----Original Message-----
> From: puppet-users@googlegroups.com 
> [mailto:puppet-users@googlegroups.com] On Behalf Of Paul Gear
> Sent: Wednesday, July 15, 2009 1:29 AM
> To: puppet-users@googlegroups.com
> Subject: [Puppet Users] Re: Generating a file from a set of 
> fragments on the puppetmaster
> 
> 
> Paul Gear wrote:
> > ...
> > 
> > Thanks for the suggestions on this.  I ended up going with 
> a templated 
> > solution that uses a bit of Ruby to intelligently grab 
> files in on the 
> > server.  I''ve documented my efforts here:
> > 
> > http://reductivelabs.com/trac/puppet/wiki/Recipes/SquidConfiguration
> > 
> > Any suggestions for improvement gratefully accepted.
> 
> Hi folks,
> 
> Any feedback on whether i could do this better?  I''d 
> definitely like to get that check_file method out of the top 
> of every file and put it in a common place.
> 
> Regards,
> Paul
> 
> 
> > 
> 
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---