Puppet users - Mar 2010 - How to efficiently manage multiple packages installing in the same directory

Hi everyone.  I am trying to manage several different packages using
Puppet.  For the sake of maintainability, each package is installed in
its own separate puppet directory.  Each package would have a separate
install_package.pp script and corresponding folder in /etc/puppet/
files on the server:

    /etc/puppet/files/foo/
                         etc/
                         root/
                         usr/local

   /etc/puppet/files/bar/
                         etc/
                         var/lib

To avoid micromanaging, I would like to simply copy over the contents
of each folder to / on the client.  While each package has directories
in common (/etc for example), all of the files in each package are
orthogonal and not shared between packages.  For example, package foo
will be the only package that will ever install /etc/foo.conf.
Package bar will never manage it.

I have tried to do this using the Puppet config below, but it errors
out with the message below.   Can any one tell me a simple solution
that does not involve micromanaging each unique file or subfolder?
Given the dynamic nature of the packages (new files are added/removed
constantly), this would add a good deal of unnecessary work.

"err: Could not retrieve catalog: Puppet::Parser::AST::Resource failed
with error ArgumentError: Duplicate definition: File[/] is already
defined in file install_foo.pp at line 8; cannot redefine at
install_ibp.pp:8 on node mybox.local"

---

site.pp

    import "classes/install_foo.pp"
    import "classes/install_bar.pp"

    node ''standard.node.local'' {
       include install_foo
       include install_bar
    }

---

classes/install_foo.pp:

    class install_foo {
       file { "/":
          ensure  => directory,
          recurse => true,
          source  => "puppet:///files/foo"
       }
    }

---

classes/install_bar.pp:

    class install_bar {
       file { "/":
          ensure  => directory,
          recurse => true,
          source  => "puppet:///files/bar"
       }
    }

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hello!

So, you want to distribute all the package contents using puppet? Can''t
you use yum+RPM or any other packaging system to distribute your
binaries, and use puppet to trigger the installation?

> Given the dynamic nature of the packages (new files are added/removed
> constantly)
That''s unusual. How comes?	

> with error ArgumentError: Duplicate definition: File[/] is already
> defined in file install_foo.pp at line 8; cannot redefine at
Each resource that is managed by puppet needs to be unique to prevent
two manifests from fighting about the same resource. I''m not sure there
is a workaround for your situation. 

Best Regards,
  Claus


On Tue, 2010-03-09 at 11:41 -0800, Mathew Binkley wrote:
> Hi everyone.  I am trying to manage several different packages using
> Puppet.  For the sake of maintainability, each package is installed in
> its own separate puppet directory.  Each package would have a separate
> install_package.pp script and corresponding folder in /etc/puppet/
> files on the server:
> 
>     /etc/puppet/files/foo/
>                          etc/
>                          root/
>                          usr/local
> 
>    /etc/puppet/files/bar/
>                          etc/
>                          var/lib
> 
> To avoid micromanaging, I would like to simply copy over the contents
> of each folder to / on the client.  
[...]


-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mar 9, 2010, at 11:41 AM, Mathew Binkley wrote:
> Hi everyone.  I am trying to manage several different packages using
> Puppet.  For the sake of maintainability, each package is installed in
> its own separate puppet directory.  Each package would have a separate
> install_package.pp script and corresponding folder in /etc/puppet/
> files on the server:
> 
>    /etc/puppet/files/foo/
>                         etc/
>                         root/
>                         usr/local
> 
>   /etc/puppet/files/bar/
>                         etc/
>                         var/lib
> 
> To avoid micromanaging, I would like to simply copy over the contents
> of each folder to / on the client.  While each package has directories
> in common (/etc for example), all of the files in each package are
> orthogonal and not shared between packages.  For example, package foo
> will be the only package that will ever install /etc/foo.conf.
> Package bar will never manage it.
> 
Can''t you split it into stuff that changes and stuff that
doesn''t, and then put the stuff that doesn''t change into a
package.  Then put what ever changes into sub directories?

If that doesn''t work, you could copy and extract a tar file with
something like this:

#Don''t use /tmp because some distro''s nuke that at startup.
#/var/puppet_tmp/ won''t exist.  Make sure you create it.
   file { "/var/puppet_tmp/test.tar.gz":
        owner => "root",
        group => "root",
        mode => 750,
        source => "puppet:///module_name/test.tar.gz",
	require => File["/var/puppet_tmp"]
        }

        exec { "/bin/tar -xvz /var/puppet_tmp/test.tar.gz":
                cwd => "/",
                subscribe => File["/var/puppet_tmp/test.tar.gz"],
		require => File["/var/puppet_tmp/test.tar.gz"]
        }

If you do that a lot, wrapping it in a define shouldn''t be too hard.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

>
> site.pp
>
>    import "classes/install_foo.pp"
>    import "classes/install_bar.pp"
>
>    node ''standard.node.local'' {
>       include install_foo
>       include install_bar
>    }
>
> ---
>
> classes/install_foo.pp:
>
>    class install_foo {
>       file { "/":
>          ensure  => directory,
>          recurse => true,
>          source  => "puppet:///files/foo"
>       }
>    }
>
> ---
>
> classes/install_bar.pp:
>
>    class install_bar {
>       file { "/":
>          ensure  => directory,
>          recurse => true,
>          source  => "puppet:///files/bar"
>       }
>    }
>
> --

Yeah, I would recommend not doing this, and would want to know more
about the use case around why you wanted to do it that way.

If you have multiple sets of applications that you don''t want to
package into something LSB compliant, it would be better to install
each seperate app in /opt or /srv, such as

/opt/foo and /opt/bar

That''s not great by "best practices'' guidelines, but it
works and gets
the job done if you don''t want to package things so that they fully
understand /etc and /var and such.

The error you get is because Puppet will not allow the same resource
to be represented twice, but the larger problem is you''re also kind of
subverting the point of the package manager
if you are overlaying files all over the file system.  You lose the
ability manage dependencies and see how what got where, hence I''d
really recommend asking why the use case is like that.

If you''re not deploying a full app, just perhaps a set of data files
or content, be more specific about where it should go:

 file { "/this/is/where/where/the/files/go":
          ensure  => directory,
          recurse => true,
          source  => "puppet:///files/wherever/bar"
       }

Versus doing paths relative to root.

Further, I don''t really know how many files you are distributing this
way, but if it''s the whole OS, that is going to be rather slow and not
entirely deseriable.   If you strictly
have to do this, you might as well just rsync the content with an Exec
task ... though again, it''s better if you can do something else.

--Michael

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Mar 10, 2010 at 9:59 AM, Michael DeHaan
<michael@reductivelabs.com> wrote:
> Yeah, I would recommend not doing this, and would want to know more
> about the use case around why you wanted to do it that way.
Hi Michael (and Patrick and Claus).  We are in heavy development of a
storage app, as well as collaborating with several universities in
testing their own applications on a storage network we manage.  We
would like to make sure machine X is always running the latest
versions of applications Foo and Bar.

We want to use a flat folder of files because it''s quick and simple
and doesn''t require writing debian build files, RPM specs, or
otherwise having to package the files up in any way given that we are
constantly (every few days) updating binaries and configs.  I don''t
mean simply upgrading an existing file, I''m talking about adding new
files and removing old files.   Essentially we''d like to use Puppet as
an rsync-like tool, albeit with a lot more intelligence and
flexibility.
> The error you get is because Puppet will not allow the same resource
> to be represented twice, but the larger problem is you''re also
kind of
> subverting the point of the package manager
> if you are overlaying files all over the file system.
I know that''s generally true, but in this instance, each
package''s
file are orthogonal.   Only package foo will have /etc/foo.conf.  I
was expecting that Puppet could manage multiple File "/" statements as
long as the files for each package on the Puppet server were unique.
> If you''re not deploying a full app, just perhaps a set of data
files
> or content, be more specific about where it should go:
In a static environment your point make sense.  However we and our
other collaborators are adding/removing files at a furious rate, and
I''d like to avoid spending 15 minutes each time tweaking Puppet config
files.  I know that this is far from the typical use case, but I do
believe it is a *valid* use case.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Mar 10, 2010 at 12:47 PM, Mathew Binkley
<mathewbinkley@gmail.com> wrote:
> On Wed, Mar 10, 2010 at 9:59 AM, Michael DeHaan
> <michael@reductivelabs.com> wrote:
>
>> Yeah, I would recommend not doing this, and would want to know more
>> about the use case around why you wanted to do it that way.
>
> Hi Michael (and Patrick and Claus).  We are in heavy development of a
> storage app, as well as collaborating with several universities in
> testing their own applications on a storage network we manage.  We
> would like to make sure machine X is always running the latest
> versions of applications Foo and Bar.
>
> We want to use a flat folder of files because it''s quick and
simple
> and doesn''t require writing debian build files, RPM specs, or
> otherwise having to package the files up in any way given that we are
> constantly (every few days) updating binaries and configs.  I
don''t
> mean simply upgrading an existing file, I''m talking about adding
new
> files and removing old files.   Essentially we''d like to use
Puppet as
> an rsync-like tool, albeit with a lot more intelligence and
> flexibility.
>
>> The error you get is because Puppet will not allow the same resource
>> to be represented twice, but the larger problem is you''re also
kind of
>> subverting the point of the package manager
>> if you are overlaying files all over the file system.
>
> I know that''s generally true, but in this instance, each
package''s
> file are orthogonal.   Only package foo will have /etc/foo.conf.  I
> was expecting that Puppet could manage multiple File "/"
statements as
> long as the files for each package on the Puppet server were unique.

It can''t, because what you are managing is the root directory
"/",
which also includes potentially conflicting
things like the permissions on those directories.

In your case, you could just use an Exec with an rsync.
>
>> If you''re not deploying a full app, just perhaps a set of data
files
>> or content, be more specific about where it should go:
>
> In a static environment your point make sense.  However we and our
> other collaborators are adding/removing files at a furious rate, and
> I''d like to avoid spending 15 minutes each time tweaking Puppet
config
> files.  I know that this is far from the typical use case, but I do
> believe it is a *valid* use case.

It could be said that time saved now does not mean time saved later.

Best practices around infrastructure are useful, and it does not take
up a tremendous amount of work to set up an RPM build
environment.

Some initial pain will be worth it down the road and keep you from
getting into a mess later.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, 2010-03-10 at 12:59 -0500, Michael DeHaan wrote:
> It could be said that time saved now does not mean time saved later.
> 
Especially if you are "adding/removing files at a furious rate", this
will lead to conflicts sooner or later, and it will be highly
appreciated if you can track down where a specific file or change came
from. 

I can understand that you want to avoid the burden of building
distribution packages. A quick alternative could be to setup separate
directories for each app, like Michael suggested, /opt/foo, /opt/bar,
but to use a version control system such as Subversion to manage and
update the contents. Just checkout the app "foo" to /opt/foo, to
make /opt/foo a working copy. Then commit the latest version to your
central repository and let puppet do the "svn update" on all your
nodes.

Best Regards,
  Claus

PS: Don''t confuse high activity with high productivity. It takes time
to
build robust systems.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.