Puppet users - Mar 2013 - apt-get update before installing a package. The right way?

Hi,

I''ve been battling with trying to get our puppet to do the following:

Install all sources.list, apt-keys
then:
Run an apt-get update
then:
install Packages

I''ve tried a few ways.

1. Set a default requires

Package {
    require => Exec[''apt_update'']
}

This doesn''t work, I think because some packages have a requires
attribute
and so this gets overridden.

2. Stages

I get into huge dependency cycles doing this because I have classes that 
need apt sources that pull in other classes too, eg nagios.

3. Collectors 

Exec[''apt_update''] -> Package <| |>

This works the best but (and it''s a huge but)  it installs all our
virtual
packages everywhere.

What would be great would be if I could do:

Exec[''apt_update''] -> Package

But this isn''t valid.


Does anyone have a sure fire way to get this to work in a large environment?

We''re using the puppetlabs apt class by the way and many other 3rd
party
modules.

Thanks,
Sam
 

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Mar 21, 2013, at 7:16 PM, Sam Morrison <sorrison@gmail.com> wrote:
> Install all sources.list, apt-keys
> then:
> Run an apt-get update
> then:
> install Packages
With the Puppet apt module, apt-get update gets called after installing a new
source.  Or at least that''s how I''ve managed to make it work. 
But it does *not* call apt-get update if there have been no changes to
sources.list.d.  If that''s what you need, I can share my config.  But
if you need to call apt-get update to find new packages added to repositories
already in sources.list, I think you have to find another solution.

It is for this (and oh so many other reasons) that I am learning to detest
Debian-based distros, but I digress.
> 2. Stages
> 
> I get into huge dependency cycles doing this because I have classes that
need apt sources that pull in other classes too, eg nagios.

This may be your best solution.  I create all my apt and yum entries in a stage
before main, then I don''t have to track which package comes from which
repo.  But you could just invoke apt-get update every time with an unrestricted
exec.  You probably want apt-get update to only update periodically or when the
upstream repositories have new packages, but it doesn''t seem to work
that way for me…

--
Brian Lalor
blalor@bravo5.org


-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Thursday, March 21, 2013 6:16:07 PM UTC-5, Sam Morrison
wrote:
>
> Hi,
>
> I''ve been battling with trying to get our puppet to do the
following:
>
> Install all sources.list, apt-keys
> then:
> Run an apt-get update
> then:
> install Packages
>
> I''ve tried a few ways.
>
> 1. Set a default requires
>
> Package {
>     require => Exec[''apt_update'']
> }
>
> This doesn''t work, I think because some packages have a requires
attribute
> and so this gets overridden.
>

That is indeed a gotcha with using resource defaults, but its not usually a 
(necessary) problem for packages.  You could consider looking at which 
packages are declaring their own ''require''s, and why.  They
probably don''t
need to do, except as a means to achieve more or less the same thing
you''re
after.

In a pinch, you can always wrap package declarations in definitions so that 
the defined type carries the explicit ''require'' and the
package just
inherits the default require, but really, I''m having trouble seeing why
it
would be necessary.

 
>
> 2. Stages
>
> I get into huge dependency cycles doing this because I have classes that 
> need apt sources that pull in other classes too, eg nagios.
>

I''m not big on stages, in large part because they significantly magnify
the
risk of cycles, but more fundamentally because they inherently place many 
more constraints on resource application order than are needed.  There is 
nothing you can do with stages that you cannot also do with ordinary 
resource relationships, though using ordinary relationships may be a lot 
more work for the manifest author.

 
>
> 3. Collectors 
>
> Exec[''apt_update''] -> Package <| |>
>
> This works the best but (and it''s a huge but)  it installs all our
virtual
> packages everywhere.
>
> What would be great would be if I could do:
>
> Exec[''apt_update''] -> Package
>
> But this isn''t valid.
>
>
If there were a suitable a selection predicate you could use in your 
collector then that might help you out.  For example:

Exec[''apt_update''] -> Package <| tag ==
''classes_I_care_about'' |>

 
>
> Does anyone have a sure fire way to get this to work in a large 
> environment?
>
> We''re using the puppetlabs apt class by the way and many other 3rd
party
> modules.
>
>
I guess you''re saying that you want to make everything work without 
modifying any third-party modules.  That might not be possible, especially 
if (as I suppose) it is a requirement that the solution be flexible and 
robust.  Your experience with run stages may, in fact, be telling you that 
there is *no* resource-application order consistent with all your 
declarations that applies all apt sources and keys before all packages.  
One simple way that could be true would be if any apt sources or keys were 
themselves installed via packages.

I actually like the resource defaults approach, but you need to audit your 
manifests, including third-party modules, to ensure that Package 
declarations do not provide their own ''require''s.  You can
allow Packages
to do so only to the extent that they are supplying their own means to your 
end, including if they need to override your general rule.  That''s the
only
reason they should need to ''require'' anyway.

For the longer term, you could consider filing a feature request that would 
help you out.  For example, a collector selection criterion or alternative 
collector syntax that collects concrete resources but does not realize 
virtual ones.  Actually, I think there may already be at least one ticket 
that bears on that (separating resource collection from realization).


John



-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Saturday, 23 March 2013 00:50:11 UTC+11, jcbollinger
wrote:
>
>
>
> On Thursday, March 21, 2013 6:16:07 PM UTC-5, Sam Morrison wrote:
>>
>> Hi,
>>
>> I''ve been battling with trying to get our puppet to do the
following:
>>
>> Install all sources.list, apt-keys
>> then:
>> Run an apt-get update
>> then:
>> install Packages
>>
>> I''ve tried a few ways.
>>
>> 1. Set a default requires
>>
>> Package {
>>     require => Exec[''apt_update'']
>> }
>>
>> This doesn''t work, I think because some packages have a
requires
>> attribute and so this gets overridden.
>>
>
>
> That is indeed a gotcha with using resource defaults, but its not usually 
> a (necessary) problem for packages.  You could consider looking at which 
> packages are declaring their own ''require''s, and why. 
They probably don''t
> need to do, except as a means to achieve more or less the same thing
you''re
> after.
>
> In a pinch, you can always wrap package declarations in definitions so 
> that the defined type carries the explicit ''require'' and
the package just
> inherits the default require, but really, I''m having trouble
seeing why it
> would be necessary.
>
>  
>
>>
>> 2. Stages
>>
>> I get into huge dependency cycles doing this because I have classes
that
>> need apt sources that pull in other classes too, eg nagios.
>>
>
>
> I''m not big on stages, in large part because they significantly
magnify
> the risk of cycles, but more fundamentally because they inherently place 
> many more constraints on resource application order than are needed.  There
> is nothing you can do with stages that you cannot also do with ordinary 
> resource relationships, though using ordinary relationships may be a lot 
> more work for the manifest author.
>
>  
>
>>
>> 3. Collectors 
>>
>> Exec[''apt_update''] -> Package <| |>
>>
>> This works the best but (and it''s a huge but)  it installs all
our
>> virtual packages everywhere.
>>
>> What would be great would be if I could do:
>>
>> Exec[''apt_update''] -> Package
>>
>> But this isn''t valid.
>>
>>
> If there were a suitable a selection predicate you could use in your 
> collector then that might help you out.  For example:
>
> Exec[''apt_update''] -> Package <| tag ==
''classes_I_care_about'' |>
>
>  
>
>>
>> Does anyone have a sure fire way to get this to work in a large 
>> environment?
>>
>> We''re using the puppetlabs apt class by the way and many other
3rd party
>> modules.
>>
>>
>
> I guess you''re saying that you want to make everything work
without
> modifying any third-party modules.  That might not be possible, especially 
> if (as I suppose) it is a requirement that the solution be flexible and 
> robust.  Your experience with run stages may, in fact, be telling you that 
> there is *no* resource-application order consistent with all your 
> declarations that applies all apt sources and keys before all packages.  
> One simple way that could be true would be if any apt sources or keys were 
> themselves installed via packages.
>
> I actually like the resource defaults approach, but you need to audit your 
> manifests, including third-party modules, to ensure that Package 
> declarations do not provide their own ''require''s.  You
can allow Packages
> to do so only to the extent that they are supplying their own means to your
> end, including if they need to override your general rule.  That''s
the only
> reason they should need to ''require'' anyway.
>
> For the longer term, you could consider filing a feature request that 
> would help you out.  For example, a collector selection criterion or 
> alternative collector syntax that collects concrete resources but does not 
> realize virtual ones.  Actually, I think there may already be at least one 
> ticket that bears on that (separating resource collection from
realization).
>
>
> John
>
>
>
>
Thank for the info John and Brian,

Separating resource collection from realization would be great. I found 
http://projects.puppetlabs.com/issues/7240 so hopefully that can be 
implemented somehow.

I''ve ended up going with stages for now. I had to modify the handful of
modules that declare apt sources and keys and put these resources into 
their own class within the module. Then I added these class to a stage that 
runs before main,
I think stages can work as long as you keep it simple. 

Cheers,
Sam


 

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.