Puppet users - May 2013 - Running all apt source and update operations first, best practice.

I''ve run into several incidences where a module attempts to install a 
package before the apt::source is added or an update is run. Result is a 
bunch apt errors and explosions.

Basically what should be done is all the apt::sources are added and and an 
update run _before_ any packages are installed to ensure you''re pulling
from the repos you want. 

I''ve gone through several iterations in my attempt to achieve that 
behavior. The one that works best so far is stages and wrapper classes. 
Here''s a terse example of what it looks like:

class myorg::common {
  include stdlib

  Apt::Source {stage => "setup"}  

  apt::source { ''puppetlabs'':
    location   => ''http://apt.puppetlabs.com'',
    repos      => ''main'',
    key        => ''4BD6EC30'',
    key_server => ''pgp.mit.edu'',
  }

  Exec[''apt_update''] -> Package<|  title !=
''ubuntu-cloud-keyring'' |>
}

node ''foo.bar.com'' {
  include stdlib
  
  class {''myorg::common'': stage => "setup"}
}


One thing that bothers me is you have to declare the stage for 
myorg::common in every node that uses it. And as the name implies,
that''s
every node.

Is there a way to get rid of that duplication? I''ve thought of node 
inheritance, but the docs seem to strongly steer you away from that pattern.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Thu, May 9, 2013 at 11:52 AM, James Kyle <lists@jameskyle.org> wrote:
> I''ve run into several incidences where a module attempts to
install a
> package before the apt::source is added or an update is run. Result is a
> bunch apt errors and explosions.
>
> Basically what should be done is all the apt::sources are added and and an
> update run _before_ any packages are installed to ensure you''re
pulling
> from the repos you want.
>
> I''ve gone through several iterations in my attempt to achieve that
> behavior. The one that works best so far is stages and wrapper classes.
> Here''s a terse example of what it looks like:
>
> class myorg::common {
>   include stdlib
>
>   Apt::Source {stage => "setup"}
>
>   apt::source { ''puppetlabs'':
>     location   => ''http://apt.puppetlabs.com'',
>     repos      => ''main'',
>     key        => ''4BD6EC30'',
>     key_server => ''pgp.mit.edu'',
>   }
>
>   Exec[''apt_update''] -> Package<|  title !=
''ubuntu-cloud-keyring'' |>
> }
>
> node ''foo.bar.com'' {
>   include stdlib
>
>   class {''myorg::common'': stage => "setup"}
> }
>
>
> One thing that bothers me is you have to declare the stage for
> myorg::common in every node that uses it. And as the name implies,
that''s
> every node.
>
> Is there a way to get rid of that duplication? I''ve thought of
node
> inheritance, but the docs seem to strongly steer you away from that
pattern.
>
Doesn''t the relationship do the right thing without stages? Does this
work?

class myorg::common (
  $staging  = ''setup'',
) { ...

Nan

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Thursday, May 9, 2013 10:23:25 PM UTC-5, Nan Liu
wrote:
>
> On Thu, May 9, 2013 at 11:52 AM, James Kyle
<li...@jameskyle.org<javascript:>
> > wrote:
>
>> I''ve run into several incidences where a module attempts to
install a
>> package before the apt::source is added or an update is run. Result is
a
>> bunch apt errors and explosions.
>>
>> Basically what should be done is all the apt::sources are added and and
>> an update run _before_ any packages are installed to ensure
you''re pulling
>> from the repos you want. 
>>
>> I''ve gone through several iterations in my attempt to achieve
that
>> behavior. The one that works best so far is stages and wrapper classes.
>> Here''s a terse example of what it looks like:
>>
>> class myorg::common {
>>   include stdlib
>>
>>   Apt::Source {stage => "setup"}  
>>
>>   apt::source { ''puppetlabs'':
>>     location   => ''http://apt.puppetlabs.com'',
>>     repos      => ''main'',
>>     key        => ''4BD6EC30'',
>>     key_server => ''pgp.mit.edu'',
>>   }
>>
>>   Exec[''apt_update''] -> Package<|  title !=
''ubuntu-cloud-keyring'' |>
>> }
>>
>> node ''foo.bar.com'' {
>>   include stdlib
>>   
>>   class {''myorg::common'': stage =>
"setup"}
>> }
>>
>>
>> One thing that bothers me is you have to declare the stage for 
>> myorg::common in every node that uses it. And as the name implies,
that''s
>> every node.
>>
>> Is there a way to get rid of that duplication? I''ve thought of
node
>> inheritance, but the docs seem to strongly steer you away from that
pattern.
>>
>
> Doesn''t the relationship do the right thing without stages?
>

It would depend on whether and how stages are used elsewhere in the 
manifest set.  With stages it''s often "in for a penny, in for a
pound."
Generally speaking, though, it should be possible to do this sort of thing 
(or anything, actually) without stages.

 
> Does this work?
>
> class myorg::common (
>   $staging  = ''setup'',
> ) { ...
>
> Nan
>

Provided that all of the relevant Apt sources are managed via Apt:::Source 
resources, and that no virtual Apt::Source or Package resources are 
declared without elsewhere being realized, this declaration should work:

Apt::Source<| |> -> Exec[''apt_update''] ->
Package<| |>

That declaration is not itself affected by or dependent on stages, but it 
is possible for the referenced resources to be assigned to stages in such a 
way that the chain closes one or more dependency cycles.  Such cycles would 
reflect an error in the stage assignments, however, for the chain 
expression succinctly and precisely describes the requested resource 
application order; as long as it does so, it *defines* the correct orders.  
It would be sufficient to make that declaration once, in some central 
place, but not harmful to make it multiple times.

There are all manner of tweaks and accommodations that could be applied to 
handle special cases, such as when there are virtual Apt::Source or Package 
resources must remain unrealized, but the chain above captures the 
essential essence of the required declaration.


John



-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

This can be achieved without stages if you put the relationship inside 
site.pp outside any class scope.

On Friday, May 10, 2013 8:25:58 AM UTC-6, jcbollinger
wrote:
>
>
>
> On Thursday, May 9, 2013 10:23:25 PM UTC-5, Nan Liu wrote:
>>
>> On Thu, May 9, 2013 at 11:52 AM, James Kyle <li...@jameskyle.org>
wrote:
>>
>>> I''ve run into several incidences where a module attempts
to install a
>>> package before the apt::source is added or an update is run. Result
is a
>>> bunch apt errors and explosions.
>>>
>>> Basically what should be done is all the apt::sources are added and
and
>>> an update run _before_ any packages are installed to ensure
you''re pulling
>>> from the repos you want. 
>>>
>>> I''ve gone through several iterations in my attempt to
achieve that
>>> behavior. The one that works best so far is stages and wrapper
classes.
>>> Here''s a terse example of what it looks like:
>>>
>>> class myorg::common {
>>>   include stdlib
>>>
>>>   Apt::Source {stage => "setup"}  
>>>
>>>   apt::source { ''puppetlabs'':
>>>     location   => ''http://apt.puppetlabs.com'',
>>>     repos      => ''main'',
>>>     key        => ''4BD6EC30'',
>>>     key_server => ''pgp.mit.edu'',
>>>   }
>>>
>>>   Exec[''apt_update''] -> Package<|  title !=
''ubuntu-cloud-keyring'' |>
>>> }
>>>
>>> node ''foo.bar.com'' {
>>>   include stdlib
>>>   
>>>   class {''myorg::common'': stage =>
"setup"}
>>> }
>>>
>>>
>>> One thing that bothers me is you have to declare the stage for 
>>> myorg::common in every node that uses it. And as the name implies,
that''s
>>> every node.
>>>
>>> Is there a way to get rid of that duplication? I''ve
thought of node
>>> inheritance, but the docs seem to strongly steer you away from that
pattern.
>>>
>>
>> Doesn''t the relationship do the right thing without stages?
>>
>
>
> It would depend on whether and how stages are used elsewhere in the 
> manifest set.  With stages it''s often "in for a penny, in for
a pound."
> Generally speaking, though, it should be possible to do this sort of thing 
> (or anything, actually) without stages.
>
>  
>
>> Does this work?
>>
>> class myorg::common (
>>   $staging  = ''setup'',
>> ) { ...
>>
>> Nan
>>
>
>
> Provided that all of the relevant Apt sources are managed via Apt:::Source 
> resources, and that no virtual Apt::Source or Package resources are 
> declared without elsewhere being realized, this declaration should work:
>
> Apt::Source<| |> -> Exec[''apt_update''] ->
Package<| |>
>
> That declaration is not itself affected by or dependent on stages, but it 
> is possible for the referenced resources to be assigned to stages in such a
> way that the chain closes one or more dependency cycles.  Such cycles would
> reflect an error in the stage assignments, however, for the chain 
> expression succinctly and precisely describes the requested resource 
> application order; as long as it does so, it *defines* the correct 
> orders.  It would be sufficient to make that declaration once, in some 
> central place, but not harmful to make it multiple times.
>
> There are all manner of tweaks and accommodations that could be applied to 
> handle special cases, such as when there are virtual Apt::Source or Package
> resources must remain unrealized, but the chain above captures the 
> essential essence of the required declaration.
>
>
> John
>
>
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

This is what I get with the above, slightly adapted to take care of an edge
case:

Apt::Source<| |> -> Exec[''apt_update''] ->
Package<| title !''ubuntu-cloud-keyring'' |>

The ubuntu-cloud-keyring is a prerequisite for adding the ubuntu cloud
archive (for openstack debs). Without this exception, it leads to a cycle
since Sources -> packages otherwise.

Dependency Cycle.
(Anchor[apt::source::puppetlabs] => Apt::Source[puppetlabs] =>
Exec[apt_update] => Class[Apt::Update] => Anchor[apt::source::puppetlabs])


Now, the problem I''m trying to solve is I constantly see these package
errors:

0 upgraded, 17 newly installed, 0 to remove and 12 not upgraded.
Need to get 5504 kB/6061 kB of archives.
After this operation, 20.8 MB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  libvirt0 libvirt-bin
E: There are problems and -y was used without --force-yes



I''m sure this has to do with getting updates to run after adding
sources
because if I hop on the host and do this:

/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install
libvirt-bin

It reproduces the error. But if I then run ''apt-get update''
and rerun it,
no errors.


On Fri, May 10, 2013 at 7:25 AM, jcbollinger
<John.Bollinger@stjude.org>wrote:
>
>
> On Thursday, May 9, 2013 10:23:25 PM UTC-5, Nan Liu wrote:
>
>> On Thu, May 9, 2013 at 11:52 AM, James Kyle <li...@jameskyle.org>
wrote:
>>
>>> I''ve run into several incidences where a module attempts
to install a
>>> package before the apt::source is added or an update is run. Result
is a
>>> bunch apt errors and explosions.
>>>
>>> Basically what should be done is all the apt::sources are added and
and
>>> an update run _before_ any packages are installed to ensure
you''re pulling
>>> from the repos you want.
>>>
>>> I''ve gone through several iterations in my attempt to
achieve that
>>> behavior. The one that works best so far is stages and wrapper
classes.
>>> Here''s a terse example of what it looks like:
>>>
>>> class myorg::common {
>>>   include stdlib
>>>
>>>   Apt::Source {stage => "setup"}
>>>
>>>   apt::source { ''puppetlabs'':
>>>     location   => ''http://apt.puppetlabs.com'',
>>>     repos      => ''main'',
>>>     key        => ''4BD6EC30'',
>>>     key_server => ''pgp.mit.edu'',
>>>   }
>>>
>>>   Exec[''apt_update''] -> Package<|  title !=
''ubuntu-cloud-keyring'' |>
>>> }
>>>
>>> node ''foo.bar.com'' {
>>>   include stdlib
>>>
>>>   class {''myorg::common'': stage =>
"setup"}
>>> }
>>>
>>>
>>> One thing that bothers me is you have to declare the stage for
>>> myorg::common in every node that uses it. And as the name implies,
that''s
>>> every node.
>>>
>>> Is there a way to get rid of that duplication? I''ve
thought of node
>>> inheritance, but the docs seem to strongly steer you away from that
pattern.
>>>
>>
>> Doesn''t the relationship do the right thing without stages?
>>
>
>
> It would depend on whether and how stages are used elsewhere in the
> manifest set.  With stages it''s often "in for a penny, in for
a pound."
> Generally speaking, though, it should be possible to do this sort of thing
> (or anything, actually) without stages.
>
>
>
>> Does this work?
>>
>> class myorg::common (
>>   $staging  = ''setup'',
>> ) { ...
>>
>> Nan
>>
>
>
> Provided that all of the relevant Apt sources are managed via Apt:::Source
> resources, and that no virtual Apt::Source or Package resources are
> declared without elsewhere being realized, this declaration should work:
>
> Apt::Source<| |> -> Exec[''apt_update''] ->
Package<| |>
>
> That declaration is not itself affected by or dependent on stages, but it
> is possible for the referenced resources to be assigned to stages in such a
> way that the chain closes one or more dependency cycles.  Such cycles would
> reflect an error in the stage assignments, however, for the chain
> expression succinctly and precisely describes the requested resource
> application order; as long as it does so, it *defines* the correct
> orders.  It would be sufficient to make that declaration once, in some
> central place, but not harmful to make it multiple times.
>
> There are all manner of tweaks and accommodations that could be applied to
> handle special cases, such as when there are virtual Apt::Source or Package
> resources must remain unrealized, but the chain above captures the
> essential essence of the required declaration.
>
>
> John
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Friday, May 10, 2013 11:13:03 AM UTC-5, James Kyle
wrote:
>
> This is what I get with the above, slightly adapted to take care of an 
> edge case:
>
> Apt::Source<| |> -> Exec[''apt_update''] ->
Package<| title !=
> ''ubuntu-cloud-keyring'' |>
>
> The ubuntu-cloud-keyring is a prerequisite for adding the ubuntu cloud 
> archive (for openstack debs). Without this exception, it leads to a cycle 
> since Sources -> packages otherwise.
>
> Dependency Cycle.
> (Anchor[apt::source::puppetlabs] => Apt::Source[puppetlabs] => 
> Exec[apt_update] => Class[Apt::Update] =>
Anchor[apt::source::puppetlabs])
>

That last link, Class[Apt::Update] => Anchor[apt::source::puppetlabs], 
looks very suspicious.  It suggests that somewhere among your manifests and 
modules and stages, you are declaring that class apt::update should be 
applied *before* whichever class declares Anchor[apt::source::puppetlabs] 
(maybe a class of the same name?).  That seems like it would be erroneous 
-- surely the idea is to get all the apt sources correct first, and only 
then run Exec[apt_update].  Right?

That''s just the kind of thing I was talking about when I said 
[paraphrasing] that the chain was fundamentally right, but it could close a 
dependency cycle if other relationships were wrong.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

That''s what I''m seeing as well, but no idea where
it''s coming from (tips on
how to run something like that down welcome).

I only have a very few modules I''ve written myself and like 3 nodes.
I''ve
removed all dependencies declared there except the one above. And I''ve
grepped the modules directory for all mentions of puuppetlabs source.

Also, the Apt::Source[''puppetlabs''] is a source I declare in
my module and
I don''t see it declared in any other modules.

Wait, this is at the bottom of the modules/apt/manifests/source.pp:

  # Need anchor to provide containment for dependencies.
  anchor { "apt::source::${name}":
    require => Class[''apt::update''],
  }

Could that do it? That''s the Class[apt::update] ->
Anchor[apt::source::puppetlabs] -> Exec[apt_update] cycle right?


On Fri, May 10, 2013 at 10:07 AM, jcbollinger
<John.Bollinger@stjude.org>wrote:
>
>
> On Friday, May 10, 2013 11:13:03 AM UTC-5, James Kyle wrote:
>>
>> This is what I get with the above, slightly adapted to take care of an
>> edge case:
>>
>> Apt::Source<| |> -> Exec[''apt_update''] ->
Package<| title !>> ''ubuntu-cloud-keyring'' |>
>>
>> The ubuntu-cloud-keyring is a prerequisite for adding the ubuntu cloud
>> archive (for openstack debs). Without this exception, it leads to a
cycle
>> since Sources -> packages otherwise.
>>
>> Dependency Cycle.
>> (Anchor[apt::source::**puppetlabs] => Apt::Source[puppetlabs] =>
>> Exec[apt_update] => Class[Apt::Update] => Anchor[apt::source::**
>> puppetlabs])
>>
>
>
> That last link, Class[Apt::Update] => Anchor[apt::source::puppetlabs],
> looks very suspicious.  It suggests that somewhere among your manifests and
> modules and stages, you are declaring that class apt::update should be
> applied *before* whichever class declares Anchor[apt::source::puppetlabs]
> (maybe a class of the same name?).  That seems like it would be erroneous
> -- surely the idea is to get all the apt sources correct first, and only
> then run Exec[apt_update].  Right?
>
> That''s just the kind of thing I was talking about when I said
> [paraphrasing] that the chain was fundamentally right, but it could close a
> dependency cycle if other relationships were wrong.
>
>
>
> John
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On 10.05.2013 18:13, James Kyle wrote:
> I''m sure this has to do with getting updates to run after adding
sources
> because if I hop on the host and do this:
>
> /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install
> libvirt-bin
>
> It reproduces the error. But if I then run ''apt-get
update'' and rerun
> it, no errors.
Yes, apt-get update refreshes the Release files and their signatures 
which are used to verify the downloaded packages.


Regards, David

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Friday, May 10, 2013 1:13:32 PM UTC-5, James Kyle
wrote:
>
> Wait, this is at the bottom of the modules/apt/manifests/source.pp: 
>
>   # Need anchor to provide containment for dependencies.
>   anchor { "apt::source::${name}":
>     require => Class[''apt::update''],
>   }
>
> Could that do it? That''s the Class[apt::update] -> 
> Anchor[apt::source::puppetlabs] -> Exec[apt_update] cycle right?
>
>
That could and would do it.  And it looks wrong, in that it appears to be 
saying that Apt sources should be applied after running "apt-get
update".

Nan: github fingers you as the person responsible for (the current version 
of) that anchor in puppetlabs-apt.  Do I misunderstand its meaning or 
purpose?  Do you agree that it is incorrect?  (Else, why not?)  If it is 
indeed incorrect, then what would you say it should be?


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Do note, by the way, that the idea appears to be that the Apt module 
automatically handles for you what you are trying to do manually: that is, 
to ensure that "apt-get update" is automatically run, once, after all
Apt
sources are managed, so that your packages can safely express relationships 
directly with the appropriate Apt::Source resources.

As I said, the implementation doesn''t look right to me, but
it''s still
Monday morning, so maybe I''m not seeing straight.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.