Puppet users - Aug 2010 - Puppet + APT, I am lost...

Hi,

For the past week, I have been trying to figure out the best way to do
package management with Puppet on an Ubuntu system. I have studied many
solutons I found on the web, but none of them seem to do exactly what I want.

Among other things, I have tried everything that is discussed here:
http://groups.google.com/group/puppet-users/browse_thread/thread/af7349171a5821a4/db96b809c97f6006

but none of that really works.

What I would like, is for Puppet to run `aptitude update` before it
installs or upgrades packages. That doesn''t seem out of the ordinary to
me, but I can''t make it work in Puppet.

The approach I like best, is this one:

    class apt-update {
        exec { "/usr/bin/aptitude update":
            refreshonly => true,
        }
    }

    define apt_package($ensure="latest") {
        include apt-update
        package { $name:
            ensure  => $ensure,
            require => Class["apt-update"],
        }
    }

    apt_package { "foo":
    }

but that doesn''t work, it doesn''t run apt-update before
installing ''foo'',
because of the "refreshonly => true". If I remove it, it works, but
then
`aptitude update` is run on /every/ Puppet run, which is worse than not
running it at all.

Is it at all possible to achieve what I want with Puppet? If so, how?

I have tried to fiddle around with the new run stages feature, but that
got me in a circular dependency mess I couldn''t get out of.

Desperately hoping for hints...

Best regards,
Martijn.

Actually nothing to do with the content of this post but is anyone else
getting a lot of mails from puppet-users at least twice? This particular
one came through 3 times with the same timestamp and same Message-Id.

On 31/08/2010 15:03, Martijn Grendelman wrote:
> Hi,
>
> For the past week, I have been trying to figure out the best way to do
> package management with Puppet on an Ubuntu system. I have studied many
> solutons I found on the web, but none of them seem to do exactly what I
want.
>
> Among other things, I have tried everything that is discussed here:
>
http://groups.google.com/group/puppet-users/browse_thread/thread/af7349171a5821a4/db96b809c97f6006
>
> but none of that really works.
>
> What I would like, is for Puppet to run `aptitude update` before it
> installs or upgrades packages. That doesn''t seem out of the
ordinary to
> me, but I can''t make it work in Puppet.
>
> The approach I like best, is this one:
>
>     class apt-update {
>         exec { "/usr/bin/aptitude update":
>             refreshonly => true,
>         }
>     }
>
>     define apt_package($ensure="latest") {
>         include apt-update
>         package { $name:
>             ensure  => $ensure,
>             require => Class["apt-update"],
>         }
>     }
>
>     apt_package { "foo":
>     }
>
> but that doesn''t work, it doesn''t run apt-update before
installing ''foo'',
> because of the "refreshonly => true". If I remove it, it
works, but then
> `aptitude update` is run on /every/ Puppet run, which is worse than not
> running it at all.
>
> Is it at all possible to achieve what I want with Puppet? If so, how?
>
> I have tried to fiddle around with the new run stages feature, but that
> got me in a circular dependency mess I couldn''t get out of.
>
> Desperately hoping for hints...
>
> Best regards,
> Martijn.
>
-- 

Trevor Hemsley
Infrastructure Engineer
.................................................
* C A L Y P S O
* Brighton, UK   

OFFICE 	+44 (0) 1273 666 350
FAX 	+44 (0) 1273 666 351

.................................................
www.calypso.com

This electronic-mail might contain confidential information intended
only for the use by the entity named. If the reader of this message is
not the intended recipient, the reader is hereby notified that any
dissemination, distribution or copying is strictly prohibited.

* P * /*/ Please consider the environment before printing this e-mail /*/

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Aug 31, 2010 at 10:03 AM, Martijn Grendelman
<martijn@grendelman.net> wrote:
>
> What I would like, is for Puppet to run `aptitude update` before it
> installs or upgrades packages. That doesn''t seem out of the
ordinary to
> me, but I can''t make it work in Puppet.
The run stages of puppet 2.6 addresses this need.  I recommend you
look into adding a class to a stage which runs before the main stage.
The class would contain an exec resource to run aptitude update
periodically.

Hope this helps,
-- 
Jeff McCune
http://www.puppetlabs.com/

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Aug 31, 2010 at 7:03 AM, Martijn Grendelman
<martijn@grendelman.net> wrote:
> Hi,
>
> For the past week, I have been trying to figure out the best way to do
> package management with Puppet on an Ubuntu system. I have studied many
> solutons I found on the web, but none of them seem to do exactly what I
want.
>
> Among other things, I have tried everything that is discussed here:
>
http://groups.google.com/group/puppet-users/browse_thread/thread/af7349171a5821a4/db96b809c97f6006
>
> but none of that really works.
>
> What I would like, is for Puppet to run `aptitude update` before it
> installs or upgrades packages. That doesn''t seem out of the
ordinary to
> me, but I can''t make it work in Puppet.
>
> The approach I like best, is this one:
>
>    class apt-update {
>        exec { "/usr/bin/aptitude update":
>            refreshonly => true,
>        }
>    }
>
>    define apt_package($ensure="latest") {
>        include apt-update
>        package { $name:
>            ensure  => $ensure,
>            require => Class["apt-update"],
>        }
>    }
>
>    apt_package { "foo":
>    }
>
> but that doesn''t work, it doesn''t run apt-update before
installing ''foo'',
> because of the "refreshonly => true". If I remove it, it
works, but then
> `aptitude update` is run on /every/ Puppet run, which is worse than not
> running it at all.
>
> Is it at all possible to achieve what I want with Puppet? If so, how?
Why do you want to run this before every package install? That will
add a fair bit of overhead. I don''t quite understand the desire to run
before every package install, and yet not simply have it run once
before all package installs.

We''ve used two approaches, both using our package::apt::update class
which runs apt-get update/upgrade/dpkg --configure -a etc.

1) Resource defaults:

In the global scope, set a resource default for package to require the
package::apt::update class
The downside is that if you wish to set an additional require for a
specific package, you need to remember to include the global default
as well.

2) Defined types:

Define a type much like yours that has the same require on the package
resource definition inside the defined type.
The downside is that you are no longer just using ''package''
resources,
which makes the use of other people''s modules difficult.
The upside is that we have a similar defined type ''apt_repo''
that sets
up repositories. With appropriate use of require in apt_package and
before in apt_repo, you know your repositories are being laid down
before apt-get update etc get run, and that the packages are installed
afterwards.

Run stages make this a bit simpler, but you can achieve the same
functionality without them.
>
> I have tried to fiddle around with the new run stages feature, but that
> got me in a circular dependency mess I couldn''t get out of.
>
> Desperately hoping for hints...
>
> Best regards,
> Martijn.
>
>


-- 
nigel

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Nigel Kersten,

Thank you for your elaborate answer. I am not sure how to proceed, though...
>> What I would like, is for Puppet to run `aptitude update` before it
>> installs or upgrades packages. That doesn''t seem out of the
ordinary to
>> me, but I can''t make it work in Puppet.
>>
>> The approach I like best, is this one:
>>
>>    class apt-update {
>>        exec { "/usr/bin/aptitude update":
>>            refreshonly => true,
>>        }
>>    }
>>
>>    define apt_package($ensure="latest") {
>>        include apt-update
>>        package { $name:
>>            ensure  => $ensure,
>>            require => Class["apt-update"],
>>        }
>>    }
>>
>>    apt_package { "foo":
>>    }
>>
>> but that doesn''t work, it doesn''t run apt-update
before installing ''foo'',
>> because of the "refreshonly => true". If I remove it, it
works, but then
>> `aptitude update` is run on /every/ Puppet run, which is worse than not
>> running it at all.
>>
>> Is it at all possible to achieve what I want with Puppet? If so, how?
> 
> Why do you want to run this before every package install? That will
> add a fair bit of overhead. I don''t quite understand the desire to
run
> before every package install, and yet not simply have it run once
> before all package installs.
I am not sure how I said I want to do an update before /every/ package
install. Once at the start of a Puppet-run, /IF/ one or more packages
need to be installed, would suffice.
> We''ve used two approaches, both using our package::apt::update
class
> which runs apt-get update/upgrade/dpkg --configure -a etc.
I think I read that in the posting that I referenced.
> 1) Resource defaults:
> 
> In the global scope, set a resource default for package to require the
> package::apt::update class
> The downside is that if you wish to set an additional require for a
> specific package, you need to remember to include the global default
> as well.
> 
> 2) Defined types:
> 
> Define a type much like yours that has the same require on the package
> resource definition inside the defined type.
Isn''t that what I decribed above? The problem is, that by including the
class in the define, it runs everytime, unless ''refreshonly =>
true'' is
set, in which case it never runs. If I don''t include the class in the
define, Puppet can''t find the class when it evaluates the
''require''.

How does that work for you? Would it be possible for you to share
(relevant parts of) your configuration as-is?
> The downside is that you are no longer just using
''package'' resources,
> which makes the use of other people''s modules difficult.
> The upside is that we have a similar defined type
''apt_repo'' that sets
> up repositories. With appropriate use of require in apt_package and
> before in apt_repo, you know your repositories are being laid down
> before apt-get update etc get run, and that the packages are installed
> afterwards.
> 
> Run stages make this a bit simpler, but you can achieve the same
> functionality without them.
I have been looking for examples using run stages, but they are scarce.
The examples in the release notes are too simple to go by.
>> I have tried to fiddle around with the new run stages feature, but that
>> got me in a circular dependency mess I couldn''t get out of.
>>
Best regards,
Martijn.

On Tue, Aug 31, 2010 at 1:02 PM, Martijn Grendelman
<martijn@grendelman.net> wrote:
> Hi Nigel Kersten,
>
> Thank you for your elaborate answer. I am not sure how to proceed,
though...
>
>>> What I would like, is for Puppet to run `aptitude update` before it
>>> installs or upgrades packages. That doesn''t seem out of
the ordinary to
>>> me, but I can''t make it work in Puppet.
>>>
>>> The approach I like best, is this one:
>>>
>>>    class apt-update {
>>>        exec { "/usr/bin/aptitude update":
>>>            refreshonly => true,
>>>        }
>>>    }
>>>
>>>    define apt_package($ensure="latest") {
>>>        include apt-update
>>>        package { $name:
>>>            ensure  => $ensure,
>>>            require => Class["apt-update"],
>>>        }
>>>    }
>>>
>>>    apt_package { "foo":
>>>    }
>>>
>>> but that doesn''t work, it doesn''t run apt-update
before installing ''foo'',
>>> because of the "refreshonly => true". If I remove it,
it works, but then
>>> `aptitude update` is run on /every/ Puppet run, which is worse than
not
>>> running it at all.
>>>
>>> Is it at all possible to achieve what I want with Puppet? If so,
how?
>>
>> Why do you want to run this before every package install? That will
>> add a fair bit of overhead. I don''t quite understand the
desire to run
>> before every package install, and yet not simply have it run once
>> before all package installs.
>
> I am not sure how I said I want to do an update before /every/ package
> install. Once at the start of a Puppet-run, /IF/ one or more packages
> need to be installed, would suffice.
Ah. I''m sorry, I must have misread your post.
>
>> We''ve used two approaches, both using our package::apt::update
class
>> which runs apt-get update/upgrade/dpkg --configure -a etc.
>
> I think I read that in the posting that I referenced.
>
>> 1) Resource defaults:
>>
>> In the global scope, set a resource default for package to require the
>> package::apt::update class
>> The downside is that if you wish to set an additional require for a
>> specific package, you need to remember to include the global default
>> as well.
>>
>> 2) Defined types:
>>
>> Define a type much like yours that has the same require on the package
>> resource definition inside the defined type.
>
> Isn''t that what I decribed above? The problem is, that by
including the
> class in the define, it runs everytime, unless ''refreshonly =>
true'' is
> set, in which case it never runs. If I don''t include the class in
the
> define, Puppet can''t find the class when it evaluates the
''require''.
So I don''t have it refreshonly. I''m perfectly happy with
apt-get
update etc running on every puppet run.

Are you using modules and environments? You will need to make sure the
class is included somewhere, but inside the define isn''t the logical
spot in my opinion.
>
> How does that work for you? Would it be possible for you to share
> (relevant parts of) your configuration as-is?
It really isn''t much more complicated than that :) but I''ll
try and
post something.
>
>> The downside is that you are no longer just using
''package'' resources,
>> which makes the use of other people''s modules difficult.
>> The upside is that we have a similar defined type
''apt_repo'' that sets
>> up repositories. With appropriate use of require in apt_package and
>> before in apt_repo, you know your repositories are being laid down
>> before apt-get update etc get run, and that the packages are installed
>> afterwards.
>>
>> Run stages make this a bit simpler, but you can achieve the same
>> functionality without them.
>
> I have been looking for examples using run stages, but they are scarce.
> The examples in the release notes are too simple to go by.
>
>>> I have tried to fiddle around with the new run stages feature, but
that
>>> got me in a circular dependency mess I couldn''t get out
of.
>>>
>
> Best regards,
> Martijn.
>


-- 
nigel

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

* Martijn Grendelman <martijn at grendelman.net> [2010/08/31
22:02]:
> What I would like, is for Puppet to run `aptitude update` before
> it installs or upgrades packages. That doesn''t seem out of the
> ordinary to me, but I can''t make it work in Puppet.
I would do the `aptitude update` in cron, outside of puppet, and
manage the cronjob via the cron type.  That would eliminate your
precedence and ordering problems completely.

-- 
There are only two hard things in Computer Science: cache invalidation
and naming things.
    -- Phil Karlton

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Aug 31, 2010 at 1:41 PM, Darren Chamberlain <darren@boston.com>
wrote:
> * Martijn Grendelman <martijn at grendelman.net> [2010/08/31 22:02]:
>> What I would like, is for Puppet to run `aptitude update` before
>> it installs or upgrades packages. That doesn''t seem out of the
>> ordinary to me, but I can''t make it work in Puppet.
>
> I would do the `aptitude update` in cron, outside of puppet, and
> manage the cronjob via the cron type.  That would eliminate your
> precedence and ordering problems completely.
Not if you wish to use puppet to manage your repositories as well, and
we do a lot of that.

We do this in our wrapper script

* apt-get update
* apt-get upgrade our-config-tools-metapackage
* run puppetd


That way we can push out a new version of puppet and associated
support tools out of band of the actual puppet run, and then puppet
runs can continue that add additional repositories and install
packages.

This also gives us an out of band way of fixing issues with broken
puppet clients by adding a package to our-config-tools-metapackage and
performing the fixes in the package scripts.


>
> --
> There are only two hard things in Computer Science: cache invalidation
> and naming things.
>    -- Phil Karlton
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
>


-- 
nigel

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi,

Excerpts from Martijn Grendelman''s message of Tue Aug 31 16:02:43 -0400
2010:
> Hi Nigel Kersten,
> 
> >> What I would like, is for Puppet to run `aptitude update` before
it
> >> installs or upgrades packages. That doesn''t seem out of
the ordinary to
> >> me, but I can''t make it work in Puppet.
> >>
> >> The approach I like best, is this one:
> >>
> >>    class apt-update {
> >>        exec { "/usr/bin/aptitude update":
> >>            refreshonly => true,
> >>        }
> >>    }
> >>
> >>    define apt_package($ensure="latest") {
> >>        include apt-update
> >>        package { $name:
> >>            ensure  => $ensure,
> >>            require => Class["apt-update"],
> >>        }
> >>    }
> >>
> >>    apt_package { "foo":
> >>    }
> >>
> >> but that doesn''t work, it doesn''t run apt-update
before installing ''foo'',
> >> because of the "refreshonly => true". If I remove it,
it works, but then
> >> `aptitude update` is run on /every/ Puppet run, which is worse
than not
> >> running it at all.
> >>
> >> Is it at all possible to achieve what I want with Puppet? If so,
how?
> > 
> > Why do you want to run this before every package install? That will
> > add a fair bit of overhead. I don''t quite understand the
desire to run
> > before every package install, and yet not simply have it run once
> > before all package installs.
> 
> I am not sure how I said I want to do an update before /every/ package
> install. Once at the start of a Puppet-run, /IF/ one or more packages
> need to be installed, would suffice.
>
You may run into a chicken-egg problem. Packages that are configured as
"ensure => latest" can only be upgraded by puppet if the local apt
cache
files are up-to-date. 

IOW puppet relies on the local apt files to figure out if packages need
to be updated. And to get new local apt files apt-get update needs to be
run.

-- 
Mathias Gug
Ubuntu Developer  http://www.ubuntu.com

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Nigel Kersten schreef:
> So I don''t have it refreshonly. I''m perfectly happy with
apt-get
> update etc running on every puppet run.
Ah, that explains ;-) But I was hoping that wouldn''t be necessary.
> Are you using modules and environments? You will need to make sure the
> class is included somewhere, but inside the define isn''t the
logical
> spot in my opinion.
Well, that''s a big shortcoming in my understanding of how Puppet works,
namespaces and stuff like that. But I''m learning ;-)
>> How does that work for you? Would it be possible for you to share
>> (relevant parts of) your configuration as-is?
> 
> It really isn''t much more complicated than that :) but
I''ll try and
> post something.
If this is it, then I guess I''m OK, thanks!

Best regards,
Martijn.

Mathias Gug schreef:
>> I am not sure how I said I want to do an update before /every/ package
>> install. Once at the start of a Puppet-run, /IF/ one or more packages
>> need to be installed, would suffice.
> 
> You may run into a chicken-egg problem. Packages that are configured as
> "ensure => latest" can only be upgraded by puppet if the local
apt cache
> files are up-to-date. 
> 
> IOW puppet relies on the local apt files to figure out if packages need
> to be updated. And to get new local apt files apt-get update needs to be
> run.
> 
I am aware of that, but that''s not a problem. I could easily run an
update every day outside of Puppet (on Debian, APT installs a cronjob
that can take care of this by default, it just needs to be configured).

That way, Puppet would pick up the new versions sooner or later. But
what I am trying to achieve, has a different background.

Our development team regularly releases new software, packaged in a
Debian package with a unique name. The idea is that we just give Puppet
a new package {foo: } and have these new packages be installed ASAP. We
even use triggered runs to speed things up and to make sure that several
servers remain identical.

But for this to work, the Puppet run needs to run aptitude update to
pick up the new package name. Running the update periodically isn''t
enough, but running an update on every catalog run is just overkill.

Hope this explains my motives, and maybe someone has any more tips!

Best regards,
Martijn.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 08/31/2010 06:41 PM, Trevor Hemsley wrote:
>  Actually nothing to do with the content of this post but is anyone else
> getting a lot of mails from puppet-users at least twice? This particular
> one came through 3 times with the same timestamp and same Message-Id.
/me too.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkx+RL8ACgkQbwltcAfKi3+YHgCfTZZnTNBNTb/IRxkSeoKR3OW5
cSMAn2cbYt6D290CQv08KObEe0VujALP
=tMxT
-----END PGP SIGNATURE-----

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Aug 31, 2010 at 04:03:35PM +0200, Martijn Grendelman
wrote:
> What I would like, is for Puppet to run `aptitude update` before it
> installs or upgrades packages.
FYI you''re likely to encounter problems when using aptitude provider
for
package management. See Debian bugs #445034 #445035

regards,
-- 
Marcin Owsiany <marcin@owsiany.pl>              http://marcin.owsiany.pl/
GnuPG: 1024D/60F41216  FE67 DA2D 0ACA FC5E 3F75  D6F6 3A0D 8AA0 60F4 1216
 
"Every program in development at MIT expands until it can read mail."
                                                              -- Unknown

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 05-09-10 12:02, Marcin Owsiany wrote:
> On Tue, Aug 31, 2010 at 04:03:35PM +0200, Martijn Grendelman wrote:
>> What I would like, is for Puppet to run `aptitude update` before it
>> installs or upgrades packages.
> 
> FYI you''re likely to encounter problems when using aptitude
provider for
> package management. See Debian bugs #445034 #445035
Thanks. The former is not really important to me, since I never use
aptitude that way. The latter is probably a little more tricky, but at
least the ''unknown package'' case is handled by Puppet itself:

# Yay, stupid aptitude doesn''t throw an error when the package is
missing.
if args.include?(:install) and output =~ /Couldn''t find any package/
  raise Puppet::Error.new(
    "Could not find package #{self.name}"
  )
end

I have now solved my problem by using a custom provider for packages, that
is based on the aptitude provider, but simply adds this:

# See if we need to run aptitude update before installing this package
if not File.exists?("/tmp/puppet-iphionapt")
  FileUtils.touch("/tmp/puppet-iphionapt")
  aptitude("update")
end

I just haven''t found a suitable way to clean up the tempfile from
within
Puppet without touching any of Puppet''s own files, so I use a cronjob
for
that now. Maybe I''ll switch to a ''postrun_command'',
or does anyone have a
better idea?

Best regards,
Martijn.

On Tue, Aug 31, 2010 at 3:35 PM, Martijn Grendelman
<martijn@grendelman.net> wrote:
> But for this to work, the Puppet run needs to run aptitude update to
> pick up the new package name. Running the update periodically
isn''t
> enough, but running an update on every catalog run is just overkill.

I understand your concern here, but have you done the timing tests?

How long are your average puppet runs and how much time does an
"apt-get update" take?

I just wonder if you''re prematurely optimizing your runs.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 01.09.2010 01:35, Martijn Grendelman wrote:
> Mathias Gug schreef:
>>> I am not sure how I said I want to do an update before /every/
package
>>> install. Once at the start of a Puppet-run, /IF/ one or more
packages
>>> need to be installed, would suffice.
>> You may run into a chicken-egg problem. Packages that are configured as
>> "ensure =>  latest" can only be upgraded by puppet if the
local apt cache
>> files are up-to-date.
>>
>> IOW puppet relies on the local apt files to figure out if packages need
>> to be updated. And to get new local apt files apt-get update needs to
be
>> run.
>>
> I am aware of that, but that''s not a problem. I could easily run
an
> update every day outside of Puppet (on Debian, APT installs a cronjob
> that can take care of this by default, it just needs to be configured).
>
> That way, Puppet would pick up the new versions sooner or later. But
> what I am trying to achieve, has a different background.
>
> Our development team regularly releases new software, packaged in a
> Debian package with a unique name. The idea is that we just give Puppet
> a new package {foo: } and have these new packages be installed ASAP. We
> even use triggered runs to speed things up and to make sure that several
> servers remain identical.
>
> But for this to work, the Puppet run needs to run aptitude update to
> pick up the new package name. Running the update periodically
isn''t
> enough, but running an update on every catalog run is just overkill.
>
> Hope this explains my motives, and maybe someone has any more tips!
>
> Best regards,
> Martijn.
As it was said earlier if you have a ensure => latest you should 
probably have (also) a cron job for updates.
For your case you may try a define like (using 
http://forge.puppetlabs.com/ripienaar/concat ; read the comments in the 
pp files)

define fastupdate_package(
     $reason,
     $date
)
{
     concat::fragment
     {"${name}_install":
         target  => "/var/log/fastupdate_package.log",
         order   => $date,
         content => "${date} Installed package ${name}
(${reason})",
     }
     package
     {$name:
     #... stuff
         require    => Exec["aptupdate"],
     }
}


when using you should also have included in a class something similar to

concat
{ "/var/log/fastupdate_package.log":
     notify => Exec["aptupdate"],
}
exec
{ "aptupdate":
     command => "/usr/bin/aptitude update":
     refreshonly => true,
}

This should create something like a log file (not an actual log file, 
since the new data is not appended, but recreated when necessary). This 
would actually create some chaos if used for normal packages, so I 
highly recommend using this in a module that only involves your 
packages, and for the rest of packages use the cron solution, which is 
way way more cleaner than this.


Hope it helps, :)
Silviu

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mon, Sep 6, 2010 at 10:56 AM, Silviu Paragina <silviu@paragina.ro>
wrote:
> As it was said earlier if you have a ensure => latest you should
probably
> have (also) a cron job for updates.
I think it''s much more difficult to manage multiple schedules, and to
deal with inconsistencies when people manually run Puppet at times
that aren''t in sync with your cron job than it is for Puppet to simply
ensure the appropriate commands are run before your packages are
installed.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 06.09.2010 21:03, Nigel Kersten wrote:
> On Mon, Sep 6, 2010 at 10:56 AM, Silviu Paragina<silviu@paragina.ro> 
wrote:
>
>> As it was said earlier if you have a ensure =>  latest you should
probably
>> have (also) a cron job for updates.
> I think it''s much more difficult to manage multiple schedules, and
to
> deal with inconsistencies when people manually run Puppet at times
> that aren''t in sync with your cron job than it is for Puppet to
simply
> ensure the appropriate commands are run before your packages are
> installed.
>
Could you be a little more specific, I''m not sure if we are referring
to
the same cron job. I''m not sure if you are referring to a "apt-get
update" or a "puppet" cron job. I was referring to a apt-get
update cron.

You are right of course. If the apt-get update cron fails (or it hasn''t
got the chance to run), puppet will have no ideea that it failed, and 
will still run the package resources, which in some scenarios might be 
catastrophic. But on the other hand if it just works like that and you 
really really need the resources, you might consider using it. Thinking 
about it an apt-get update shouldn''t have more than 3-4 small files per
repository if the cache is hit, and otherwise it should have at most 10 
files around a few hundred bytes (in case of incremental repositories, 
not sure if incremental is the right technical word, but I hope you 
understand what I meant) so there isn''t much of a penalty there, more 
supporting your solution. I should of probably noted this in my answer, 
but most of it was already discussed in the thread already.

Now my solution was proposed in the idea that under no circumstance 
apt-get update should be run every run (and, yes, I didn''t take into 
account that the reason for that might be wrong).

I feel that I should point out again, that the solution is hackish and 
as such, if used, it should be used only in a single module, preferably 
in a single manifest, detailing in comments all the reasons for doing 
things the way it does. And most of all if somebody doesn''t follow the 
convention it all goes down the drain. :) So my solution should be 
avoided if possible, but if it can not be avoided, it works with the 
notes above.



Silviu

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mon, Sep 6, 2010 at 11:52 AM, Silviu Paragina <silviu@paragina.ro>
wrote:
>  On 06.09.2010 21:03, Nigel Kersten wrote:
>>
>> On Mon, Sep 6, 2010 at 10:56 AM, Silviu
Paragina<silviu@paragina.ro>
>>  wrote:
>>
>>> As it was said earlier if you have a ensure =>  latest you
should
>>> probably
>>> have (also) a cron job for updates.
>>
>> I think it''s much more difficult to manage multiple schedules,
and to
>> deal with inconsistencies when people manually run Puppet at times
>> that aren''t in sync with your cron job than it is for Puppet
to simply
>> ensure the appropriate commands are run before your packages are
>> installed.
>>
> Could you be a little more specific, I''m not sure if we are
referring to the
> same cron job. I''m not sure if you are referring to a
"apt-get update" or a
> "puppet" cron job. I was referring to a apt-get update cron.
Yep. That''s the one I was talking about.
>
> You are right of course. If the apt-get update cron fails (or it
hasn''t got
> the chance to run), puppet will have no ideea that it failed, and will
still
> run the package resources, which in some scenarios might be catastrophic.
> But on the other hand if it just works like that and you really really need
> the resources, you might consider using it. Thinking about it an apt-get
> update shouldn''t have more than 3-4 small files per repository if
the cache
> is hit, and otherwise it should have at most 10 files around a few hundred
> bytes (in case of incremental repositories, not sure if incremental is the
> right technical word, but I hope you understand what I meant) so there
isn''t
> much of a penalty there, more supporting your solution. I should of
probably
> noted this in my answer, but most of it was already discussed in the thread
> already.
>
> Now my solution was proposed in the idea that under no circumstance apt-get
> update should be run every run (and, yes, I didn''t take into
account that
> the reason for that might be wrong).
>
> I feel that I should point out again, that the solution is hackish and as
> such, if used, it should be used only in a single module, preferably in a
> single manifest, detailing in comments all the reasons for doing things the
> way it does. And most of all if somebody doesn''t follow the
convention it
> all goes down the drain. :) So my solution should be avoided if possible,
> but if it can not be avoided, it works with the notes above.
I''m pretty sure we''re in agreement here :)


>
>
>
> Silviu
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>


-- 
nigel

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi,
>> But for this to work, the Puppet run needs to run aptitude update to
>> pick up the new package name. Running the update periodically
isn''t
>> enough, but running an update on every catalog run is just overkill.
> 
> I understand your concern here, but have you done the timing tests?
> 
> How long are your average puppet runs and how much time does an
> "apt-get update" take?
> 
> I just wonder if you''re prematurely optimizing your runs.
It''s not actually about optimizing puppet runs, it''s about the
load
imposed on the package repository servers, when hundreds of servers do
updates twice an hour.

And even if that load isn''t a real problem (which it probably
isn''t), it''s
about politeness to the sysadmins of those servers. The behaviour that
I''m
trying to avoid could easily be explained as abuse. I just like to play nice.

Best regards,
Martijn.

Hi,

Just to let you know how I implemented it in the end. Earlier I wrote:
> I have now solved my problem by using a custom provider for packages, that
> is based on the aptitude provider, but simply adds this:
> 
> # See if we need to run aptitude update before installing this package
> if not File.exists?("/tmp/puppet-iphionapt")
>   FileUtils.touch("/tmp/puppet-iphionapt")
>   aptitude("update")
> end
> 
> I just haven''t found a suitable way to clean up the tempfile from
within
> Puppet without touching any of Puppet''s own files, so I use a
cronjob for
> that now. Maybe I''ll switch to a
''postrun_command'', or does anyone have a
> better idea?
I have put the custom package provider in a module. This module also has a
''cleanup'' class, that removes the tempfile. The cleanup class
is run in a
''post'' runstage. The custom provider is used by default for
all packages.

On the first package install, aptitude update is run, but not for
subsequent installs in the same run. At the end of the run, the tempfile
is removed.

Best regards,
Martijn.

> For the past week, I have been trying to figure out the best way to do
> package management with Puppet on an Ubuntu system. I have studied many
> solutons I found on the web, but none of them seem to do exactly what I
want.

I am not sure if somebody proposed this already, but may be you can
use this approach:

In the puppetmaster:

 * you have a puppet-file repository called files/debian-reps-sums or
something like that.
 * You have an script scheduled each 5, 15 or 30m, that will write
files for each repository with the sums. Something like this:

   wget http://ftp.fr.debian.org/debian/dists/lenny/Release -q -O - |grep
main/binary-i386 > ftp.fr.debian.org_lenny_main_binary-i386.sums

In the configuration:
 * Each host that depens on that repository, will has a reference on
that file, storing it somewhere. You add a "notify =>
Exec["update-repository"]. Exec["update-repository"], of
cuorse,
should be notify-only.

     # Force apt-get update if needed
    file {
"${debianmirror}_${distribution}_${component}_binary-${architecture}.sums":
        source =>
"puppet:///debian-reps-sums/${debianmirror}_${distribution}_${component}_binary-${architecture}.sums",
        path   =>
"/var/lib/puppet/debian-reps-sums/${debianmirror}_${distribution}_${component}_binary-${architecture}.sums",
        owner  => "root",
        group  => 0,
        mode   => "0664",
        notify => Exec["apt-get_update"]
    }

You can create a definition to configure repositories and add this dependency.

I found this method in a to control "stow" installations from
http://trac.cae.tntech.edu/infrastructure/

With this method:
 * apt-get update will run on client only when needed.
 * Only one node (puppetmaster) will check if there are new updates.
 * You can easily force apt-get update if needed

--
Atentamente
Héctor Rivas

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Sep 7, 2010, at 1:22 AM, Martijn Grendelman wrote:
> Hi,
> 
>>> But for this to work, the Puppet run needs to run aptitude update
to
>>> pick up the new package name. Running the update periodically
isn''t
>>> enough, but running an update on every catalog run is just
overkill.
>> 
>> I understand your concern here, but have you done the timing tests?
>> 
>> How long are your average puppet runs and how much time does an
>> "apt-get update" take?
>> 
>> I just wonder if you''re prematurely optimizing your runs.
> 
> It''s not actually about optimizing puppet runs, it''s
about the load
> imposed on the package repository servers, when hundreds of servers do
> updates twice an hour.
I use the apt-cacher-ng proxy to reduce load on the package servers and for
speed.  It means that I can download packages over a 100Mbps connection.
> And even if that load isn''t a real problem (which it probably
isn''t), it''s
> about politeness to the sysadmins of those servers. The behaviour that
I''m
> trying to avoid could easily be explained as abuse. I just like to play
nice.
> 
> Best regards,
> Martijn.
> 
> 
> 
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Patrick <kc7zzv@gmail.com> writes:
> On Sep 7, 2010, at 1:22 AM, Martijn Grendelman wrote:
>
>>>> But for this to work, the Puppet run needs to run aptitude
update to pick
>>>> up the new package name. Running the update periodically
isn''t enough,
>>>> but running an update on every catalog run is just overkill.
>>>
>>> I understand your concern here, but have you done the timing tests?
>>>
>>> How long are your average puppet runs and how much time does an
>>> "apt-get update" take?
>>>
>>> I just wonder if you''re prematurely optimizing your runs.
>>
>> It''s not actually about optimizing puppet runs, it''s
about the load imposed
>> on the package repository servers, when hundreds of servers do updates
>> twice an hour.
>
> I use the apt-cacher-ng proxy to reduce load on the package servers and for
> speed.  It means that I can download packages over a 100Mbps connection.
So do we; this is nothing but polite behaviour when you have a large pool of
machines hitting the same resources time and again.

A word of warning from painful discoveries of the past: while apt can be a bit
of a mirror and link load if you make it update, yum is a huge risk: it will
do a mirror update every time you invoke it that it thinks the lists are out
of date.

On one deployment we had that running every hour and yum updated every hour,
generating gigabytes of traffic until we put a proxy in place between it and
the outside world.

You can tune these things, but out of the box you run a huge risk if you
don''t
funnel your updates through some sort of sensible proxy server...

        Daniel

-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.