Anyone care to share how they handle yum updates? We have a script that runs yum update nightly and then emails us the results when there are packages to be updated. Ideally I''d like a way to keep a class in site.pp that can be uncommented to run yum update -y after the update list is reviewed by a human. Tim
On Sat, Nov 03, 2007 at 08:33:09PM -0400, Tim.Metz@cox.com wrote:> Anyone care to share how they handle yum updates?Not yum, but packages is packages (at least at this level)> We have a script that runs yum update nightly and then emails us the > results when there are packages to be updated. Ideally I''d like a way to > keep a class in site.pp that can be uncommented to run yum update -y > after the update list is reviewed by a human.Blech. I use custom package repositories for packages from third-party sites that I might want to use. We monitor the sites for new packages, then download and test them manually, and once we''re happy they''re OK, they go into the local package repo and get automatically installed the next evening. The problem with the uncommenting-the-class solution is many, and they''re mostly the standard push-vs-pull ones -- what happens if a machine is down during the time period I''ve got the class uncommented, what happens if another update sneaks in while I''ve got the class uncommented, that sort of thing. All the usual pitfalls. - Matt
> Ideally I''d like a way to keep a class in site.pp that can be uncommented to run yum update -y >after the update list is reviewed by a human.See the last segment of http://reductivelabs.com/trac/puppet/wiki/DebianRecipies for one strategy. Obviously, replace the apt-related lines with the appropriate yum commands. -- Mike Renfro / R&D Engineer, Center for Manufacturing Research 931 372-3601 / Tennessee Technological University
Creative, I like it. Tim -----Original Message----- From: puppet-users-bounces@madstop.com [mailto:puppet-users-bounces@madstop.com] On Behalf Of Mike Renfro Sent: Sunday, November 04, 2007 12:29 PM To: puppet-users@madstop.com Subject: Re: [Puppet-users] Package management> Ideally I''d like a way to keep a class in site.pp that can beuncommented to run yum update -y>after the update list is reviewed by a human.See the last segment of http://reductivelabs.com/trac/puppet/wiki/DebianRecipies for one strategy. Obviously, replace the apt-related lines with the appropriate yum commands. -- Mike Renfro / R&D Engineer, Center for Manufacturing Research 931 372-3601 / Tennessee Technological University _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On 04/11/2007, at 18.29, Mike Renfro wrote:> See the last segment of http://reductivelabs.com/trac/puppet/wiki/ > DebianRecipies for one strategy. Obviously, replace the apt-related > lines with the appropriate yum commands.And if you combine it with a schedule that fx. says "only run it between 0700 and 0800" you''ll know when to be ready for possible problems, caused by the updates... ;-) -- Med venlig hilsen Juri Rischel Jensen Fab:IT ApS Vesterbrogade 50 DK-1620 København Tlf: 70 202 407 / Fax: 33 313 640 www.fab-it.dk / juri@fab-it.dk
On 11/4/07, Juri Rischel Jensen <juri@fab-it.dk> wrote:> And if you combine it with a schedule that fx. says "only run it > between 0700 and 0800" you''ll know when to be ready for possible > problems, caused by the updates... ;-)Honestly, that''s within 30 minutes after you change the trigger file, of course :) No need for a schedule, I think. -- Gegroet, Tim
On 05/11/2007, at 8.50, Tim Stoop wrote:> Honestly, that''s within 30 minutes after you change the trigger file, > of course :) No need for a schedule, I think.Well, if you have an SLA with your customers, it''s nice to be able to update the file any time of day, and know that the update only happens at the specified time (and I personally like that the updates happens in the morning, which gives me the whole day to fix things if they blow up...) -- Med venlig hilsen Juri Rischel Jensen Fab:IT ApS Vesterbrogade 50 DK-1620 København Tlf: 70 202 407 / Fax: 33 313 640 www.fab-it.dk / juri@fab-it.dk
On Sun, 04 Nov 2007 11:29:00 -0600, Mike Renfro wrote:>> Ideally I''d like a way to keep a class in site.pp that can be >> uncommented to run yum update -y >>after the update list is reviewed by a human. > > See the last segment of > http://reductivelabs.com/trac/puppet/wiki/DebianRecipies for one > strategy. Obviously, replace the apt-related lines with the appropriate > yum commands.On the subject of Debian package management, I''m wondering how people handle security upgrades for their packages. Not all my systems have libperl5.8 installed, but some of them do as a result of dependencies for other packages that I have specifically told puppet to install. Because I dont have libperl5.8 explicitly managed in puppet, when it comes time for a security upgrade, the machines that have it will need upgrading, and the machines that don''t do not. I dont want to install it on all the other systems by just making a global package type. I also dont want to be chasing package dependencies that were dragged in all over my manifests. How are other people handling this? I was thinking that maybe if I had a way to say "if package is installed, then ensure => ''version.string'' ", otherwise ensure => absent... that way when the security upgrade comes out I can explicitly note that I want this package updated, only on the systems that it is installed on. The ''check'' metaparameter seems like it could be used for this, but it doesn''t seem like its there yet, similarly the ''status'' parameter, but likewise seemingly internal at the moment. I can do: exec { "apt-get install libperl5.8=5.8.8-7etch1": onlyif => "dpkg --get-selections |grep libperl5.8 |grep -q install" } but thats a bit ugly too... Micah
micah <micah@riseup.net> writes:> On the subject of Debian package management, I''m wondering how people > handle security upgrades for their packages. Not all my systems have > libperl5.8 installed, but some of them do as a result of dependencies > for other packages that I have specifically told puppet to > install. Because I dont have libperl5.8 explicitly managed in puppet, > when it comes time for a security upgrade, the machines that have it > will need upgrading, and the machines that don''t do not. I dont want to > install it on all the other systems by just making a global package > type. I also dont want to be chasing package dependencies that were > dragged in all over my manifests.> How are other people handling this? I was thinking that maybe if I had a > way to say "if package is installed, then ensure => ''version.string'' ", > otherwise ensure => absent... that way when the security upgrade comes > out I can explicitly note that I want this package updated, only on the > systems that it is installed on.I''m not sure this is really a place where Puppet is the right tool. aptitude already knows how to check for newer packages, and if you''re running Debian stable, the only newer packages you see are security updates or fixes for bugs so major that you want the new package anyway. Rather than having Puppet add another layer of complexity over the package manager, I''d just periodically run aptitude update and then aptitude upgrade (via whatever system you want for manual checking of the actions first, testing of new packages first, etc. based on your local requirements). It''s generally good practice to run aptitude upgrade periodically on any Debian stable system. -- Russ Allbery (rra@stanford.edu) <http://www.eyrie.org/~eagle/>
On Wed, 07 Nov 2007 10:27:35 -0800, Russ Allbery wrote:> micah <micah@riseup.net> writes:> I''m not sure this is really a place where Puppet is the right tool. > aptitude already knows how to check for newer packages, and if you''re > running Debian stable, the only newer packages you see are security > updates or fixes for bugs so major that you want the new package anyway. > Rather than having Puppet add another layer of complexity over the > package manager, I''d just periodically run aptitude update and then > aptitude upgrade (via whatever system you want for manual checking of > the actions first, testing of new packages first, etc. based on your > local requirements). > > It''s generally good practice to run aptitude upgrade periodically on any > Debian stable system.This would be fine, but I''m not so comfortable with unmanaged upgrades. Even in stable, package updates come that have unintended regressions, and if I have installed something from volatile the chances of something odd happening is even higher. I would much rather test an upgrade, confirm its ok, then initiate the upgrade environment wide than find out that at 3am aptitude upgrade was run and mail delivery has been down all night because of something unexpected.
micah wrote:> On Wed, 07 Nov 2007 10:27:35 -0800, Russ Allbery wrote: > > This would be fine, but I''m not so comfortable with unmanaged > upgrades. Even in stable, package updates come that have unintended > regressions, and if I have installed something from volatile the > chances of something odd happening is even higher.You could schedule a job (via cron, puppet, or whatever) to run your metapackage manager (such as apt-get) in dry-run mode. Essentially, this will generate a report from the nodes when they think they have old software. Then, you can install the upgrades as desired. For example: apt-get -qq update; apt-get -s upgrade Really, this is a change management problem, and Puppet is not a change management tool. But puppet can certainly help tell you when changes are needed.
On Wed, 07 Nov 2007 14:03:02 -0500, Brendan Strejcek wrote:> micah wrote: > >> On Wed, 07 Nov 2007 10:27:35 -0800, Russ Allbery wrote: >> >> This would be fine, but I''m not so comfortable with unmanaged upgrades. >> Even in stable, package updates come that have unintended regressions, >> and if I have installed something from volatile the chances of >> something odd happening is even higher. > > You could schedule a job (via cron, puppet, or whatever) to run your > metapackage manager (such as apt-get) in dry-run mode. Essentially, this > will generate a report from the nodes when they think they have old > software. Then, you can install the upgrades as desired. > > For example: > > apt-get -qq update; apt-get -s upgrade > > Really, this is a change management problem, and Puppet is not a change > management tool. But puppet can certainly help tell you when changes are > needed.I have no problem knowing *when* I need to upgrade packages on the various systems. There are a number of tools to provide me with this information (such as apticron), that is not the problem I am trying to solve. Its more of how to manage the package upgrades, once I know this information. micah
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday 07 November 2007, micah wrote:> On Sun, 04 Nov 2007 11:29:00 -0600, Mike Renfro wrote: > >> Ideally I''d like a way to keep a class in site.pp that can be > >> uncommented to run yum update -y > >>after the update list is reviewed by a human. > > > > See the last segment of > > http://reductivelabs.com/trac/puppet/wiki/DebianRecipies for one > > strategy. Obviously, replace the apt-related lines with the appropriate > > yum commands. > > On the subject of Debian package management, I''m wondering how people > handle security upgrades for their packages. Not all my systems have > libperl5.8 installed, but some of them do as a result of dependencies for > other packages that I have specifically told puppet to install. Because I > dont have libperl5.8 explicitly managed in puppet, when it comes time for > a security upgrade, the machines that have it will need upgrading, and > the machines that don''t do not. I dont want to install it on all the > other systems by just making a global package type. I also dont want to > be chasing package dependencies that were dragged in all over my > manifests. > > How are other people handling this?There was a discussion on IRC recently about this topic. The conclusion seemed to be that the "clean" way to handle this is to have a local repo where only tested upgrades are entered. Then it is save to unconditionally apt-get dist-upgrade. Regards, David - -- The primary freedom of open source is not the freedom from cost, but the free- dom to shape software to do what you want. This freedom is /never/ exercised without cost, but is available /at all/ only by accepting the very different costs associated with open source, costs not in money, but in time and effort. - -- http://www.schierer.org/~luke/log/20070710-1129/on-forks-and-forking -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFHMhyC/Pp1N6Uzh0URAlvkAJ47VzWF3QxeVXtGKF1DAOwv4hR+oACcDCvG o4I0QI/q+RvdlPkqqpmSEjA=6l5/ -----END PGP SIGNATURE-----
On Wed, Nov 07, 2007 at 06:03:12PM +0000, micah wrote:> On the subject of Debian package management, I''m wondering how people > handle security upgrades for their packages. Not all my systems have > libperl5.8 installed, but some of them do as a result of dependencies for > other packages that I have specifically told puppet to install. Because I > dont have libperl5.8 explicitly managed in puppet, when it comes time for > a security upgrade, the machines that have it will need upgrading, and > the machines that don''t do not. I dont want to install it on all theThe way I deal with this is to have apt-cron download but not install packages for upgrade overnight. Once I''m happy that the upgrade is OK, I use the Apt recipe on the wiki to initiate the upgrade on all the various machines. Because it''s not downloading new packages, but merely installing those that have already been downloaded, I can be sure that no little surprises creep in. None of the CentOS machines I look after are currently managed by Puppet, but if someone has a way of doing this for yum I''d be grateful. At present I''m simply using local repositories with yum upgrade running from cron. -- Sam Bashton - Bashton Ltd, Manchester, England Linux Consultancy / VOIP Telephony / High Availability Systems www.bashton.com - 0161-424-9600 _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Wed, Nov 07, 2007 at 06:03:12PM +0000, micah wrote:> On the subject of Debian package management, I''m wondering how people > handle security upgrades for their packages. Not all my systems have > libperl5.8 installed, but some of them do as a result of dependencies for > other packages that I have specifically told puppet to install. Because I > dont have libperl5.8 explicitly managed in puppet, when it comes time for > a security upgrade, the machines that have it will need upgrading, and > the machines that don''t do not. I dont want to install it on all the > other systems by just making a global package type. I also dont want to > be chasing package dependencies that were dragged in all over my > manifests. > > How are other people handling this?I have a script that installs all needing-to-be-upgraded packages except ones whose name matches one or more of a series of regular expressions I''ve defined (since, for local reasons, some packages can''t stand to be upgraded automatically). That, combined with a fairly anal-retentive policy on letting new updates into my local repositories, has kept everything sane so far. - Matt
On Wed, 07 Nov 2007 20:30:30 +0000, Sam Bashton wrote:> On Wed, Nov 07, 2007 at 06:03:12PM +0000, micah wrote: >> On the subject of Debian package management, I''m wondering how people >> handle security upgrades for their packages. Not all my systems have >> libperl5.8 installed, but some of them do as a result of dependencies >> for other packages that I have specifically told puppet to install. >> Because I dont have libperl5.8 explicitly managed in puppet, when it >> comes time for a security upgrade, the machines that have it will need >> upgrading, and the machines that don''t do not. I dont want to install >> it on all the > > The way I deal with this is to have apt-cron download but not install > packages for upgrade overnight. Once I''m happy that the upgrade is OK, > I use the Apt recipe on the wiki to initiate the upgrade on all the > various machines. Because it''s not downloading new packages, but merely > installing those that have already been downloaded, I can be sure that > no little surprises creep in.I feel like the point of my post has either been missed, probably because it wasn''t clear enough. Mine isn''t a problem of knowing when upgrades are necessary, nor how to do those upgrades. Mine is a problem of how to manage, within puppet, these upgrades on a set of heterogenously configured servers within an environment. Everything is running Debian stable, so thats not an issue, the issue is I dont have libperl5.8 installed on *all* machines, but I do have it installed on *some* machines. I''ve never installed that package explicitly, but it was required as a dependency for another package that I did explicitly install. Now, when there is a security upgrade for that package (as there is now), I am faced with a few options: 1. clusherssh to the machines that have this installed and do the upgrade 2. Install this version of the package on all the machines 3. Add the package and the desired version to my site.pp in the node definition for the machines that have the package installed 4. for every security upgrade, download the package, test it, then upload it to a local repository that is pinned so my machines can apt-get upgrade from it 5. determine if the package is installed, and if so, upgrade to the version specified 1-3 are gross, 4 seems like a lot of overhead. 5 is what I am wanting to do with puppet, unless someone can come up with a better way that doesn''t involve a lot of overhead or other silly things. Micah
On Wed, Nov 07, 2007 at 09:35:44PM +0000, micah wrote:> 5. determine if the package is installed, and if so, upgrade to the > version specified > > 1-3 are gross, 4 seems like a lot of overhead. 5 is what I am wanting to > do with puppet, unless someone can come up with a better way that doesn''t > involve a lot of overhead or other silly things.I''m really failing to understand what the issue is here. apt-get upgrade only upgrades packages you have installed. If libperl isn''t installed then apt-get upgrade won''t install it. cron-apt deals with the ''version specified'' bit by making sure you have the appropriate files on disk, and as you''re not running apt-get update new packages won''t be fetched. -- Sam Bashton - Bashton Ltd, Manchester, England Linux Consultancy / VOIP Telephony / High Availability Systems www.bashton.com - 0161-424-9600 _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Wed, 07 Nov 2007 21:42:36 +0000, Sam Bashton wrote:> On Wed, Nov 07, 2007 at 09:35:44PM +0000, micah wrote: >> 5. determine if the package is installed, and if so, upgrade to the >> version specified >> >> 1-3 are gross, 4 seems like a lot of overhead. 5 is what I am wanting >> to do with puppet, unless someone can come up with a better way that >> doesn''t involve a lot of overhead or other silly things. > > I''m really failing to understand what the issue is here. apt-get > upgrade only upgrades packages you have installed. If libperl isn''t > installed then apt-get upgrade won''t install it. cron-apt deals with the > ''version specified'' bit by making sure you have the appropriate files on > disk, and as you''re not running apt-get update new packages won''t be > fetched.I want to manage my upgrades, not let them be upgraded by apt-get upgrade automatically. Using apt-get upgrade is an all-or-none scenario. If there are upgrades available for four packages on a node, and one of those packages is going to cause a service outage, I do not want to run apt-get upgrade on that system. I likely will want to upgrade the other three packages as soon as possible, and then schedule a service interruption to do what is needed to upgrade the fourth. That service interruption may take a week or more, depending on the service and how critical the exposed vulnerability is to my environment. Additionally, If I use apt-get upgrade, I either have to run it on all my nodes all at once (not fun), or I insert it manually into the nodes that I want it to be run, when I want it to be run. The latter still leaves me with the all-or-none upgrade scenario.
micah <micah@riseup.net> writes:> I have no problem knowing *when* I need to upgrade packages on the > various systems. There are a number of tools to provide me with this > information (such as apticron), that is not the problem I am trying to > solve. Its more of how to manage the package upgrades, once I know this > information.Well, it''s not horribly helpful without a Kerberos infrastructure, but we treat all servers and applications as services that export an API. We use remctl (http://www.eyrie.org/~eagle/software/remctl/) as the protocol, but you can use anything with sufficient authentication properties. Every service gets a remctl interface that allows appropriate remote actions. So, for example, all of our servers export a set of puppet commands that let us stop, start, lock, unlock, or run in one-shot mode, among other things. And similarly, all of our Debian servers export an aptitude interface that lets us remotely run aptitude commands, or use a central trusted host to kick off aptitude commands on a bunch of servers. I guess I can kind of see a role for Puppet in deploying security fixes if I squint, but it really isn''t a natural fit for how Puppet thinks of packages. -- Russ Allbery (rra@stanford.edu) <http://www.eyrie.org/~eagle/>
On Wed, Nov 07, 2007 at 11:03:01PM +0000, micah wrote:> I want to manage my upgrades, not let them be upgraded by apt-get upgrade > automatically. Using apt-get upgrade is an all-or-none scenario. If there > are upgrades available for four packages on a node, and one of those > packages is going to cause a service outage, I do not want to run apt-get > upgrade on that system. I likely will want to upgrade the other three > packages as soon as possible, and then schedule a service interruption to > do what is needed to upgrade the fourth. That service interruption may > take a week or more, depending on the service and how critical the > exposed vulnerability is to my environment.OK, I''m with you now - you''d thrown me with all your talk of libperl. For me this isn''t an issue, as all services are replicated on multiple machines. If, for example, Apache is stopped for an upgrade on one node, the load balancer simply stops sending requests to that particular node until Apache is started again. Have you thought about having puppet put ''problematic'' packages (ie, those likely to cause service interuptions) on hold? Once you''re ready to deploy these packages, you can do an apt-get --ignore-hold -y dist-upgrade. This could be either done manually or as per the ''update_initiator'' solution under the Debian Recipies section of the wiki. -- Sam Bashton - Bashton Ltd, Manchester, England Linux Consultancy / VOIP Telephony / High Availability Systems www.bashton.com - 0161-424-9600 _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On 11/8/07, micah <micah@riseup.net> wrote:> > I want to manage my upgrades, not let them be upgraded by apt-get upgrade > automatically.Not sure how far this project is, but a while ago I was attended to http://cetico.org/nwu (network wide updates). You might want to look into it. It''s alpha though and the developer seems dormant. -- Gegroet, Tim _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Thu, 08 Nov 2007 11:46:54 +0100, Tim Stoop wrote:> On 11/8/07, micah <micah@riseup.net> wrote: >> >> I want to manage my upgrades, not let them be upgraded by apt-get >> upgrade automatically. > > > Not sure how far this project is, but a while ago I was attended to > http://cetico.org/nwu (network wide updates). You might want to look > into it. It''s alpha though and the developer seems dormant.Amusing because the author writes on that page: "My interests have changed a bit since I started to work on it, and in any case nobody seemed to be interested. I suppose people just prefer puppet, cfengine etc. They are much more complex and difficult to setup, but do the job. Well, who knows."