Has anyone done rollbacks with puppet? The story goes that last night I pushed a change out, which intentionally changed the way certain services where managed. The change involved pushing out a few files, cronjobs, etc., which was okay until it was discovered there was a bug with the service. My configurations are kept in subversion and rolling back to a previous configuration would have been easy enough however I''d want to take back all the files/cronjobs I pushed. This is also pretty easy to do for the most part, with setting files and cron-jobs to have the flag: "ensure => absent". I ended up fixing most things with puppet and a few by hand, but it would be nice to have a feature that allows for a roll back (of some sort). Has any work been done in that area? Maybe the puppetmaster tags something in: /var/lib/puppet/state/tag/current -> YYYYMMDDHHMMSS.newer which is created when configurations are compiled for a particular client /var/lib/puppet/state/tag/previous -> YYYYMMDDHHMMSS.older Which is the previous configuration. Then we could say something like: puppetrb --tag <tag>@YYYYMMDDHHMMSS.older Once that run is done, current is renamed to previous. I dunno, just thinking out loud here and curious more than anything. Cheers, Ryan
Ryan, I agree that this kind of functionality would be great. I have ended up doing exactly what you did with "ensure => absent" for when invalid files have been rolled out, which admittedly says more about my testing procedures than anything else, but it would still be tremendously useful. Considering the filebucket on the server, I guess you could technically roll back anything except for "exec" statements. On 23/01/2008, Ryan Dooley <rd@powerset.com> wrote:> Has anyone done rollbacks with puppet? > > The story goes that last night I pushed a change out, which > intentionally changed the way certain services where managed. The > change involved pushing out a few files, cronjobs, etc., which was okay > until it was discovered there was a bug with the service. > > My configurations are kept in subversion and rolling back to a previous > configuration would have been easy enough however I''d want to take back > all the files/cronjobs I pushed. This is also pretty easy to do for the > most part, with setting files and cron-jobs to have the flag: "ensure => > absent". > > I ended up fixing most things with puppet and a few by hand, but it > would be nice to have a feature that allows for a roll back (of some > sort). Has any work been done in that area? Maybe the puppetmaster > tags something in: > > /var/lib/puppet/state/tag/current -> YYYYMMDDHHMMSS.newer > > which is created when configurations are compiled for a particular client > > /var/lib/puppet/state/tag/previous -> YYYYMMDDHHMMSS.older > > Which is the previous configuration. > > Then we could say something like: > > puppetrb --tag <tag>@YYYYMMDDHHMMSS.older > > Once that run is done, current is renamed to previous. > > I dunno, just thinking out loud here and curious more than anything. > > Cheers, > Ryan > > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users >-- /peter
Ryan, I''ve been generally considering it a best practice to create anti-classes along with my normal classes. The normal class ensures that things are going while the anti-class does all the work necessary to remove the effects of the class. It''s double the effort (sometimes more) but saves a bunch of pain when things go wrong. T On Jan 22, 2008 6:32 PM, Ryan Dooley <rd@powerset.com> wrote:> Has anyone done rollbacks with puppet? > > The story goes that last night I pushed a change out, which > intentionally changed the way certain services where managed. The > change involved pushing out a few files, cronjobs, etc., which was okay > until it was discovered there was a bug with the service. > > My configurations are kept in subversion and rolling back to a previous > configuration would have been easy enough however I''d want to take back > all the files/cronjobs I pushed. This is also pretty easy to do for the > most part, with setting files and cron-jobs to have the flag: "ensure => > absent". > > I ended up fixing most things with puppet and a few by hand, but it > would be nice to have a feature that allows for a roll back (of some > sort). Has any work been done in that area? Maybe the puppetmaster > tags something in: > > /var/lib/puppet/state/tag/current -> YYYYMMDDHHMMSS.newer > > which is created when configurations are compiled for a particular client > > /var/lib/puppet/state/tag/previous -> YYYYMMDDHHMMSS.older > > Which is the previous configuration. > > Then we could say something like: > > puppetrb --tag <tag>@YYYYMMDDHHMMSS.older > > Once that run is done, current is renamed to previous. > > I dunno, just thinking out loud here and curious more than anything. > > Cheers, > Ryan > > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users >
Trevor, I like that idea, but how do you effectively test the anti-classes? Roll forward and backwards of all classes before you deploy? On 23/01/2008, Trevor Vaughan <peiriannydd@gmail.com> wrote:> Ryan, > > I''ve been generally considering it a best practice to create > anti-classes along with my normal classes. > > The normal class ensures that things are going while the anti-class > does all the work necessary to remove the effects of the class. > > It''s double the effort (sometimes more) but saves a bunch of pain when > things go wrong. > > T > > On Jan 22, 2008 6:32 PM, Ryan Dooley <rd@powerset.com> wrote: > > Has anyone done rollbacks with puppet? > > > > The story goes that last night I pushed a change out, which > > intentionally changed the way certain services where managed. The > > change involved pushing out a few files, cronjobs, etc., which was okay > > until it was discovered there was a bug with the service. > > > > My configurations are kept in subversion and rolling back to a previous > > configuration would have been easy enough however I''d want to take back > > all the files/cronjobs I pushed. This is also pretty easy to do for the > > most part, with setting files and cron-jobs to have the flag: "ensure => > > absent". > > > > I ended up fixing most things with puppet and a few by hand, but it > > would be nice to have a feature that allows for a roll back (of some > > sort). Has any work been done in that area? Maybe the puppetmaster > > tags something in: > > > > /var/lib/puppet/state/tag/current -> YYYYMMDDHHMMSS.newer > > > > which is created when configurations are compiled for a particular client > > > > /var/lib/puppet/state/tag/previous -> YYYYMMDDHHMMSS.older > > > > Which is the previous configuration. > > > > Then we could say something like: > > > > puppetrb --tag <tag>@YYYYMMDDHHMMSS.older > > > > Once that run is done, current is renamed to previous. > > > > I dunno, just thinking out loud here and curious more than anything. > > > > Cheers, > > Ryan > > > > _______________________________________________ > > Puppet-users mailing list > > Puppet-users@madstop.com > > https://mail.madstop.com/mailman/listinfo/puppet-users > > > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users >-- /peter
I kinda like that idea. I wonder if I could automate the creation of the anti-class.... hmmmm. Cheers, Ryan ________________________________________ From: puppet-users-bounces@madstop.com [puppet-users-bounces@madstop.com] On Behalf Of Trevor Vaughan [peiriannydd@gmail.com] Sent: Wednesday, January 23, 2008 6:10 AM To: Puppet User Discussion Subject: Re: [Puppet-users] rollbacks Ryan, I''ve been generally considering it a best practice to create anti-classes along with my normal classes. The normal class ensures that things are going while the anti-class does all the work necessary to remove the effects of the class. It''s double the effort (sometimes more) but saves a bunch of pain when things go wrong. T
For me, it''d be easy enough to do with a Xen instance that I
don''t care about. One to test the push of the proposed changed, once
validated that the change was good, then switch the class to the anti-class,
puppetrun... and validate that the change was undone.
I''m think that you could always whip up a script to distribute with the
anti-class that ran a command to undo/reset any exec {} statement that was run
by the original class.
Cheers,
Ryan
________________________________________
From: puppet-users-bounces@madstop.com [puppet-users-bounces@madstop.com] On
Behalf Of Peter Hoeg [peter@hoeg.com]
Sent: Wednesday, January 23, 2008 6:19 AM
To: Puppet User Discussion
Subject: Re: [Puppet-users] rollbacks
Trevor, I like that idea, but how do you effectively test the
anti-classes? Roll forward and backwards of all classes before you
deploy?
Ryan Dooley <rd@powerset.com> writes:> Has anyone done rollbacks with puppet?Yes. We use subversion to tag releases for some clusters. Scripts on the server and on the clients controls what is available from the puppet master, and the file server. A fact called "role" on each server returns the role of the server, for instance a string like "prod", "stage", "test" or "devel". All file sources for the server group "acme" looks like like puppet:///acme/$role/sofware/configfile.cfg, the same goes for templates. A script run on the client sets or gets the role. A script run on the puppetmaster checks out tags, branches or trunk to the role name under /var/lib/puppet/svn/acme/: /var/lib/puppet/svn/acme/prod may refer to the svn tag acme/2008-01-24-001, and /var/lib/puppet/svn/acme/test may refer to the svn trunk. To test a new setup on a cluster client, we set its role to "test", link the role "test" to an SVN branch or tag different from the role for "prod", and run puppetd -t. If something''s not right, revert the role of the server back to "prod". Note that this is used for files served and generated only, not for manifests. We serve those out of trunk. -- Stig Sandbeck Mathisen, Linpro
Ryan, I usually test things out by doing exactly that. First, I record the log output from adding the normal class to a fresh system. Next, I run the anti-class. Then, I re-run the normal class and see where my logged outputs differ. It''s still a bit manual, but generally fast enough to not worry about it too much. Trevor On Jan 23, 2008 9:19 AM, Peter Hoeg <peter@hoeg.com> wrote:> Trevor, I like that idea, but how do you effectively test the > anti-classes? Roll forward and backwards of all classes before you > deploy? > > > On 23/01/2008, Trevor Vaughan <peiriannydd@gmail.com> wrote: > > Ryan, > > > > I''ve been generally considering it a best practice to create > > anti-classes along with my normal classes. > > > > The normal class ensures that things are going while the anti-class > > does all the work necessary to remove the effects of the class. > > > > It''s double the effort (sometimes more) but saves a bunch of pain when > > things go wrong. > > > > T > > > > On Jan 22, 2008 6:32 PM, Ryan Dooley <rd@powerset.com> wrote: > > > Has anyone done rollbacks with puppet? > > > > > > The story goes that last night I pushed a change out, which > > > intentionally changed the way certain services where managed. The > > > change involved pushing out a few files, cronjobs, etc., which was okay > > > until it was discovered there was a bug with the service. > > > > > > My configurations are kept in subversion and rolling back to a previous > > > configuration would have been easy enough however I''d want to take back > > > all the files/cronjobs I pushed. This is also pretty easy to do for the > > > most part, with setting files and cron-jobs to have the flag: "ensure => > > > absent". > > > > > > I ended up fixing most things with puppet and a few by hand, but it > > > would be nice to have a feature that allows for a roll back (of some > > > sort). Has any work been done in that area? Maybe the puppetmaster > > > tags something in: > > > > > > /var/lib/puppet/state/tag/current -> YYYYMMDDHHMMSS.newer > > > > > > which is created when configurations are compiled for a particular client > > > > > > /var/lib/puppet/state/tag/previous -> YYYYMMDDHHMMSS.older > > > > > > Which is the previous configuration. > > > > > > Then we could say something like: > > > > > > puppetrb --tag <tag>@YYYYMMDDHHMMSS.older > > > > > > Once that run is done, current is renamed to previous. > > > > > > I dunno, just thinking out loud here and curious more than anything. > > > > > > Cheers, > > > Ryan > > > > > > _______________________________________________ > > > Puppet-users mailing list > > > Puppet-users@madstop.com > > > https://mail.madstop.com/mailman/listinfo/puppet-users > > > > > _______________________________________________ > > Puppet-users mailing list > > Puppet-users@madstop.com > > https://mail.madstop.com/mailman/listinfo/puppet-users > > > > > -- > /peter > > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users >
Ryan,
I think that you can automate a great deal of creating anti-classes as
long as you don''t run into execs.
You might be able to automatically stub a lot of the anti class by
parsing the normal class.
Normal Class:
file { "name": ensure => present, .... }
exec { "name": do...stuff }
Anti-Class:
file { "name": ensure => absent }
exec { "stub": stubbed }
Also, you''re going to have to be REALLY careful on the dependencies.
I''m not sure if it wouldn''t just be easier to write them by
hand :-/.
Trevor
On Jan 23, 2008 11:31 AM, Ryan Dooley <rd@powerset.com>
wrote:> I kinda like that idea. I wonder if I could automate the creation of the
anti-class.... hmmmm.
>
> Cheers,
> Ryan
> ________________________________________
> From: puppet-users-bounces@madstop.com [puppet-users-bounces@madstop.com]
On Behalf Of Trevor Vaughan [peiriannydd@gmail.com]
> Sent: Wednesday, January 23, 2008 6:10 AM
> To: Puppet User Discussion
> Subject: Re: [Puppet-users] rollbacks
>
> Ryan,
>
> I''ve been generally considering it a best practice to create
> anti-classes along with my normal classes.
>
> The normal class ensures that things are going while the anti-class
> does all the work necessary to remove the effects of the class.
>
> It''s double the effort (sometimes more) but saves a bunch of pain
when
> things go wrong.
>
> T
>
>
> _______________________________________________
> Puppet-users mailing list
> Puppet-users@madstop.com
> https://mail.madstop.com/mailman/listinfo/puppet-users
>
<84E2AE771361E9419DD0EFBD31F09C4D4C8BBEFEB8@EXVMBX015-1.exch015.msoutlookonline.net> Reply-To: puppet-users@madstop.com I tried to send this message yesterday, but was rejected. On Wed, Jan 23, 2008 at 08:31:03AM -0800, Ryan Dooley wrote:> I kinda like that idea. I wonder if I could automate the creation of the > anti-class.... hmmmm.Would it be possible that puppet know how to reverse each resource? For example stopping instead of starting a service. And from the class and the anti-resources a anti-class could be "generated". It would also be nice if the client get to know how the configuration differs between each run. In the case some node mistakenly includes a class starting a service, the admin could remove the include. Then let the client pull and puppet would determine the anti-class and anti-configure the resource. Fantacy? -- Martin Eriksen
On Jan 25, 2008, at 1:53 AM, Martin Børstad Eriksen wrote:> <84E2AE771361E9419DD0EFBD31F09C4D4C8BBEFEB8@EXVMBX015-1.exch015.msoutlookonline.net > > > Reply-To: puppet-users@madstop.com > > I tried to send this message yesterday, but was rejected. > > On Wed, Jan 23, 2008 at 08:31:03AM -0800, Ryan Dooley wrote: >> I kinda like that idea. I wonder if I could automate the creation >> of the >> anti-class.... hmmmm. > > Would it be possible that puppet know how to reverse each resource? > For > example stopping instead of starting a service. And from the class > and the > anti-resources a anti-class could be "generated". It would also be > nice if > the client get to know how the configuration differs between each run. > In the case some node mistakenly includes a class starting a service, > the admin could remove the include. Then let the client pull and > puppet > would determine the anti-class and anti-configure the resource.I''ve been meaning to respond to this thread since the beginning, but client work plus travel has so far forbade it. Puppet supports rollbacks internally, but there''s currently no mechanism for using them. As mentioned by someone, you can''t do much with the execs, although we could add a ''reverse'' parameter or something to enable it. Puppet can take any change and reverse the change, so we could roll back a transaction. However, transactions are not recorded anywhere, so there''s no way to actually do so. As with so much of this stuff, the plumbing is done, but it won''t get actually supported unless someone cares enough about it to do it themselves or pay to have it done. And there''s still a good bit of design work left here, too, because of the usability aspects -- how do you decide when to roll back, what do you do about the server side configuration when rolling back (i.e., what if the configuration wants you to redo the work again). -- Love is a snowmobile racing across the tundra and then suddenly it flips over, pinning you underneath. At night, the ice weasels come. --Matt Groening --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Luke Kanies schrieb:> Puppet supports rollbacks internally, but there''s currently no > mechanism for using them. As mentioned by someone, you can''t do much > with the execs, although we could add a ''reverse'' parameter or > something to enable it. > > Puppet can take any change and reverse the change, so we could roll > back a transaction. > > However, transactions are not recorded anywhere, so there''s no way to > actually do so. > > As with so much of this stuff, the plumbing is done, but it won''t get > actually supported unless someone cares enough about it to do it > themselves or pay to have it done. And there''s still a good bit of > design work left here, too, because of the usability aspects -- how do > you decide when to roll back, what do you do about the server side > configuration when rolling back (i.e., what if the configuration wants > you to redo the work again).My question on the last point is, hoe many scenarios there are where you wouldn''t want to "just" use something like "exclude broken_class" (after the prior "include broken_class)? Regards, David
On Jan 25, 2008, at 6:04 PM, David Schmitt wrote:> My question on the last point is, hoe many scenarios there are where > you > wouldn''t want to "just" use something like "exclude > broken_class" (after > the prior "include broken_class)?In every case where you''ve got interconnected classes that you upgraded all at once. Really, though, I''m not sure; I just know that 1) there''s a good bit of work to do so we can record transactions and 2) there''s a lot more work to do designing and implementing an interface for controling the use of that transaction log. -- You''ve achieved success in your field when you don''t know whether what you''re doing is work or play. -- Warren Beatty --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com