Hi all I have a scenario in mind for MCollective and I was looking for some feedback The basic idea is this: 1) A code push to Git triggers a Jenkins build 2) A successful build triggers automatic deployment to an automated functional test environment 3) Functional tests run automatically and the results are reported etc This is just basic CI practice and 1) and 3) are sorted, the only issues are with 2) and the trigger for 3). The deployment job in Jenkins just copies files up to the Puppet master ready for deployment and service restarts etc etc, however with a basic puppet setup one has to wait for the clients to run to pickup the new changes and restart services etc and only then can Jenkins run the automated tests. So the plan is for Jenkins to trigger a run on all the required Puppet clients via MCollective (filtered by environment), synchronously wait for the clients to finish and then run the tests. This has another advantage in that we can back off the Puppet run interval (as important changes are triggered by MCollective) which gives the Puppet master a bit of breathing room, and the clients can just check-in every hour or so to check everything is in order. This also extends to deployments to other environments that have manual deployment triggers such as a QA environment. Even though the deployment is triggered by the QA team, the upload of the artifacts, running of puppet clients and notification of a successful deployment should all be automated. 1): Does this make sense as a pattern, if not what is the best practice for implementing this part of the CI and CD pipeline with puppet / jenkins / mcollective etc? 2): How would this work in practice. I can run "mco puppet runonce" which is fine, but if I filter by environment "mco puppet runonce --environment aft" I get the "Cannot specify any custom puppet options when the daemon is running" error. I can workaround this by stopping the daemons on every box, however this means that the clients no-longer check in periodically. Is there a fix for this or would you recommend turning them off anyway when using a tool such as MCollective and doing ALL configuration updates via MCollective triggers? 3) I need to do some more testing now, but just in case anyone knows off the top of their heads, will an MCollective runonce call be synchronous, i.e. will the mco call only finish once each client has reported that they have finished their runs? If it''s not, does anyone know of a good way to check for a finished deployment (some command line wizardry with mco puppet status / mco puppet summary, etc)? I have done my research with Google and read a fair amount about MCollective from various sources during the investigation and implementation, however if this is already documented or I''m missing something stupid I apologise in advance Thanks -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
----- Original Message -----> From: "Tom Poulton" <poulton.tm@gmail.com> > To: puppet-users@googlegroups.com > Sent: Monday, May 13, 2013 3:38:34 PM > Subject: [Puppet Users] MCollective deployment pattern > > Hi all > > I have a scenario in mind for MCollective and I was looking for > some feedback > > The basic idea is this: > 1) A code push to Git triggers a Jenkins build > 2) A successful build triggers automatic deployment to an automated > functional test environment > 3) Functional tests run automatically and the results are reported etc > > This is just basic CI practice and 1) and 3) are sorted, the only issues > are with 2) and the trigger for 3). The deployment job in Jenkins just > copies files up to the Puppet master ready for deployment and service > restarts etc etc, however with a basic puppet setup one has to wait for the > clients to run to pickup the new changes and restart services etc and only > then can Jenkins run the automated tests. > > So the plan is for Jenkins to trigger a run on all the required Puppet > clients via MCollective (filtered by environment), synchronously wait for > the clients to finish and then run the tests. This has another advantage in > that we can back off the Puppet run interval (as important changes are > triggered by MCollective) which gives the Puppet master a bit of breathing > room, and the clients can just check-in every hour or so to check > everything is in order. > > This also extends to deployments to other environments that have manual > deployment triggers such as a QA environment. Even though the deployment is > triggered by the QA team, the upload of the artifacts, running of puppet > clients and notification of a successful deployment should all be automated. > > 1): Does this make sense as a pattern, if not what is the best practice for > implementing this part of the CI and CD pipeline with puppet / jenkins / > mcollective etc? > > 2): How would this work in practice. I can run "mco puppet runonce" which > is fine, but if I filter by environment "mco puppet runonce --environment > aft" I get the "Cannot specify any custom puppet options when the daemon is > running" error. I can workaround this by stopping the daemons on every box, > however this means that the clients no-longer check in periodically. Is > there a fix for this or would you recommend turning them off anyway when > using a tool such as MCollective and doing ALL configuration updates via > MCollective triggers?--environment here is not a filter, you''re asking it to do a one-off run against a different puppet environment. If you want to filter use -W environment=foo> 3) I need to do some more testing now, but just in case anyone knows off > the top of their heads, will an MCollective runonce call be synchronous, > i.e. will the mco call only finish once each client has reported that they > have finished their runs? If it''s not, does anyone know of a good way to > check for a finished deployment (some command line wizardry with mco puppet > status / mco puppet summary, etc)?it is not, it triggers a run and returns - puppet runs can take any number of times and do any number of things including restart mcollective, so this makes doing a synch run a rather nasty prospect. We def need some better approach and have plans for that but right now you need to involve triggering runs and either inspecting them or querying your reporting infrastructure for status. I''d have loved to be able to influence the config version - or similar - so that you can tag a specific puppet run with some identifiable piece of information that you can later correlate against reporting for instance to be able to say all the nodes run configuration tagged foo, and periodically query the reporting infra for nodes successfully having run a configuration tagged foo but this is not posible today> > I have done my research with Google and read a fair amount about > MCollective from various sources during the investigation and > implementation, however if this is already documented or I''m missing > something stupid I apologise in advance > > Thanks > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Thanks for the quick reply The -W environment=foo tip is very useful and definitely solves one problem. I appreciate that the synchronous part could get pretty nasty, rather you than me :) In the meantime have you got any tips on the best (most reliable) way to check up on triggered runs, you mentioned inspecting them or querying my reporting infrastructure: to be honest I''m pretty familiar with puppet in terms of writing modules etc, but I haven''t really looked into the reporting or querying side of things at all, and I''m definitely a newcomer to MCollective, feel free to just dump a load of keywords and I can google around if that''s easier? We''re running Puppet Open Source but we do have Puppet Dashboard running if that offers any querying options against the DB or something? Thanks -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
----- Original Message -----> From: "Tom Poulton" <poulton.tm@gmail.com> > To: puppet-users@googlegroups.com > Sent: Monday, May 13, 2013 4:06:56 PM > Subject: Re: [Puppet Users] MCollective deployment pattern > > Thanks for the quick reply > > The -W environment=foo tip is very useful and definitely solves onesee http://docs.puppetlabs.com/mcollective/reference/basic/basic_cli_usage.html for the details on the cli, discovery etc, thats a little primer to get you going> problem. I appreciate that the synchronous part could get pretty nasty, > rather you than me :) In the meantime have you got any tips on the best > (most reliable) way to check up on triggered runs, you mentioned inspecting > them or querying my reporting infrastructure: to be honest I''m > pretty familiar with puppet in terms of writing modules etc, but I haven''t > really looked into the reporting or querying side of things at all, and > I''m definitely a newcomer to MCollective, feel free to just dump a load of > keywords and I can google around if that''s easier? > > We''re running Puppet Open Source but we do have Puppet Dashboard running if > that offers any querying options against the DB or something?well lets say your deployment updates some package, or you have a define like myapp{"acme": ensure => 1.2.3} and your upgrade is gong to do that, then this is something you can interrogate via mco to figure out the state of things. but that sux. if you know you dont manage mcollective and these deploy runs tend to have a reasonable short run time that doesnt fluctuate hugely, I''d just write a little agent that runs puppet in the foreground for you to do it synchronously there''s nothing inherently wrong with running puppet synchronously like this as long as you know you wont be restarting mco or doing stuff that fluctuate in times heavily. alternative you can use the config_version feature in puppet and set the current catalog version to lets say a git tag or svn revision and interrogate the number via the last_run_summary action on the puppet agent which will show you the catalog version of the most recent run or puppet reports etc, there''s an annoying amount of solutions to this problem and the right one depends on your environment unfortunately> > Thanks > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
I''ll have a play around in the next few days when I get a chance and report back I''m thinking a little foreground agent is probably the way to go for now as a "quick fix", and then I''ll work on something cleaner once we start phase 2 in a couple of months Thanks for the pointers, much appreciated -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Well I''ve been using an initial solution for the past month or two which seems to be working ok. I have a custom puppet function that reads the version of the artifact being deployed and I add that to the mcollective facts.yaml file at the same time that the artifact is deployed. I''ve also got a groovy script which I''ve put up on GitHub ( https://github.com/TomPoulton/mcollective-synchronous-puppet/blob/master/deployment_trigger.groovy) that runs as part of the Jenkins deployment job that finds the correct instances, triggers a puppet run, then waits until all the instances show the new version in their facts. It handles the situation where puppet is already running, and also has timeouts so the script doesn''t run forever. The only problem is that sometimes (although it''s started to get more frequent) the script reaches it''s timeout waiting for the new version to show in the facts, the timeout is pretty generous and is certainly longer than a puppet run and an mcollective restart would take, and the facts are being updated correctly. Does mcollective have a cache or something that would prevent the new values being returned to the mco client on the Jenins box? Thanks -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.