Puppet users - Jan 2011 - Using puppet to redeploy staging app

Hi,

I''m building a staging environement and I would like it to be able to
redeploy automatically, after each commit - or every hour, some
applications. Those applications are right now deployed and set up by
puppet. I wonder if I should use puppet to redeploy those applications after
each commit, and if I should how trigger, on the puppet, a full
redeployement of the application ?

-- 
Romain PELISSE,
*"The trouble with having an open mind, of course, is that people will
insist on coming along and trying to put things in it" -- Terry Pratchett*
http://belaran.eu/wordpress/belaran

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Fri, Jan 14, 2011 at 13:41, Romain Pelisse <belaran@gmail.com> wrote:
> I''m building a staging environement and I would like it to be able
to
> redeploy automatically, after each commit - or every hour, some
> applications. Those applications are right now deployed and set up by
> puppet. I wonder if I should use puppet to redeploy those applications
after
> each commit, and if I should how trigger, on the puppet, a full
> redeployement of the application ?
Personally, I would use some sort of CI solution like Hudson that
would be triggered on commit.  As part of the configured project you
could have a full test suite run and if (and only if) it was entirely
successful build new OS packages and push that artefact up to your
repository.  Puppet can then be set to ''latest'' for those
packages and
trigger the redeployment based on that change.

That gives a much more robust, repeatable, and automated deployment
solution that ensures you push only if things are work.  However, if
you don''t care so much you could just use the
''puppetrun'' tool to
trigger an agent run on the staging machines, and use whatever
mechanism you prefer to have puppet install the latest version.

Triggering puppetrun should be easy enough to automate based on
whatever post-commit or post-push hooks your VCS supports.

Regards,
    Daniel
-- 
✉ Daniel Pittman <daniel@rimspace.net>
⌨ daniel@rimspace.net (XMPP)
☎ +1 503 893 2285
♻ made with 100 percent post-consumer electrons

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Jan 14, 1:41 pm, Romain Pelisse <bela...@gmail.com>
wrote:
> Hi,
>
> I''m building a staging environement and I would like it to be able
to
> redeploy automatically, after each commit - or every hour, some
> applications. Those applications are right now deployed and set up by
> puppet. I wonder if I should use puppet to redeploy those applications
after
> each commit, and if I should how trigger, on the puppet, a full
> redeployement of the application ?
>
It depends on how you deploy your applications.

In the case of packaged applications I do the following after tests
pass in Hudson:

1) using extconf to specify versions and overwrite the package version
of the file with a script
2) use mcollective to go out and run apt-get update and install a
specific version
3) use mcollective to audit the package versions and send a report

I use the extconf file because I do the same for production and want
to pin versions.
You could just set your stage to latest and run an mcollective client
to just kick off your install.

-c

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

>
> Personally, I would use some sort of CI solution like Hudson that
> would be triggered on commit.  As part of the configured project you
> could have a full test suite run and if (and only if) it was entirely
> successful build new OS packages and push that artefact up to your
> repository.

That''s exactly what we are doing (I should have perhaps be more clear)

> Puppet can then be set to ''latest'' for those packages and
> trigger the redeployment based on that change.
>
This is exactly what I want to do, however those application are not package
as rpm or deb but simply zipfile. Indeed, the applications are Java based
application runned by daemontools. We simply unzip them and let daemontools
start them.

"In lieu of" Yum or Apt, we use Ivy. But even if we were to use
Yum/Apt, and
set upgradeable to the appropriate package, I don''t think (or I
didn''t
understand something) that it would trigger the puppet to upgrade
automatically, as soon as the package is made available by the CI tool (in
our case Hudson).

... or I miss something ?

> That gives a much more robust, repeatable, and automated deployment
> solution that ensures you push only if things are work.  However, if
> you don''t care so much you could just use the
''puppetrun'' tool to
> trigger an agent run on the staging machines, and use whatever
> mechanism you prefer to have puppet install the latest version.
>
> Triggering puppetrun should be easy enough to automate based on
> whatever post-commit or post-push hooks your VCS supports.
>
At first glance, sounds that puppetrun is probably a better option. I have
to look into it (never really used it).

>
> Regards,
>    Daniel
> --
> ✉ Daniel Pittman <daniel@rimspace.net>
> ⌨ daniel@rimspace.net (XMPP)
> ☎ +1 503 893 2285
> ♻ made with 100 percent post-consumer electrons
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>

-- 
Romain PELISSE,
*"The trouble with having an open mind, of course, is that people will
insist on coming along and trying to put things in it" -- Terry Pratchett*
http://belaran.eu/wordpress/belaran

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

>
> It depends on how you deploy your applications.
>
> In the case of packaged applications I do the following after tests
> pass in Hudson:
>
> 1) using extconf to specify versions and overwrite the package version
> of the file with a script
> 2) use mcollective to go out and run apt-get update and install a
> specific version
> 3) use mcollective to audit the package versions and send a report
>
> I use the extconf file because I do the same for production and want
> to pin versions.
> You could just set your stage to latest and run an mcollective client
> to just kick off your install.

Thanks for your answer, however, I do not use package application - which
would be, in this regards, far more easy to handle.

-- 
Romain PELISSE,
*"The trouble with having an open mind, of course, is that people will
insist on coming along and trying to put things in it" -- Terry Pratchett*
http://belaran.eu/wordpress/belaran

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Jan 14, 2:27 pm, Romain Pelisse <bela...@gmail.com>
wrote:
> > It depends on how you deploy your applications.
>
> > In the case of packaged applications I do the following after tests
> > pass in Hudson:
>
> > 1) using extconf to specify versions and overwrite the package version
> > of the file with a script
> > 2) use mcollective to go out and run apt-get update and install a
> > specific version
> > 3) use mcollective to audit the package versions and send a report
>
> > I use the extconf file because I do the same for production and want
> > to pin versions.
> > You could just set your stage to latest and run an mcollective client
> > to just kick off your install.
>
> Thanks for your answer, however, I do not use package application - which
> would be, in this regards, far more easy to handle.
How are you versioning/distributing your software?

We also do something similar with git for interpreted languages.

-c

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

How are you versioning/distributing your software?
>
The commits go in github and are polled regularly by our Hudson (CI server)
which builds our code (Scala/Java code and also some Javascript source
code), run the tests and package all of that either in jar or zip files.
Hudson deploys everything into a "maven repository" (Nexus).

On puppets, as we don''t want to install a build tool as maven, we
simply use
Ivy - as a simple executable, to fetch the version from Nexus.

Even if this all process sounds very "java" style, it could be applied
to
any languages (even if all the tools are actually Java based).

> We also do something similar with git for interpreted languages.

Sounds interesting :)

Could tell us more ?

-- 
Romain PELISSE,
*"The trouble with having an open mind, of course, is that people will
insist on coming along and trying to put things in it" -- Terry Pratchett*
http://belaran.eu/wordpress/belaran

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Fri, Jan 14, 2011 at 14:26, Romain Pelisse <belaran@gmail.com> wrote:
>> Personally, I would use some sort of CI solution like Hudson that
>> would be triggered on commit.  As part of the configured project you
>> could have a full test suite run and if (and only if) it was entirely
>> successful build new OS packages and push that artefact up to your
>> repository.
>
> That''s exactly what we are doing (I should have perhaps be more
clear)
*nod*
>> Puppet can then be set to ''latest'' for those packages
and
>> trigger the redeployment based on that change.
>
> This is exactly what I want to do, however those application are not
package
> as rpm or deb but simply zipfile. Indeed, the applications are Java based
> application runned by daemontools. We simply unzip them and let daemontools
> start them.
...so you need to take manual action to have puppet notice that the
version has changed.  One of the advantages of using something like an
RPM is that someone else already wrote all the infrastructure to make
that happen and all. :)

OTOH, if you just drop that zip into place where a puppet file server
can see it, and use a ''file'' resource with a source of
''puppet://ci-server.local/application.zip'', then
''notify'' to an exec
that unpacks and installs the zip file you can get the right thing
done...
> "In lieu of" Yum or Apt, we use Ivy. But even if we were to use
Yum/Apt, and
> set upgradeable to the appropriate package, I don''t think (or I
didn''t
> understand something) that it would trigger the puppet to upgrade
> automatically, as soon as the package is made available by the CI tool (in
> our case Hudson).
> ... or I miss something ?
Sorry, I was assuming that you already have puppet running routinely
on the staging machines.  If you need something faster than your
normal checkin interval...
> At first glance, sounds that puppetrun is probably a better option. I have
> to look into it (never really used it).
...then puppetrun or mcollective would be the right tool for the job.
(...or any other tool to run puppet agent on those machines, of
course. :)

My assumptions were showing there, too, where I didn''t make what I was
envisioning clear enough.

Regards,
    Daniel
-- 
✉ Daniel Pittman <daniel@rimspace.net>
⌨ daniel@rimspace.net (XMPP)
☎ +1 503 893 2285
♻ made with 100 percent post-consumer electrons

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

To manage application deployments I''ve written this Puppi module:
https://github.com/example42/puppet-modules/tree/master/puppi
it''s been designed originally to configure deployments procedures via 
puppet  but it has
to be run directly from the target node (until I''ll write a mcollective
agent) .
So via Puppet you can manage the whole configuration, but not the execution 
itself.

In the project dir there are some examples of deploy procedures that can 
adapt to different cases,
(incidentally puppi::project::maven is intended to pull from a Nexus 
repository (but it''s currenly under redesign and it works just for wars
deploys)
but these can be  totally customized or created from scratch using using 
custom scripts.

The advantages I see with this Puppi are:
- A sort of bridge between Puppet and the OS for managing "shot"
operations.
- Standard syntax  to manage deploys (and rollbacks) for different cases: 
you always have to write:
puppi deploy <name>
- Once configured your deployment templates it''s very quick and easy to
setup deploy procedures on different servers with Puppet
- Since the puppi command just executes, in a given seguence, a serie of 
commands, these can be totally customized, existing scritps can be adapted 
and different scripting languages used.
- The "puppi check" command to see if on your node everything is
running
fine is incredibly handy (but for having this to work in an automated way 
you need the Example42 monitor classes)
- The notification of the deploy result is customizable. For the moment it 
just sends and email, but you can do whatever you may want to do with a 
custom script.

In the future I''d like to integrate Puppi with mcollective and provide
a web
frontend, at least for reporting.
But works are in progresss and various things will change.

Al

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

@Daniel Pittman : don''t worry about I will call the
"assumption" - in a ml,
one has always to compromise between describe his all situation (hence, 20
pages mail) or do something rather shorter but clearly spotty. Your answers
were quite helpful anyway

@lab42: Your Puppi module sounds quite interesting indeed. I guess I could
probably reduce the amount of "junk" in my current puppet
configuration
using it. I''ll look into it in the next days.

FYI: As I need to have some kind of "staging auto deploy" for
tomorrow, I
will just do a crontab job that erase the deployed services every hour.
Puppet will notice that (hopefully :) ) and this will trigger a
full redeployment. This is far from behind perfect, but this will allow me
to have "something working" for the time being. I''ll explore
Puppi just
after that.

On 15 January 2011 15:35, Al @ Lab42 <lab42.it@gmail.com> wrote:
> To manage application deployments I''ve written this Puppi module:
> https://github.com/example42/puppet-modules/tree/master/puppi
> it''s been designed originally to configure deployments procedures
via
> puppet  but it has
> to be run directly from the target node (until I''ll write a
mcollective
> agent) .
> So via Puppet you can manage the whole configuration, but not the execution
> itself.
>
> In the project dir there are some examples of deploy procedures that can
> adapt to different cases,
> (incidentally puppi::project::maven is intended to pull from a Nexus
> repository (but it''s currenly under redesign and it works just for
wars
> deploys)
> but these can be  totally customized or created from scratch using using
> custom scripts.
>
> The advantages I see with this Puppi are:
> - A sort of bridge between Puppet and the OS for managing "shot"
> operations.
> - Standard syntax  to manage deploys (and rollbacks) for different cases:
> you always have to write:
> puppi deploy <name>
> - Once configured your deployment templates it''s very quick and
easy to
> setup deploy procedures on different servers with Puppet
> - Since the puppi command just executes, in a given seguence, a serie of
> commands, these can be totally customized, existing scritps can be adapted
> and different scripting languages used.
> - The "puppi check" command to see if on your node everything is
running
> fine is incredibly handy (but for having this to work in an automated way
> you need the Example42 monitor classes)
> - The notification of the deploy result is customizable. For the moment it
> just sends and email, but you can do whatever you may want to do with a
> custom script.
>
> In the future I''d like to integrate Puppi with mcollective and
provide a
> web frontend, at least for reporting.
> But works are in progresss and various things will change.
>
> Al
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>


-- 
Romain PELISSE,
*"The trouble with having an open mind, of course, is that people will
insist on coming along and trying to put things in it" -- Terry Pratchett*
http://belaran.eu/wordpress/belaran

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Monday, January 17, 2011 10:33:33 AM UTC+1, Romain PELISSE
wrote:
>
>
> @lab42: Your Puppi module sounds quite interesting indeed. I guess I could 
> probably reduce the amount of "junk" in my current puppet
configuration
> using it. I''ll look into it in the next days.
>
Sweet. If you have any questions about it don''t hesitate to ask. 
It'' currently under active development and things might be tweaked and 
refined.
There are also few examples so maybe it''s not clear how to use it.
For example you can do things like this (these are real life exaples with 
some names changed):
           puppi::project::war { "myapp":
               source           => 
"http://deploy.example42.com/myapp/myapp.war",
               user             => "myappuser",
               init_script      => "tomcat-myapp",
               deploy_root      => "/store/tomcat/myapp/webapps",
               report_email     => "deploys@example42.com",
               enable           => "true",
               disable_services    => "monit puppet apache2",
           }

or (the maven project layout to get stuff from a Nexus repo will be 
rewritten)
    puppi::project::maven { "myapp":
        source           => 
"http://nexus.${domain}/nexus/it/example42/myapp/",
        user             => "${apache::params::username}",
        deploy_root      => "/store/tomcat/myapp/webapps",
        init_script      => "tomcat-myapp",
        firewall_src_ip  => $type ? {
            dr   => "192.168.50.1/30",
            prod => "192.168.38.1/30",
        },
        report_email     => "deploys@example42.com",
        enable           => "true",
    }

    puppi::project::tar { "myapp":
        source           => 
"http://release.example42.com/deploy/myapp/release.tgz",
        init_source      => 
"ssh://deploy@debian.${domain}/var/www/deploy/initdir/myapp/",
        init_script      => "apache",
        deploy_root      => "/store/www/myapp/",
        report_email     => "deploys@example42.com",
        enable           => "true",
    }

Note finally, that during deploys puppi makes pre and post local checks 
which are autopopulated if you use the example42 monitor classes. Since you 
won''t probably have them you might find empty checks (or an error,
haven''t
tried this condition). 

Feedbacks welcomed

Alessandro

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.