Puppet users - Mar 2010 - Using Puppet for application deployment

I''m using puppet to deploy new versions of our application to our
server instances.

I do this by having a custom puppet node classifier that talks to a
database that defines what version of an application is supposed to be
on a particular node:

parameters:
  application: "webapp",
  webapp_version: "0.5",
  webapp_config: "123",
  webapp_symlink: "0.4"

My puppet recipe then makes sure that webapp version 0.5 is installed
(via yum and rpms), makes sure the right versioned configuration files
are in place, and makes sure that the "current" symlink points at 0.4
so that we can roll to a different version by flipping a symlink. The
puppet recipe has a lot of "requires" to make sure that the upgrade of
the application is graceful and does things in the right order.

My upgrade then goes like this, all controlled via a custom web interface:

1) Insert a jobgroup for the upgrade and a job for each server
instance into a db
2) A job processor then takes the first group of machines to upgrade,
changes their webapp version to the new version, and runs puppet on
them
3) Lather, rinse, repeat step 2 until complete (roll the application
out to prevent downtime), although if there is a puppet failure, all
pending jobs get cancelled.

Given that there are so many ways to skin the same cat, I''m wondering
how others are doing their application deployments using Puppet. Or,
if you are using Puppet but do your app deployments via some other
mechanism, how do you do it, and why aren''t you using Puppet to do it?

Pete

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Pete Emerson wrote:
> I''m using puppet to deploy new versions of our application to our
> server instances.
>
> I do this by having a custom puppet node classifier that talks to a
> database that defines what version of an application is supposed to be
> on a particular node:
>
> parameters:
>   application: "webapp",
>   webapp_version: "0.5",
>   webapp_config: "123",
>   webapp_symlink: "0.4"
>
> My puppet recipe then makes sure that webapp version 0.5 is installed
> (via yum and rpms), makes sure the right versioned configuration files
> are in place, and makes sure that the "current" symlink points at
0.4
> so that we can roll to a different version by flipping a symlink. The
> puppet recipe has a lot of "requires" to make sure that the
upgrade of
> the application is graceful and does things in the right order.
>
> My upgrade then goes like this, all controlled via a custom web interface:
>
> 1) Insert a jobgroup for the upgrade and a job for each server
> instance into a db
> 2) A job processor then takes the first group of machines to upgrade,
> changes their webapp version to the new version, and runs puppet on
> them
> 3) Lather, rinse, repeat step 2 until complete (roll the application
> out to prevent downtime), although if there is a puppet failure, all
> pending jobs get cancelled.
>
> Given that there are so many ways to skin the same cat, I''m
wondering
> how others are doing their application deployments using Puppet. Or,
> if you are using Puppet but do your app deployments via some other
> mechanism, how do you do it, and why aren''t you using Puppet to do
it?
>
> Pete
>
>   
I use capistrano for application deployment.

-- 
Joe McDonagh
AIM: YoosingYoonickz
IRC: joe-mac on freenode
L''ennui est contre-révolutionnaire


-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

>
>
>> Given that there are so many ways to skin the same cat, I''m
wondering
>> how others are doing their application deployments using Puppet. Or,
>> if you are using Puppet but do your app deployments via some other
>> mechanism, how do you do it, and why aren''t you using Puppet
to do it?
>>
>> Pete
>>
>>
>>
> I use capistrano for application deployment.

I''ve managed to use both Puppet and Capistrano for deployment: Cap for
the
business code, and Puppet to ensure that dependencies are met.  Puppet was
called from Capistrano.

J.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mon, Mar 8, 2010 at 9:46 AM, Julian Simpson <simpsonjulian@gmail.com>
wrote:
>>>
>>> Given that there are so many ways to skin the same cat,
I''m wondering
>>> how others are doing their application deployments using Puppet.
Or,
>>> if you are using Puppet but do your app deployments via some other
>>> mechanism, how do you do it, and why aren''t you using
Puppet to do it?
>>>
>>> Pete
>>>
>>>
>>
>> I use capistrano for application deployment.
>
> I''ve managed to use both Puppet and Capistrano for deployment: Cap
for the
> business code, and Puppet to ensure that dependencies are met.  Puppet was
> called from Capistrano.
> J.
>

Can you share some examples of where your dividing line is and some of
the tasks that are hard to handle in Puppet today?

Ideally I''d like to figure out how to incorporate more of that into
puppet-core.    Bruce has been working on a version control type, that
could be used to check sources
out of version control and eliminate some packaging needs for hosted
apps, though I''m curious as to what other niches might not be covered.

--Michael

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

>
>
> Can you share some examples of where your dividing line is and some of
> the tasks that are hard to handle in Puppet today?
>
Here''s how it worked:
http://www.build-doctor.com/2009/12/09/dependency-hell/

Mainly, it was there to complement the lack of Rubygem dependencies, but it
meant that there was a way for developers to maintain crontabs.  We used
Passenger and Rails so there wasn''t much need to do anything else.  I
could
have checked that there were processes running but the hosting peeps already
had Monit.
>
> Ideally I''d like to figure out how to incorporate more of that
into
> puppet-core.    Bruce has been working on a version control type, that
> could be used to check sources
> out of version control and eliminate some packaging needs for hosted
> apps, though I''m curious as to what other niches might not be
covered.
>

I''ve considered this for about ten seconds, but how about supporting
some
deployment tools in Puppet?  So you could  write a recipe that deploys
middleware, application and other infrastructure for example.

J.

> --Michael
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>

-- 
Julian Simpson
Software Build and Deployment
http://www.build-doctor.com
http://twitter.com/builddoctor

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

>
> I''ve considered this for about ten seconds, but how about
supporting some
> deployment tools in Puppet?  So you could  write a recipe that deploys
> middleware, application and other infrastructure for example.
Personally, I''d rather figure out what doesn''t fit into the
model well
than force a lot of external dependencies on other tools someone would
need to grok.   The idea that Puppet automation and
''deployment'' are
different shouldn''t
be the case, IMHO.    That all being said, there''s nothing stopping
the creation of a Puppet type to run a deployment tool, I think that
if you did that, you''re doing the analog of an exec task, and
that''s
not so interesting.

I do agree though -- Puppet needs to have examples of how to construct
manifests for middleware, applications, etc... though I don''t think
that''s too hard to do today provided things lurk on a single server,
and addressing the multi-server case is
something we''ll eventually want to conquer.

I think there''s a huge advantage of having all of your deployment + OS
+ config modelled in the same system.

(Then again, I also believe in creating RPMs where possible so you can
have package dependencies, but that''s another story ... I know that
doesn''t fit into everyone''s workflow so we have to make it
easier to
deploy things both ways.  I think the VC
type will go a pretty good way towards that.)

--Michael

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Michael DeHaan wrote:
> On Mon, Mar 8, 2010 at 9:46 AM, Julian Simpson
<simpsonjulian@gmail.com> wrote:
>   
>>>> Given that there are so many ways to skin the same cat,
I''m wondering
>>>> how others are doing their application deployments using
Puppet. Or,
>>>> if you are using Puppet but do your app deployments via some
other
>>>> mechanism, how do you do it, and why aren''t you using
Puppet to do it?
>>>>
>>>> Pete
>>>>
>>>>
>>>>         
>>> I use capistrano for application deployment.
>>>       
>> I''ve managed to use both Puppet and Capistrano for deployment:
Cap for the
>> business code, and Puppet to ensure that dependencies are met.  Puppet
was
>> called from Capistrano.
>> J.
>>
>>     
>
>
> Can you share some examples of where your dividing line is and some of
> the tasks that are hard to handle in Puppet today?
>
> Ideally I''d like to figure out how to incorporate more of that
into
> puppet-core.    Bruce has been working on a version control type, that
> could be used to check sources
> out of version control and eliminate some packaging needs for hosted
> apps, though I''m curious as to what other niches might not be
covered.
>
> --Michael
>
>   
Example of what puppet can do best:

Configure a system with apache2, a basic security configuration, some 
enabled modules, a bunch of packages admins expect to have available 
(screen, irb, etc).

Example of what capistrano can do best:

A. Deploy your puppet manifests to your puppet server.
B. Deploy your web content out of svn onto your web server.

They''re two distinct areas of ''deployment''. Puppet is
more like, systems
deployment, cap is app or content deployment in my eyes.

-- 
Joe McDonagh
AIM: YoosingYoonickz
IRC: joe-mac on freenode
L''ennui est contre-révolutionnaire

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Michael DeHaan wrote:
> On Mon, Mar 8, 2010 at 9:46 AM, Julian Simpson
<simpsonjulian@gmail.com> wrote:
>   
>>>> Given that there are so many ways to skin the same cat,
I''m wondering
>>>> how others are doing their application deployments using
Puppet. Or,
>>>> if you are using Puppet but do your app deployments via some
other
>>>> mechanism, how do you do it, and why aren''t you using
Puppet to do it?
>>>>
>>>> Pete
>>>>
>>>>
>>>>         
>>> I use capistrano for application deployment.
>>>       
>> I''ve managed to use both Puppet and Capistrano for deployment:
Cap for the
>> business code, and Puppet to ensure that dependencies are met.  Puppet
was
>> called from Capistrano.
>> J.
>>
>>     
>
>
> Can you share some examples of where your dividing line is and some of
> the tasks that are hard to handle in Puppet today?
>
> Ideally I''d like to figure out how to incorporate more of that
into
> puppet-core.    Bruce has been working on a version control type, that
> could be used to check sources
> out of version control and eliminate some packaging needs for hosted
> apps, though I''m curious as to what other niches might not be
covered.
>
> --Michael
>
>   
Sorry for double e-mails and slightly OT, but cap is also kung-fu ninja 
awesome at ad-hoc command execution across multiple hosts. (Like your 
func). Except I think func may scale to a lot more hosts since the 
ssh-agent isn''t so good at a lot of parallel connections.

-- 
Joe McDonagh
AIM: YoosingYoonickz
IRC: joe-mac on freenode
L''ennui est contre-révolutionnaire

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

>
> Example of what capistrano can do best:
>
> A. Deploy your puppet manifests to your puppet server.
> B. Deploy your web content out of svn onto your web server.
>
> They''re two distinct areas of ''deployment''.
Puppet is more like, systems
> deployment, cap is app or content deployment in my eyes.
Yes, I want to change that observation.

For instance, there are pretty good examples of deploying rails apps
with Puppet, but it''s not well shared.   We similarly need to create
examples for other types of applications.

The Source Control Management type that Bruce is working on (should be
available soon?) covers your case B and should be tremendously helpful
there.

I believe the two should not necessarily be distinct and it''s very
good to have them in the same model so you can have dependencies
between them.

This all goes down the line of doing minimal kickstart + Puppet.

I don''t want Puppet to be like kickstart and reach a arbitrary
deployment line and stop -- it is too interesting for that.

--Michael

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

>
> > They''re two distinct areas of ''deployment''.
Puppet is more like, systems
> > deployment, cap is app or content deployment in my eyes.
>
> Yes, I want to change that observation.
>
>
I use Puppet for deploying my blog.  It''s a WordPress distro with a
bunch of
plugins, etc.  It was too slow.  I ended up inelegantly using rsync from
puppet to push out the distro instead, which was insanely fast.  Even that
didn''t come for free as I''m not using a Puppetmaster at the
moment, and
paths got annoying.  (I do accept that''s probably my issue, however).

Anyway, my point is that Puppet needs to give you a way to deploy a tree of
files, fast.  If the new type mentioned above starts to give you that, so
much the better: it gets tedious to push files to your version control
system, ensure that your puppetmaster is up to date, and then invoke puppet
on the node that you''re deploying to.

What''s appealing about some developer-style deploy tools is that you
can
push exactly what you''re working on to one or more nodes and test it. 
My
experiences with Puppet suggest that it''s not so convenient.  My 6
years as
a build and release manager tell me that it''s generally a poor idea to
use
two application deployment tools.  Hmmm.

Julian



J.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I was asking about this months ago and was pointed to
http://dev2ops.org/storage/downloads/FullyAutomatedProvisioning_Whitepaper.pdfas
an example and explanation of why Puppet was not "good" for
application
deployment, better at being part of a stack of apps to achieve it.

Not that I understand it much :-P. But in general, I really didnt see why
application deployment would be different for system deployment... except in
operational terms. Things like "ok, new version of jar files on a load
balanced enviroment, have to do one leg first, then the other, to ensure no
outage...", that are very much procedural while Puppet manifest are more
useful on a description of required software level. Cause you could do some
weird thing like using a different class or parameter or something to update
one leg first and then converging to the correct version, but at the default
30 minutes interval it would take you 1 hour to deploy, for example.

------------------------------

Jesús Couto F.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Interesting, I''ll read that paper, thanks.

For our setup the version of software is exposed in the
puppet_node_classifier which connects to a database. Additionally, the
version of the configuration files and which puppet environment to use
are available as well. To do an upgrade, I change the version in the
db for a particular host to whatever I want it to be and then force
puppet to run. No waiting for 30 minutes, I can''t not have control
over when the upgrade is going to happen. Rollback is an upgrade to
the previous version, and when rolling out to all of our production
boxes, our software modifies the database in slices and then verifies
that the upgrade was successful before proceeding to the next slice,
and if puppet fails to upgrade, all future boxes waiting to be
upgraded get canceled. I think it''s a pretty slick system.

Pete

On Mar 12, 2:46 am, Jesús Couto <jesus.co...@gmail.com>
wrote:
> I was asking about this months ago and was pointed
tohttp://dev2ops.org/storage/downloads/FullyAutomatedProvisioning_White...
> an example and explanation of why Puppet was not "good" for
> application
> deployment, better at being part of a stack of apps to achieve it.
>
> Not that I understand it much :-P. But in general, I really didnt see why
> application deployment would be different for system deployment... except
in
> operational terms. Things like "ok, new version of jar files on a load
> balanced enviroment, have to do one leg first, then the other, to ensure no
> outage...", that are very much procedural while Puppet manifest are
more
> useful on a description of required software level. Cause you could do some
> weird thing like using a different class or parameter or something to
update
> one leg first and then converging to the correct version, but at the
default
> 30 minutes interval it would take you 1 hour to deploy, for example.
>
> ------------------------------
>
> Jesús Couto F.
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

> that are very much procedural while Puppet manifest are more
> useful on a description of required software level.
Sort of.

The long story is that we don''t have a really native feeling way to
model multinode deployments and workflow now, but we can think of
modeling it based on a set of checkpoint conditions.

Meanwhile you can see a bit of a preview here:

http://github.com/reductivelabs/puppet-external-resource

external_resource { "some remote condition":
    frequency => 1,
    timeout => 30,
    check => "/some/external/script/that/will/return/true"
}

The trick is of course that the remote check software does not exist
as part of anything in Puppet core, yet, so that would be up to you.

However blocking until a database is present on a remote server is
actually possible now.   (Provided you write the script to check for
it).

I would agree that the concept of taking a node out of a load balanced
configuration and modelling that configuration is more difficult to do
intelligently, as the system needs to understand a fair amount of
context
about your HA environment.

It''s not so much of an issue of declarative vs procedural, but more so
a question of can the system model your configuration.   Right now,
those systems today that are ''procedural'' don''t model
at all, they are
more of raw code.

--Michael

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mon, Mar 15, 2010 at 3:18 PM, Michael DeHaan
<michael@reductivelabs.com>wrote:
> > that are very much procedural while Puppet manifest are more
> > useful on a description of required software level.
>
> Sort of.
>
> The long story is that we don''t have a really native feeling way
to
> model multinode deployments and workflow now, but we can think of
> modeling it based on a set of checkpoint conditions.
>
>
On a complete pipe-dream, "I''m not the one with the skills to do
this"
comment, I think it would be great to extend the Puppet language toward
"site" configurations. As exported resources, but more. If you could
define,
say, an "application" resource that is not on a node but on several
nodes,
that would model the application - this app is this and this running on
those 2 servers who are on loadbalancing and this and that on those other 2,
and the parts on the webservers requires the parts on the appservers that
requires the parts on the database servers...dont know at what level could
it be modeled to be flexible enough and not one size "deploy" model
for all,
but the idea would be to make it like Puppet goes from "let me script
this"
to "let me describe how it should be", with you describing your
application
structucture and relationships and such.





-- 
------------------------------

Jesús Couto F.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

[crossposting to puppet-dev, please trim follow-ups appropriately]

On 3/16/2010 11:52 AM, Jesús Couto wrote:
>
>
> On Mon, Mar 15, 2010 at 3:18 PM, Michael DeHaan
> <michael@reductivelabs.com <mailto:michael@reductivelabs.com>>
wrote:
>
>      > that are very much procedural while Puppet manifest are more
>      > useful on a description of required software level.
>
>     Sort of.
>
>     The long story is that we don''t have a really native feeling
way to
>     model multinode deployments and workflow now, but we can think of
>     modeling it based on a set of checkpoint conditions.
>
>
> On a complete pipe-dream, "I''m not the one with the skills to
do this"
> comment, I think it would be great to extend the Puppet language toward
> "site" configurations. As exported resources, but more. If you
could
> define, say, an "application" resource that is not on a node but
on
> several nodes, that would model the application - this app is this and
> this running on those 2 servers who are on loadbalancing and this and
> that on those other 2, and the parts on the webservers requires the
> parts on the appservers that requires the parts on the database
> servers...dont know at what level could it be modeled to be flexible
> enough and not one size "deploy" model for all, but the idea
would be to
> make it like Puppet goes from "let me script this" to "let
me describe
> how it should be", with you describing your application structucture
and
> relationships and such.
>

If you want to prototype something like this, you can use a define 
outside of a node in the site.pp and use checks against $fqdn to 
distribute resources among hosts.

Maybe even use the external_resource type that''s currently floating 
around to sequence the deployment.


Which leads me to another idea: inter-node dependencies:

| node a { mysql_db { "foo": ... } }
|
| node b { app { "x": after => A>>Mysql_db["foo"];
} }


(please ignore the crude syntax)


Best Regards, David
-- 
dasz.at OG              Tel: +43 (0)664 2602670     Web: http://dasz.at
Klosterneuburg                                         UID: ATU64260999

        FB-Nr.: FN 309285 g          FB-Gericht: LG Korneuburg

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

David Schmitt wrote:
>
>
>
> Which leads me to another idea: inter-node dependencies:
>
> | node a { mysql_db { "foo": ... } }
> |
> | node b { app { "x": after =>
A>>Mysql_db["foo"]; } }
>
>
>
David, are you suggesting this, or are you saying that this works?

-- 
Joe McDonagh
AIM: YoosingYoonickz
IRC: joe-mac on freenode
L''ennui est contre-révolutionnaire

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 3/16/2010 6:13 PM, Joe McDonagh wrote:
> David Schmitt wrote:
>>
>>
>>
>> Which leads me to another idea: inter-node dependencies:
>>
>> | node a { mysql_db { "foo": ... } }
>> |
>> | node b { app { "x": after =>
A>>Mysql_db["foo"]; } }
>>
>>
>>
> David, are you suggesting this, or are you saying that this works?
>
I''m suggesting something like this. Internally it''d probably
work like
external_resource (see -dev list for details) but would query the 
storeconfig DB instead of calling a script on the client.


Regards, David
-- 
dasz.at OG              Tel: +43 (0)664 2602670     Web: http://dasz.at
Klosterneuburg                                         UID: ATU64260999

        FB-Nr.: FN 309285 g          FB-Gericht: LG Korneuburg

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

David Schmitt wrote:
> [crossposting to puppet-dev, please trim follow-ups appropriately]
>
> On 3/16/2010 11:52 AM, Jesús Couto wrote:
>>
>>
>> On Mon, Mar 15, 2010 at 3:18 PM, Michael DeHaan
>> <michael@reductivelabs.com
<mailto:michael@reductivelabs.com>> wrote:
>>
>>      > that are very much procedural while Puppet manifest are more
>>      > useful on a description of required software level.
>>
>>     Sort of.
>>
>>     The long story is that we don''t have a really native
feeling way to
>>     model multinode deployments and workflow now, but we can think of
>>     modeling it based on a set of checkpoint conditions.
>>
>>
>> On a complete pipe-dream, "I''m not the one with the
skills to do this"
>> comment, I think it would be great to extend the Puppet language toward
>> "site" configurations. As exported resources, but more. If
you could
>> define, say, an "application" resource that is not on a node
but on
>> several nodes, that would model the application - this app is this and
>> this running on those 2 servers who are on loadbalancing and this and
>> that on those other 2, and the parts on the webservers requires the
>> parts on the appservers that requires the parts on the database
>> servers...dont know at what level could it be modeled to be flexible
>> enough and not one size "deploy" model for all, but the idea
would be to
>> make it like Puppet goes from "let me script this" to
"let me describe
>> how it should be", with you describing your application
structucture and
>> relationships and such.
>>
>
>
> If you want to prototype something like this, you can use a define 
> outside of a node in the site.pp and use checks against $fqdn to 
> distribute resources among hosts.
>
> Maybe even use the external_resource type that''s currently
floating
> around to sequence the deployment.
>
>
> Which leads me to another idea: inter-node dependencies:
>
> | node a { mysql_db { "foo": ... } }
> |
> | node b { app { "x": after =>
A>>Mysql_db["foo"]; } }
>
>
> (please ignore the crude syntax)
>
>
> Best Regards, David
I don''t immediately see it in my dev list, but I''ll take your
word for
it. I am with you on the inter-node dependencies.

-- 
Joe McDonagh
AIM: YoosingYoonickz
IRC: joe-mac on freenode
L''ennui est contre-révolutionnaire

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Mar 16, 2010 at 1:14 PM, David Schmitt <david@dasz.at>
wrote:
> On 3/16/2010 6:13 PM, Joe McDonagh wrote:
>>
>> David Schmitt wrote:
>>>
>>>
>>>
>>> Which leads me to another idea: inter-node dependencies:
>>>
>>> | node a { mysql_db { "foo": ... } }
>>> |
>>> | node b { app { "x": after =>
A>>Mysql_db["foo"]; } }
>>>
>>>
>>>
>> David, are you suggesting this, or are you saying that this works?
>>
>
> I''m suggesting something like this. Internally it''d
probably work like
> external_resource (see -dev list for details) but would query the
> storeconfig DB instead of calling a script on the client.
>
>
Yeah I had the same idea.

I think Luke was suggesting a higher level of modelling, but that can
work now, provided there''s a way to easily create the content for the
remote resource check.

The Func-like approach would be for each node to surface what it''s
"gates" were, and then set up authorization to allow other nodes to
ask them if a particular task was performed.

I''m not /quite/ sure that is sufficient though, or otherwise the gates
would have to be pretty smart -- for controlling an app upgrade you
would need more of a check than just seeing if the DB was present or a
version number was in a table.

--Michael

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 3/16/2010 10:04 PM, Joe McDonagh wrote:
> I don''t immediately see it in my dev list, but I''ll take
your word for
> it. I am with you on the inter-node dependencies.

Should have better checked. It was here:

   http://groups.google.com/group/puppet-users/msg/1d54d9db85dae153


Regards, David
-- 
dasz.at OG              Tel: +43 (0)664 2602670     Web: http://dasz.at
Klosterneuburg                                         UID: ATU64260999

        FB-Nr.: FN 309285 g          FB-Gericht: LG Korneuburg

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mar 9, 2010, at 9:54 AM, Joe McDonagh wrote:
> Michael DeHaan wrote:
>> On Mon, Mar 8, 2010 at 9:46 AM, Julian Simpson
<simpsonjulian@gmail.com
>> > wrote:
>>
>>>>> Given that there are so many ways to skin the same cat,
I''m
>>>>> wondering
>>>>> how others are doing their application deployments using
Puppet.
>>>>> Or,
>>>>> if you are using Puppet but do your app deployments via
some other
>>>>> mechanism, how do you do it, and why aren''t you
using Puppet to
>>>>> do it?
>>>>>
>>>>> Pete
>>>>>
>>>>>
>>>>>
>>>> I use capistrano for application deployment.
>>>>
>>> I''ve managed to use both Puppet and Capistrano for
deployment: Cap
>>> for the
>>> business code, and Puppet to ensure that dependencies are met.   
>>> Puppet was
>>> called from Capistrano.
>>> J.
>>>
>>>
>>
>>
>> Can you share some examples of where your dividing line is and some  
>> of
>> the tasks that are hard to handle in Puppet today?
>>
>> Ideally I''d like to figure out how to incorporate more of that
into
>> puppet-core.    Bruce has been working on a version control type,  
>> that
>> could be used to check sources
>> out of version control and eliminate some packaging needs for hosted
>> apps, though I''m curious as to what other niches might not be
>> covered.
>>
>> --Michael
>>
>>
> Example of what puppet can do best:
>
> Configure a system with apache2, a basic security configuration,  
> some enabled modules, a bunch of packages admins expect to have  
> available (screen, irb, etc).
>
> Example of what capistrano can do best:
>
> A. Deploy your puppet manifests to your puppet server.
> B. Deploy your web content out of svn onto your web server.
>
> They''re two distinct areas of ''deployment''.
Puppet is more like,
> systems deployment, cap is app or content deployment in my eyes.

I would essentially agree with that, with one caveat - that''s  
something Puppet''s not very good at today, but I''m willing to
make
some bets (not promises, quite yet) about it being good at it within a  
year or so.

-- 
Fallacies do not cease to be fallacies because they become fashions.
     --G. K. Chesterton
---------------------------------------------------------------------
Luke Kanies  -|-   http://reductivelabs.com   -|-   +1(615)594-8199

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mar 11, 2010, at 12:53 PM, Julian Simpson wrote:
> > They''re two distinct areas of ''deployment''.
Puppet is more like,
> systems
> > deployment, cap is app or content deployment in my eyes.
>
> Yes, I want to change that observation.
>
>
> I use Puppet for deploying my blog.  It''s a WordPress distro with
a
> bunch of plugins, etc.  It was too slow.  I ended up inelegantly  
> using rsync from puppet to push out the distro instead, which was  
> insanely fast.  Even that didn''t come for free as I''m not
using a
> Puppetmaster at the moment, and paths got annoying.  (I do accept  
> that''s probably my issue, however).
>
> Anyway, my point is that Puppet needs to give you a way to deploy a  
> tree of files, fast.  If the new type mentioned above starts to give  
> you that, so much the better: it gets tedious to push files to your  
> version control system, ensure that your puppetmaster is up to date,  
> and then invoke puppet on the node that you''re deploying to.
>
> What''s appealing about some developer-style deploy tools is that
you
> can push exactly what you''re working on to one or more nodes and  
> test it.  My experiences with Puppet suggest that it''s not so  
> convenient.  My 6 years as a build and release manager tell me that  
> it''s generally a poor idea to use two application deployment
tools.
> Hmmm.

I essentially agree on both sides:  Puppet isn''t yet that great at  
solving this problem, but using two tools sucks.

We''re basically still in the formative stage of what we''re
going to do
here, but we''ve got some pretty good ideas.  Mostly, it builds around  
a much more powerful data model in the catalog, combined with a  
centralized system that tracks inter-node dependencies.

Keeping in mind that this is all still a bit vague:

We''ll be extending the data model to support service state  
dependencies and simple state machines, along with (probably) some  
kind of fill-in-the-value (which Markus insists on calling
''futures'')
catalog.  This, I think, gets us most of the way to what we need on a  
single machine:

If you have two services, where one service specifies that it being in  
a ''running'' state requires that the other service must also be
in a
''running'' state, when you got to restart the required service
then the
requiring service can automatically be brought down first.

If you have a single composite service (e.g., a "web app" composed of
a db and apache), then you could have a specification that defines how  
it should behave depending on whether it should be running or not, and  
then external data that defines which value you want in a given case.

Once we start tracking resource collection as a dependency (because  
one host''s configuration is now dependent on another''s), plus
a
centralized database containing all compiled catalogs, we''ve basically
got builtin node-order dependencies, all emergent rather than  
declared.  You could get dependencies through declaration, I think,  
but it doesn''t scale as well.

The ''external_resource'' module I sent in last week[1] is a
good
example of what the first bit will look like, but it has a lot of  
"here be dragons" associated with it.  To make it supportable in the  
long term, you need to add builtin interfaces for checking service  
state (probably with central correlation rather than direct  
connectivity) plus the inter-node dependency system needs to recognize  
this as a dependency.

Obviously, this is 80% theory, 15% plan, and 5% bs, but we do have  
ideas for how it''ll work, and we''re making baby steps toward
getting
some of it actually done.

In the meantime, I''d appreciate any results from people testing this  
external resource stuff.

1 - http://github.com/reductivelabs/puppet-external-resource

-- 
The Internet, of course, is more than just a place to find pictures of
people having sex with dogs. -- Time Magazine, 3 July 1995
---------------------------------------------------------------------
Luke Kanies  -|-   http://reductivelabs.com   -|-   +1(615)594-8199

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mar 16, 2010, at 4:07 AM, David Schmitt wrote:
> [crossposting to puppet-dev, please trim follow-ups appropriately]
>
> On 3/16/2010 11:52 AM, Jesús Couto wrote:
>>
>>
>> On Mon, Mar 15, 2010 at 3:18 PM, Michael DeHaan
>> <michael@reductivelabs.com
<mailto:michael@reductivelabs.com>> wrote:
>>
>>     > that are very much procedural while Puppet manifest are more
>>     > useful on a description of required software level.
>>
>>    Sort of.
>>
>>    The long story is that we don''t have a really native
feeling way
>> to
>>    model multinode deployments and workflow now, but we can think of
>>    modeling it based on a set of checkpoint conditions.
>>
>>
>> On a complete pipe-dream, "I''m not the one with the
skills to do
>> this"
>> comment, I think it would be great to extend the Puppet language  
>> toward
>> "site" configurations. As exported resources, but more. If
you could
>> define, say, an "application" resource that is not on a node
but on
>> several nodes, that would model the application - this app is this  
>> and
>> this running on those 2 servers who are on loadbalancing and this and
>> that on those other 2, and the parts on the webservers requires the
>> parts on the appservers that requires the parts on the database
>> servers...dont know at what level could it be modeled to be flexible
>> enough and not one size "deploy" model for all, but the idea
would
>> be to
>> make it like Puppet goes from "let me script this" to
"let me
>> describe
>> how it should be", with you describing your application  
>> structucture and
>> relationships and such.
>>
>
>
> If you want to prototype something like this, you can use a define  
> outside of a node in the site.pp and use checks against $fqdn to  
> distribute resources among hosts.
>
> Maybe even use the external_resource type that''s currently
floating
> around to sequence the deployment.
>
>
> Which leads me to another idea: inter-node dependencies:
>
> | node a { mysql_db { "foo": ... } }
> |
> | node b { app { "x": after =>
A>>Mysql_db["foo"]; } }
The external resource stuff does just about exactly this, except that  
it''s not a special syntax.  It''s not as manageable as
I''d like --
there''s no way, for instance, to really understand the dependencies  
between hosts without reading all of the code -- but it''s a good start
and should help us set the model.

-- 
Seize opportunity by the beard, for it is bald behind.
     -- Bulgarian Proverb
---------------------------------------------------------------------
Luke Kanies  -|-   http://reductivelabs.com   -|-   +1(615)594-8199

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 3/17/2010 8:41 PM, Luke Kanies wrote:
> If you have two services, where one service specifies that it being in a
> ''running'' state requires that the other service must also
be in a
> ''running'' state, when you got to restart the required
service then the
> requiring service can automatically be brought down first.
+1
> Once we start tracking resource collection as a dependency (because one
> host''s configuration is now dependent on another''s), plus
a centralized
> database containing all compiled catalogs, we''ve basically got
builtin
> node-order dependencies, all emergent rather than declared.
+1
> Obviously, this is 80% theory, 15% plan, and 5% bs, but we do have ideas
> for how it''ll work, and we''re making baby steps toward
getting some of
> it actually done.
+1


Regards, David
-- 
dasz.at OG              Tel: +43 (0)664 2602670     Web: http://dasz.at
Klosterneuburg                                         UID: ATU64260999

        FB-Nr.: FN 309285 g          FB-Gericht: LG Korneuburg

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thu, Mar 18, 2010 at 10:58:52AM +0100, David Schmitt
wrote:
> On 3/17/2010 8:41 PM, Luke Kanies wrote:
>> If you have two services, where one service specifies that it being in
a
>> ''running'' state requires that the other service must
also be in a
>> ''running'' state, when you got to restart the required
service then the
>> requiring service can automatically be brought down first.
>
> +1
There might be exceptions, though. Consider an application that requires
a database to be up for startup, but once it''s running, is able to
survive a database restart, degrading gracefully (serving from cache or
whatever). You would not want to degrade ungracefully by shutting it
down with the DB.

-- 
Marcin Owsiany <marcin@owsiany.pl>              http://marcin.owsiany.pl/
GnuPG: 1024D/60F41216  FE67 DA2D 0ACA FC5E 3F75  D6F6 3A0D 8AA0 60F4 1216
 
"Every program in development at MIT expands until it can read mail."
                                                              -- Unknown

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 3/19/2010 9:42 AM, Marcin Owsiany wrote:
> On Thu, Mar 18, 2010 at 10:58:52AM +0100, David Schmitt wrote:
>> On 3/17/2010 8:41 PM, Luke Kanies wrote:
>>> If you have two services, where one service specifies that it being
in a
>>> ''running'' state requires that the other service
must also be in a
>>> ''running'' state, when you got to restart the
required service then the
>>> requiring service can automatically be brought down first.
>>
>> +1
>
> There might be exceptions, though. Consider an application that requires
> a database to be up for startup, but once it''s running, is able to
> survive a database restart, degrading gracefully (serving from cache or
> whatever). You would not want to degrade ungracefully by shutting it
> down with the DB.
Indeed. But this is a question of modelling, not of the required 
mechnics. Such an application would need to have running, degraded and 
stopped states defined. This could even be used to start a lighttpd 
serving a static "we''re offline" site instead of a normally
running
application container.



Regards, David
-- 
dasz.at OG              Tel: +43 (0)664 2602670     Web: http://dasz.at
Klosterneuburg                                         UID: ATU64260999

        FB-Nr.: FN 309285 g          FB-Gericht: LG Korneuburg

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.