Puppet users - Jan 2012 - Puppet proxies

I''ve seen a couple of examples of using puppet to control services
where you can''t run a puppet agent on the resource. As far as I can
tell they work by having puppet proxy through some external-service
control mechanism, such as fog. The question is: where does the proxy
run? I assume a puppet agent needs to run somewhere. Where do you put
that agent, how do you configure it, and how do you identify it as a
puppet node? I''m having visions of sugar plum fairies at the thought
of using Puppet to do what CloudFormation does (automate the
configuration of an entire AWS environment). But it''s not clear to me
how to wire it together.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Jan 10, 2012 at 3:12 PM, Jeff Sussna <jes@ingineering.it>
wrote:
> I''ve seen a couple of examples of using puppet to control services
> where you can''t run a puppet agent on the resource. As far as I
can
> tell they work by having puppet proxy through some external-service
> control mechanism, such as fog. The question is: where does the proxy
> run? I assume a puppet agent needs to run somewhere. Where do you put
> that agent, how do you configure it, and how do you identify it as a
> puppet node? I''m having visions of sugar plum fairies at the
thought
> of using Puppet to do what CloudFormation does (automate the
> configuration of an entire AWS environment). But it''s not clear to
me
> how to wire it together.
You might have a look at
http://puppetlabs.com/blog/using-cloudformation-to-build-out-fully-functional-stacks-of-puppet-enterprise/

>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I''m not looking to create Puppet environments in AWS. Rather the
opposite: use Puppet to configure AWS services, including things like
RDS, ElastiCache, ELB, that can''t have puppet agents running on them.
I had hoped to use CloudFormation itself for that purpose. Their
support for change management is still incomplete. Without change
management CF is IMHO worse than useless.

The only thing I''m unsure of is how/where to run my proxy. Seems like
the simplest way to do it is to run a single puppet agent somewhere
(maybe even on the master machine). Then create the necessary AWS-
control modules and include them in the manifest for my proxy node.

I wonder if it might be useful to add the concept of a pseudo-node to
puppet for cases like this.


On Jan 10, 5:13 pm, Michael Stahnke <stah...@puppetlabs.com>
wrote:
> On Tue, Jan 10, 2012 at 3:12 PM, Jeff Sussna <j...@ingineering.it>
wrote:
> > I''ve seen a couple of examples of using puppet to control
services
> > where you can''t run a puppet agent on the resource. As far as
I can
> > tell they work by having puppet proxy through some external-service
> > control mechanism, such as fog. The question is: where does the proxy
> > run? I assume a puppet agent needs to run somewhere. Where do you put
> > that agent, how do you configure it, and how do you identify it as a
> > puppet node? I''m having visions of sugar plum fairies at the
thought
> > of using Puppet to do what CloudFormation does (automate the
> > configuration of an entire AWS environment). But it''s not
clear to me
> > how to wire it together.
>
> You might have a look
athttp://puppetlabs.com/blog/using-cloudformation-to-build-out-fully-fu...
>
>
>
>
>
>
>
>
>
> > --
> > You received this message because you are subscribed to the Google
Groups "Puppet Users" group.
> > To post to this group, send email to puppet-users@googlegroups.com.
> > To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> > For more options, visit this group
athttp://groups.google.com/group/puppet-users?hl=en.
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Jan 11, 2012 at 8:57 AM, Jeff Sussna <jes@ingineering.it>
wrote:
> I''m not looking to create Puppet environments in AWS. Rather the
> opposite: use Puppet to configure AWS services, including things like
> RDS, ElastiCache, ELB, that can''t have puppet agents running on
them.
> I had hoped to use CloudFormation itself for that purpose. Their
> support for change management is still incomplete. Without change
> management CF is IMHO worse than useless.
Yeah, at the moment that''s one of the killer limitations of
Cloudformation.
> The only thing I''m unsure of is how/where to run my proxy. Seems
like
> the simplest way to do it is to run a single puppet agent somewhere
> (maybe even on the master machine). Then create the necessary AWS-
> control modules and include them in the manifest for my proxy node.
How do you plan to configure EC2 services? through existing EC2
command line tools, fog, or some other third party tool?
> I wonder if it might be useful to add the concept of a pseudo-node to
> puppet for cases like this.
In Puppet 2.7 the puppet device application was introduced to support
network devices that can''t run puppet agent. A proxy puppet agent
server will interact with the puppet master and enforce the catalog by
configuring the device through the configured transport. Puppet device
is extensible, and this actually suitable beyond network device for
things like storage, and possibly your use case for configuring EC2
resources. It would be really awesome to extend support for EC2.

Here''s an example how we could model EC2 resources and use puppet
device to manage it.

/etc/puppet/ec2.conf
[amazon]
fog /path/to/fog.credentials

on the master specify the amazon node and puppet device will manage
those EC2 resources.
node amazon {
  ec2_instance {...}
  ec2_rds {...}
  ec2_s3 {...}
}

$ puppet device --deviceconfig /etc/puppet/ec2.conf

Things like s3, rds are simple to map, because we can specify the name
of the instance on creation:

ec2_rds { ''demodb'':
  ensure => present,
  class => ''db.m1.small'',
  engine => ''mysql'',
  zone => ''us/east'',
  username => ''...'',
...
}

But for instances, vpc this is a bit more problematic because the
instance_id is determined after they are launched. Also I think for
production use cases, people probably don''t care about the
instance_id, but rather how many systems of this role I need with this
ami-id perhaps. This is one of the areas I need some feedback and
suggestion.

For instances perhaps:

ec2_instances { "instance_name (???)":
  instance_id => ''can''t really manage this.'',
...
}

ec2_vpc { "??? what uniquely identifies this ???":
...
}

I''m definitely looking see if there''s interest and whether the
model
above fits people usage of EC2.

Thanks,

Nan

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Either fog or the existing AWS API tools. Haven''t looked deeply enough
into fog to see whether it has sufficient support for Amazon services
that don''t have analogs elsewhere (RDS, ELB, etc.).

Came across puppet device, but it wasn''t clear to me how it actually
worked. On closer inspection, it looks like just the thing. Ideally
I''d define an "app" (ELB instance + some EC2 instances + RDS
instance
for example), and be able to instantiate it in multiple places (US-
East prod, US-East dev, US-West DR), each time with different
parameters (RDS size, number of EC2 instances, etc.).  I assume I
could do something like:

/etc/puppet/ec2.conf
[app1-prod]
[app1-dev]
[app1-dr]


/etc/puppet/manifests/nodes.pp

node app1-prod {
   app1 { ''app1-prod'': region => ''east'',
webservers => 4, rds_size =>
''large'', }
}

node app1-dev {
   app1 { ''app1-dev'': region => ''east'',
webservers => 1, rds_size =>
''small'', }
}

# app1 is a puppet define I create to encapsulate ec2_instances,
ec2_rds, etc.)


If I''m right then this approach has serious holy-grail potential and
I''m definitely interested in going further with it.



On Jan 12, 12:41 am, Nan Liu <n...@puppetlabs.com>
wrote:
> On Wed, Jan 11, 2012 at 8:57 AM, Jeff Sussna <j...@ingineering.it>
wrote:
> > I''m not looking to create Puppet environments in AWS. Rather
the
> > opposite: use Puppet to configure AWS services, including things like
> > RDS, ElastiCache, ELB, that can''t have puppet agents running
on them.
> > I had hoped to use CloudFormation itself for that purpose. Their
> > support for change management is still incomplete. Without change
> > management CF is IMHO worse than useless.
>
> Yeah, at the moment that''s one of the killer limitations of
Cloudformation.
>
> > The only thing I''m unsure of is how/where to run my proxy.
Seems like
> > the simplest way to do it is to run a single puppet agent somewhere
> > (maybe even on the master machine). Then create the necessary AWS-
> > control modules and include them in the manifest for my proxy node.
>
> How do you plan to configure EC2 services? through existing EC2
> command line tools, fog, or some other third party tool?
>
> > I wonder if it might be useful to add the concept of a pseudo-node to
> > puppet for cases like this.
>
> In Puppet 2.7 the puppet device application was introduced to support
> network devices that can''t run puppet agent. A proxy puppet agent
> server will interact with the puppet master and enforce the catalog by
> configuring the device through the configured transport. Puppet device
> is extensible, and this actually suitable beyond network device for
> things like storage, and possibly your use case for configuring EC2
> resources. It would be really awesome to extend support for EC2.
>
> Here''s an example how we could model EC2 resources and use puppet
> device to manage it.
>
> /etc/puppet/ec2.conf
> [amazon]
> fog /path/to/fog.credentials
>
> on the master specify the amazon node and puppet device will manage
> those EC2 resources.
> node amazon {
>   ec2_instance {...}
>   ec2_rds {...}
>   ec2_s3 {...}
>
> }
>
> $ puppet device --deviceconfig /etc/puppet/ec2.conf
>
> Things like s3, rds are simple to map, because we can specify the name
> of the instance on creation:
>
> ec2_rds { ''demodb'':
>   ensure => present,
>   class => ''db.m1.small'',
>   engine => ''mysql'',
>   zone => ''us/east'',
>   username => ''...'',
> ...
>
> }
>
> But for instances, vpc this is a bit more problematic because the
> instance_id is determined after they are launched. Also I think for
> production use cases, people probably don''t care about the
> instance_id, but rather how many systems of this role I need with this
> ami-id perhaps. This is one of the areas I need some feedback and
> suggestion.
>
> For instances perhaps:
>
> ec2_instances { "instance_name (???)":
>   instance_id => ''can''t really manage this.'',
> ...
>
> }
>
> ec2_vpc { "??? what uniquely identifies this ???":
> ...
>
> }
>
> I''m definitely looking see if there''s interest and
whether the model
> above fits people usage of EC2.
>
> Thanks,
>
> Nan
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Nan,

On Thu, 2012-01-12 at 00:41 -0600, Nan Liu wrote:
> On Wed, Jan 11, 2012 at 8:57 AM, Jeff Sussna <jes@ingineering.it>
wrote:
> > I''m not looking to create Puppet environments in AWS. Rather
the
> > opposite: use Puppet to configure AWS services, including things like
> > RDS, ElastiCache, ELB, that can''t have puppet agents running
on them.
> > I had hoped to use CloudFormation itself for that purpose. Their
> > support for change management is still incomplete. Without change
> > management CF is IMHO worse than useless.
> 
> Yeah, at the moment that''s one of the killer limitations of
Cloudformation.
> 
> > The only thing I''m unsure of is how/where to run my proxy.
Seems like
> > the simplest way to do it is to run a single puppet agent somewhere
> > (maybe even on the master machine). Then create the necessary AWS-
> > control modules and include them in the manifest for my proxy node.
> 
> How do you plan to configure EC2 services? through existing EC2
> command line tools, fog, or some other third party tool?
> 
> > I wonder if it might be useful to add the concept of a pseudo-node to
> > puppet for cases like this.
> 
> In Puppet 2.7 the puppet device application was introduced to support
> network devices that can''t run puppet agent. A proxy puppet agent
> server will interact with the puppet master and enforce the catalog by
> configuring the device through the configured transport. Puppet device
> is extensible, and this actually suitable beyond network device for
> things like storage, and possibly your use case for configuring EC2
> resources. It would be really awesome to extend support for EC2.
> 
> Here''s an example how we could model EC2 resources and use puppet
> device to manage it.
> 
> /etc/puppet/ec2.conf
> [amazon]
> fog /path/to/fog.credentials
Or directly fog credentials...
> on the master specify the amazon node and puppet device will manage
> those EC2 resources.
> node amazon {
>   ec2_instance {...}
>   ec2_rds {...}
>   ec2_s3 {...}
> }
> 
> $ puppet device --deviceconfig /etc/puppet/ec2.conf
> 
> Things like s3, rds are simple to map, because we can specify the name
> of the instance on creation:
> 
> ec2_rds { ''demodb'':
>   ensure => present,
>   class => ''db.m1.small'',
>   engine => ''mysql'',
>   zone => ''us/east'',
>   username => ''...'',
> ...
> }
> 
> But for instances, vpc this is a bit more problematic because the
> instance_id is determined after they are launched. Also I think for
> production use cases, people probably don''t care about the
> instance_id, but rather how many systems of this role I need with this
> ami-id perhaps. This is one of the areas I need some feedback and
> suggestion.
> 
> For instances perhaps:
> 
> ec2_instances { "instance_name (???)":
>   instance_id => ''can''t really manage this.'',
> ...
> }
That''s where you would use ec2 tags. We can have an ec2
''puppet_resource'' tag that would map back to the ec2_instance
namevar.
The ec2_instance provider would then be able to look-up the instances in
ec2 and manage them appropriatly.
> ec2_vpc { "??? what uniquely identifies this ???":
> ...
> }
I don''t know if you can tag vpc, but if you can I suggest using the
same
model.
> I''m definitely looking see if there''s interest and
whether the model
> above fits people usage of EC2.
Based on discussions I had at the last PuppetConf with a lot of people,
I''m sure this would be a terriffic feature.

Imagine all what you can do, including setting up notification events
between ec2 resources and so on.

Granted this might not be dynamic (in the elastic meaning), but I
believe this can fit a need for people that don''t want to introduce a
different tool to manage their ec2 cloud.
-- 
Brice Figureau
Follow the latest Puppet Community evolutions on www.planetpuppet.org!

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.