Puppet users - Apr 2013 - High Availability of Puppet server for separate geographical location

Hi Team,

I have to configure puppet server in Primary-Secondary mode for 2
distributed location .

Site A is already running 1 Puppet server .Now I need to configure
another puppet secondary server at site B ,so that

all client at location B ,only connect to that server .And site A
puppet client only connect to site A .

But mine requirement is of that suppose site A server goes down then
Site B server handle all client request and

same if Site B server down then server A handles all .

Kindly guide me to implement this scenario .

Thanks in Advance

Shyam

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Monday, April 29, 2013 8:55:39 AM UTC-5, shyam sundar Keshari
wrote:
>
> Hi Team, 
>
> I have to configure puppet server in Primary-Secondary mode for 2 
> distributed location . 
>
> Site A is already running 1 Puppet server .Now I need to configure 
> another puppet secondary server at site B ,so that 
>
> all client at location B ,only connect to that server .And site A 
> puppet client only connect to site A . 
>
> But mine requirement is of that suppose site A server goes down then 
> Site B server handle all client request and 
>
> same if Site B server down then server A handles all . 
>
> Kindly guide me to implement this scenario . 
>
>
For the most part, this is outside Puppet''s scope.  You need to stand
up
some kind appliance or service in front of your Puppet servers to monitor 
when one goes down, so as to redirect traffic to the other when needed.  
That cannot be part of Puppet, else it would go down with Puppet if ever 
Puppet went down.

With an external failover system in place, there would still be a few 
things to arrange on the Puppet side.  In order for site B to properly 
serve site A''s nodes, it must have

   - site A''s SSL certificate, or at least a certificate that site
A''s
   nodes will trust;
   - ENC and/or node declarations for site A''s nodes, matching those on
   site A;
   - modules (and standalone classes, if any) matching those on site A;
   - external data (if applicable) appropriate for site A''s nodes; and
   - stored configs for site A, if applicable.
   
I see two reasonably good approaches here:

   1. the site A and B masters are maintained as a mirrored pair.  This has 
   the effect that from Puppet''s perspective, physical sites A and B
are
   meaningless partitions of a single large, federated site.  A shared, 
   central certificate authority would be required (and that would need to be 
   the site A master if you want to avoid issuing new certificates to all the 
   site A nodes).
   2. A mirror of site A''s master is maintained as a hot spare at site
B,
   and vice versa.  Each spare could run on the same (possibly virtual) 
   hardware as the local master (but on a different port), or it could run on 
   different hardware and merely be geographically colocated.  This would 
   maintain a logical distinction between sites A and B, and would be overall 
   easier to set up and maintain.

If you consider option 2, however, you should soon realize that in that 
case you don''t get any particular advantage from putting the spare
masters
on the same machine as the local site''s main master.  Indeed, it might
be
easier to set them up as ordinary nodes (for the site in which they are 
actually located), and even to let the local site''s masters have 
responsibility for maintaining the spares'' puppet master config, each
as a
mirror of the other site''s master.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

----- Original Message -----
> From: "jcbollinger" <John.Bollinger@stJude.org>
> To: puppet-users@googlegroups.com
> Sent: Tuesday, April 30, 2013 2:39:52 PM
> Subject: [Puppet Users] Re: High Availability of Puppet server for separate
geographical location
> 
> 
> 
> On Monday, April 29, 2013 8:55:39 AM UTC-5, shyam sundar Keshari wrote:
> >
> > Hi Team,
> >
> > I have to configure puppet server in Primary-Secondary mode for 2
> > distributed location .
> >
> > Site A is already running 1 Puppet server .Now I need to configure
> > another puppet secondary server at site B ,so that
> >
> > all client at location B ,only connect to that server .And site A
> > puppet client only connect to site A .
> >
> > But mine requirement is of that suppose site A server goes down then
> > Site B server handle all client request and
> >
> > same if Site B server down then server A handles all .
> >
> > Kindly guide me to implement this scenario .
> >
> >
> For the most part, this is outside Puppet''s scope.  You need to
stand up
> some kind appliance or service in front of your Puppet servers to monitor
> when one goes down, so as to redirect traffic to the other when needed.
> That cannot be part of Puppet, else it would go down with Puppet if ever
> Puppet went down.
> 
> With an external failover system in place, there would still be a few
> things to arrange on the Puppet side.  In order for site B to properly
> serve site A''s nodes, it must have
> 
>    - site A''s SSL certificate, or at least a certificate that site
A''s
>    nodes will trust;
>    - ENC and/or node declarations for site A''s nodes, matching
those on
>    site A;
>    - modules (and standalone classes, if any) matching those on site A;
>    - external data (if applicable) appropriate for site A''s nodes;
and
>    - stored configs for site A, if applicable.
SRV goes a long way towards solving some of this, it is possible to share 
a CA between masters too which solves some of that.  Gets a lot harder when
you involve PuppetDB.

http://docs.puppetlabs.com/guides/scaling_multiple_masters.html

>    
> I see two reasonably good approaches here:
> 
>    1. the site A and B masters are maintained as a mirrored pair.  This has
>    the effect that from Puppet''s perspective, physical sites A and
B are
>    meaningless partitions of a single large, federated site.  A shared,
>    central certificate authority would be required (and that would need to
be
>    the site A master if you want to avoid issuing new certificates to all
the
>    site A nodes).
>    2. A mirror of site A''s master is maintained as a hot spare at
site B,
>    and vice versa.  Each spare could run on the same (possibly virtual)
>    hardware as the local master (but on a different port), or it could run
on
>    different hardware and merely be geographically colocated.  This would
>    maintain a logical distinction between sites A and B, and would be
overall
>    easier to set up and maintain.
> 
> If you consider option 2, however, you should soon realize that in that
> case you don''t get any particular advantage from putting the spare
masters
> on the same machine as the local site''s main master.  Indeed, it
might be
> easier to set them up as ordinary nodes (for the site in which they are
> actually located), and even to let the local site''s masters have
> responsibility for maintaining the spares'' puppet master config,
each as a
> mirror of the other site''s master.
> 
> 
> John
> 
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
> 
> 
> 
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Mon, Apr 29, 2013 at 9:55 AM, shyam sundar Keshari
<shyamrkg@gmail.com> wrote:
> I have to configure puppet server in Primary-Secondary mode for 2
> distributed location .
I am in a similar situation. Not liking the options available, I am
building "puppetgit"
https://groups.google.com/forum/?fromgroups#!topic/puppet-users/OilxMytnD_k

cheers,



m
--
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don''t get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Is Puppetlabs planning some easy solution for this ?

The real problem I see is that you can''t separate the Puppet CA from
the
Puppet Master. So even that you can have multiple puppet masters, the CA 
must run in one of them. So if that server goes down your multiple puppet 
master setup is screwed as well.

I would love to see a Puppet CA high availability solution out of the box.

On Tuesday, April 30, 2013 6:43:08 AM UTC-7, R.I. Pienaar
wrote:
>
>
>
> ----- Original Message ----- 
> > From: "jcbollinger" <John.Bollinger@stJude.org> 
> > To: puppet...@googlegroups.com <javascript:> 
> > Sent: Tuesday, April 30, 2013 2:39:52 PM 
> > Subject: [Puppet Users] Re: High Availability of Puppet server for 
> separate geographical location 
> > 
> > 
> > 
> > On Monday, April 29, 2013 8:55:39 AM UTC-5, shyam sundar Keshari
wrote:
> > > 
> > > Hi Team, 
> > > 
> > > I have to configure puppet server in Primary-Secondary mode for 2
> > > distributed location . 
> > > 
> > > Site A is already running 1 Puppet server .Now I need to
configure
> > > another puppet secondary server at site B ,so that 
> > > 
> > > all client at location B ,only connect to that server .And site A
> > > puppet client only connect to site A . 
> > > 
> > > But mine requirement is of that suppose site A server goes down
then
> > > Site B server handle all client request and 
> > > 
> > > same if Site B server down then server A handles all . 
> > > 
> > > Kindly guide me to implement this scenario . 
> > > 
> > > 
> > For the most part, this is outside Puppet''s scope.  You need
to stand up
> > some kind appliance or service in front of your Puppet servers to 
> monitor 
> > when one goes down, so as to redirect traffic to the other when
needed.
> > That cannot be part of Puppet, else it would go down with Puppet if
ever
> > Puppet went down. 
> > 
> > With an external failover system in place, there would still be a few 
> > things to arrange on the Puppet side.  In order for site B to properly
> > serve site A''s nodes, it must have 
> > 
> >    - site A''s SSL certificate, or at least a certificate that
site A''s
> >    nodes will trust; 
> >    - ENC and/or node declarations for site A''s nodes,
matching those on
> >    site A; 
> >    - modules (and standalone classes, if any) matching those on site
A;
> >    - external data (if applicable) appropriate for site A''s
nodes; and
> >    - stored configs for site A, if applicable. 
>
> SRV goes a long way towards solving some of this, it is possible to share 
> a CA between masters too which solves some of that.  Gets a lot harder 
> when 
> you involve PuppetDB. 
>
> http://docs.puppetlabs.com/guides/scaling_multiple_masters.html 
>
>
> >     
> > I see two reasonably good approaches here: 
> > 
> >    1. the site A and B masters are maintained as a mirrored pair. 
This
> has 
> >    the effect that from Puppet''s perspective, physical sites
A and B are
> >    meaningless partitions of a single large, federated site.  A
shared,
> >    central certificate authority would be required (and that would
need
> to be 
> >    the site A master if you want to avoid issuing new certificates to 
> all the 
> >    site A nodes). 
> >    2. A mirror of site A''s master is maintained as a hot
spare at site
> B, 
> >    and vice versa.  Each spare could run on the same (possibly
virtual)
> >    hardware as the local master (but on a different port), or it could
> run on 
> >    different hardware and merely be geographically colocated.  This 
> would 
> >    maintain a logical distinction between sites A and B, and would be 
> overall 
> >    easier to set up and maintain. 
> > 
> > If you consider option 2, however, you should soon realize that in
that
> > case you don''t get any particular advantage from putting the
spare
> masters 
> > on the same machine as the local site''s main master.  Indeed,
it might
> be 
> > easier to set them up as ordinary nodes (for the site in which they
are
> > actually located), and even to let the local site''s masters
have
> > responsibility for maintaining the spares'' puppet master
config, each as
> a 
> > mirror of the other site''s master. 
> > 
> > 
> > John 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups 
> > "Puppet Users" group. 
> > To unsubscribe from this group and stop receiving emails from it, send
> an 
> > email to puppet-users...@googlegroups.com <javascript:>. 
> > To post to this group, send email to
puppet...@googlegroups.com<javascript:>.
>
> > Visit this group at http://groups.google.com/group/puppet-users?hl=en.
> > For more options, visit https://groups.google.com/groups/opt_out. 
> > 
> > 
> > 
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On 9 May 2013 05:57, Felipe Salum <fsalum@gmail.com> wrote:
> Is Puppetlabs planning some easy solution for this ?
>
I run 12 puppet servers around the world. They work in a multiple puppet
master solution where any client from any location can work with any puppet
server in any location with dns_alt_names. We have an easy/simple solution:

One puppet server is designated puppet-ca.example.com. All client''s
configuration files look like this:

    ca_server = puppet-ca.example.com

A second puppet server is randomly chosen to be puppet-ca2.example.com. A
rsync job runs every minute on puppet-ca2 to only suck down the
.../etc/ssl/ca directory from puppet-ca

If puppet-ca becomes unavailable, we move the puppet-ca CNAME to
puppet-ca2. That lag is acceptable to us. You may chose to use other load
balancing options like an F5

John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

<br><br><span style="font-family:Prelude, Verdana,
san-serif;"><br><br></span><span
id="signature"><div style="font-family: arial, sans-serif;
font-size: 12px;color: #999999;">-- Sent from my HP
Veer</div><br></span><span style="color:navy;
font-family:Prelude, Verdana, san-serif; "><hr align="left"
style="width:75%">On May 8, 2013 19:58, John Warburton
&lt;jwarburton@gmail.com&gt; wrote:
<br><br></span><div dir="ltr"><div
class="gmail_extra"><div class="gmail_quote">On 9
May 2013 05:57, Felipe Salum <span dir="ltr">&lt;<a
href="mailto:fsalum@gmail.com"
target="_blank">fsalum@gmail.com</a>&gt;</span>
wrote:<br><blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">

Is Puppetlabs planning some easy solution for this
?<br></blockquote><div><br></div><div>I run
12 puppet servers around the world. They work in a multiple puppet master
solution where any client from any location can work with any puppet server in
any location with dns_alt_names. We have an easy/simple solution:<br>

<br></div><div>One puppet server is designated <a
href="http://puppet-ca.example.com">puppet-ca.example.com</a>.
All client&#39;s configuration files look like
this:<br><br></div><div>    ca_server = <a
href="http://puppet-ca.example.com">puppet-ca.example.com</a><br>

<br></div><div>A second puppet server is randomly chosen to be
<a
href="http://puppet-ca2.example.com">puppet-ca2.example.com</a>.
A rsync job runs every minute on puppet-ca2 to only suck down the .../etc/ssl/ca
directory from puppet-ca<br>

<br></div><div>If puppet-ca becomes unavailable, we move the
puppet-ca CNAME to puppet-ca2. That lag is acceptable to us. You may chose to
use other load balancing options like an
F5<br><br></div><div>John<br></div></div>

</div></div>

<p></p>

-- <br />
You received this message because you are subscribed to the Google Groups
&quot;Puppet Users&quot; group.<br />
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.<br />
To post to this group, send email to puppet-users@googlegroups.com.<br />
Visit this group at <a
href="http://groups.google.com/group/puppet-users?hl=en">http://groups.google.com/group/puppet-users?hl=en</a>.<br
/>
For more options, visit <a
href="https://groups.google.com/groups/opt_out">https://groups.google.com/groups/opt_out</a>.<br
/>
&nbsp;<br />
&nbsp;<br />

<p></p>

-- <br />
You received this message because you are subscribed to the Google Groups
&quot;Puppet Users&quot; group.<br />
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.<br />
To post to this group, send email to puppet-users@googlegroups.com.<br />
Visit this group at <a
href="http://groups.google.com/group/puppet-users?hl=en">http://groups.google.com/group/puppet-users?hl=en</a>.<br
/>
For more options, visit <a
href="https://groups.google.com/groups/opt_out">https://groups.google.com/groups/opt_out</a>.<br
/>
&nbsp;<br />
&nbsp;<br />

Yes it works perfectly, I run 2 puppet servers and one of them is the CA 
using a CNAME as well for the puppet master/ca.

After it is set and working it is wonderful, however until you figure out 
that you need to remove the ssldir, then run puppet with dns_alt_names, 
then sign with dns_alt_names in the other side, then start the 
apache2/passenger, then do a lot of more other tricks until you get out of 
the certificate error messages it is a headache :)

Easy solution would be something that would be easier to setup, or maybe it 
is just me trying to complicate things. :)

On Wednesday, May 8, 2013 4:58:21 PM UTC-7, John Warburton
wrote:
>
> On 9 May 2013 05:57, Felipe Salum <fsa...@gmail.com
<javascript:>> wrote:
>
>> Is Puppetlabs planning some easy solution for this ?
>>
>
> I run 12 puppet servers around the world. They work in a multiple puppet 
> master solution where any client from any location can work with any puppet
> server in any location with dns_alt_names. We have an easy/simple solution:
>
> One puppet server is designated puppet-ca.example.com. All
client''s
> configuration files look like this:
>
>     ca_server = puppet-ca.example.com
>
> A second puppet server is randomly chosen to be puppet-ca2.example.com. A 
> rsync job runs every minute on puppet-ca2 to only suck down the 
> .../etc/ssl/ca directory from puppet-ca
>
> If puppet-ca becomes unavailable, we move the puppet-ca CNAME to 
> puppet-ca2. That lag is acceptable to us. You may chose to use other load 
> balancing options like an F5
>
> John
>  
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On 9 May 2013 10:59, Felipe Salum <fsalum@gmail.com> wrote:
>
> Easy solution would be something that would be easier to setup, or maybe
> it is just me trying to complicate things. :)
>
> I suppose all HA solutions are difficult - they have to suit your
site''s
definition of HA and meet cost/complexity needs. Sometime a one pager set
of instructions is good enough, but Puppet Labs aren''t going to be able
to
meet everyone''s HA needs

I was lucky that this was my second puppet deployment and I had a design
that was HA from the start - so built it all in from the start. Keep
plugging away

John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Thu, May 9, 2013 at 12:06 AM, John Warburton <jwarburton@gmail.com>
wrote:
> I suppose all HA solutions are difficult
Nah. A service correctly designed to be resilient can be HA with
trivial investment.

DNS is a good example. It may have blemishes but nobody stresses about
its availability. Setup as many tiers of redundancy as you want,
easily.

Puppet has no need to be centralized -- a git-based puppet setup can
handle it just fine.

I am writing some tooling for git+puppet (search for ppg in recent
posts to this list), and it''s trivial to add N-tiers of redundant
servers...

cheers,



m
--
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don''t get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Thu, May 9, 2013 at 10:42 AM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
> I am writing some tooling for git+puppet (search for ppg in recent
> posts to this list), and it''s trivial to add N-tiers of redundant
> servers...
Heh, so trivial in fact that you can use round-robin DNS and it''ll just
work :-)

I looked into adding the "feature" to ppg... then realized that git
plays very well with round-robin DNS.



m
--
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don''t get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On 5/9/2013 7:42 AM, Martin Langhoff wrote:
> On Thu, May 9, 2013 at 12:06 AM, John Warburton
<jwarburton@gmail.com> wrote:
>> I suppose all HA solutions are difficult
>
> Nah. A service correctly designed to be resilient can be HA with
> trivial investment.
>
> DNS is a good example. It may have blemishes but nobody stresses about
> its availability. Setup as many tiers of redundancy as you want,
> easily.
>
> Puppet has no need to be centralized -- a git-based puppet setup can
> handle it just fine.
>
> I am writing some tooling for git+puppet (search for ppg in recent
> posts to this list), and it''s trivial to add N-tiers of redundant
> servers...
Hubris, today thy name is Martin. :-)

	I''d argue that people have stressed about DNS availability for just 
under three decades and that we are currently enjoying the fruits of 
that labor. Personally, I have yet to work at a company where DNS has 
not caused a significant outage. I do agree that the tools are there to 
build resilience, but implementation matters and it is a hard problem to 
solve in a non trivial environment.

	Your ppg tooling does look interesting, but there is a large trade off 
in functionality. You could use some other systems to collect data and 
push it into a repo which is then queried in order to generate data for 
clients... starting to sound an awful lot like centralized Puppet to me 
and all the problems therein regardless of transport.

Ramin

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Thu, May 9, 2013 at 2:31 PM, Ramin K <ramin-list@badapple.net>
wrote:
> Hubris, today thy name is Martin. :-)
Fair enough. I am happy about the tool I am writing (almost finished!)
but, as the followup post makes clear, it isn''t about the designe of
ppg. It is about the design of git.
> I''d argue that people have stressed about DNS availability for
just
> under three decades and that we are currently enjoying the fruits of that
> labor. Personally, I have yet to work at a company where DNS has not caused
> a significant outage.
I am really surprised at your statement. Of course mishaps can happen,
or someone can mess up configuration DNS royally. But setting up a
primary and secondary setup is trivial.

SMTP and LDAP are also examples where resilience was baked into the
design. With those two, the quality of implementation, and
complications in setup make for a lot more breakage.

Compare to HTTP, databases etc where there''s a whole industry of tools
to make things somewhat reliable.

Maybe we are talking about different things.
> Your ppg tooling does look interesting, but there is a large trade
> off in functionality
What is the loss of functionality you see? Anything that you use in practice?

(Reading here
https://groups.google.com/forum/?fromgroups=#!topic/puppet-users/7ZpAMrMb2NQ
I can''t spot anything major, but I may be missing something...)

cheers,


m
--
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don''t get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On 5/9/2013 1:51 PM, Martin Langhoff wrote:
> On Thu, May 9, 2013 at 2:31 PM, Ramin K <ramin-list@badapple.net>
wrote:
>> Hubris, today thy name is Martin. :-)
>
> Fair enough. I am happy about the tool I am writing (almost finished!)
> but, as the followup post makes clear, it isn''t about the designe
of
> ppg. It is about the design of git.
	This is where I think we diverge. :-) As someone with a fair amount of 
operational experience it''s not about the design of git, it''s
about the
implementation created on top of git. Or Puppet.
>> I''d argue that people have stressed about DNS availability for
just
>> under three decades and that we are currently enjoying the fruits of
that
>> labor. Personally, I have yet to work at a company where DNS has not
caused
>> a significant outage.
>
> I am really surprised at your statement. Of course mishaps can happen,
> or someone can mess up configuration DNS royally. But setting up a
> primary and secondary setup is trivial.
>
> SMTP and LDAP are also examples where resilience was baked into the
> design. With those two, the quality of implementation, and
> complications in setup make for a lot more breakage.
>
> Compare to HTTP, databases etc where there''s a whole industry of
tools
> to make things somewhat reliable.
>
> Maybe we are talking about different things.
	Not different things, but perspectives.

	I''d agree that your simple primary/secondary name server is easy to 
setup and it''ll probably work just fine. However it supports a very 
limited number of use cases and traffic levels.

	My experience with DNS and administrating it in various incarnations 
since the bad old days of Bind 4 informs me that it can be incredibly 
fragile. It is only the implementation of the current DNS system that is 
reasonably resilient or at least able to localize failure. Certainly 
some designs and technology are better than others, but implementation 
always matters.

	The same goes for just about system/protocol you''d care to name.
>> Your ppg tooling does look interesting, but there is a large trade
>> off in functionality
>
> What is the loss of functionality you see? Anything that you use in
practice?
>
> (Reading here
https://groups.google.com/forum/?fromgroups=#!topic/puppet-users/7ZpAMrMb2NQ
> I can''t spot anything major, but I may be missing something...)
>
> cheers,
> m
	Masterless Puppet with git as a distribution method does have some 
things going for it as a design. You are giving up things like collected 
resources and standard reporting which may or may not matter you. 
Additionally you''re building a distribution system of some sort even if
it''s just git and ssh where you''d need to decide how to deal
with the
failure of the parts.

	In any case I''d like to see more discussion on highly available Puppet
regardless of way it''s implemented.

Ramin

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

I''m trying to make a manifest to auto setup Puppet High Availability,
but
it is the chicken-egg issue here. As for your secondary/tertiary/etc
puppetmasters, you need to copy the private key and certificate used by
your puppet1 server in order for it to accept the requests coming from
puppet.yourcompanydomain.com or whatever you choose there.

Also you need to maintain the /var/lib/puppet/ssl/ca in sync between all
puppetmasters so you can easily failover if puppet1 is down to any other.

Since Puppet is about automation, when I mentioned something easier to
setup I was expecting a puppet module to implement it, not manual changes.
As the implementation is not very straightforward it is not easy to
automate it IMHO.



On Fri, May 10, 2013 at 10:52 AM, Ramin K <ramin-list@badapple.net> wrote:
> On 5/9/2013 1:51 PM, Martin Langhoff wrote:
>
>> On Thu, May 9, 2013 at 2:31 PM, Ramin K <ramin-list@badapple.net>
wrote:
>>
>>> Hubris, today thy name is Martin. :-)
>>>
>>
>> Fair enough. I am happy about the tool I am writing (almost finished!)
>> but, as the followup post makes clear, it isn''t about the
designe of
>> ppg. It is about the design of git.
>>
>
>         This is where I think we diverge. :-) As someone with a fair
> amount of operational experience it''s not about the design of git,
it''s
> about the implementation created on top of git. Or Puppet.
>
>
>  I''d argue that people have stressed about DNS availability for
just
>>> under three decades and that we are currently enjoying the fruits
of that
>>> labor. Personally, I have yet to work at a company where DNS has
not
>>> caused
>>> a significant outage.
>>>
>>
>> I am really surprised at your statement. Of course mishaps can happen,
>> or someone can mess up configuration DNS royally. But setting up a
>> primary and secondary setup is trivial.
>>
>> SMTP and LDAP are also examples where resilience was baked into the
>> design. With those two, the quality of implementation, and
>> complications in setup make for a lot more breakage.
>>
>> Compare to HTTP, databases etc where there''s a whole industry
of tools
>> to make things somewhat reliable.
>>
>> Maybe we are talking about different things.
>>
>
>         Not different things, but perspectives.
>
>         I''d agree that your simple primary/secondary name server
is easy
> to setup and it''ll probably work just fine. However it supports a
very
> limited number of use cases and traffic levels.
>
>         My experience with DNS and administrating it in various
> incarnations since the bad old days of Bind 4 informs me that it can be
> incredibly fragile. It is only the implementation of the current DNS system
> that is reasonably resilient or at least able to localize failure.
> Certainly some designs and technology are better than others, but
> implementation always matters.
>
>         The same goes for just about system/protocol you''d care to
name.
>
>
>  Your ppg tooling does look interesting, but there is a large trade
>>> off in functionality
>>>
>>
>> What is the loss of functionality you see? Anything that you use in
>> practice?
>>
>> (Reading here https://groups.google.com/**forum/?fromgroups=#!topic/**
>>
puppet-users/7ZpAMrMb2NQ<https://groups.google.com/forum/?fromgroups=#!topic/puppet-users/7ZpAMrMb2NQ>
>> I can''t spot anything major, but I may be missing
something...)
>>
>> cheers,
>> m
>>
>
>         Masterless Puppet with git as a distribution method does have some
> things going for it as a design. You are giving up things like collected
> resources and standard reporting which may or may not matter you.
> Additionally you''re building a distribution system of some sort
even if
> it''s just git and ssh where you''d need to decide how to
deal with the
> failure of the parts.
>
>         In any case I''d like to see more discussion on highly
available
> Puppet regardless of way it''s implemented.
>
> Ramin
>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/**
>
topic/puppet-users/**Ze5QFJ95y3E/unsubscribe?hl=en<https://groups.google.com/d/topic/puppet-users/Ze5QFJ95y3E/unsubscribe?hl=en>
> .
> To unsubscribe from this group and all its topics, send an email to
>
puppet-users+unsubscribe@**googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at
http://groups.google.com/**group/puppet-users?hl=en<http://groups.google.com/group/puppet-users?hl=en>
> .
> For more options, visit
https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
> .
>
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Fri, May 10, 2013 at 1:52 PM, Ramin K <ramin-list@badapple.net>
wrote:
> reasonably resilient or at least able to localize failure. Certainly some
> designs and technology are better than others, but implementation always
> matters.
Of course. I think we''re saying the same thing, at the end of the day.

>         Masterless Puppet with git as a distribution method does have some
> things going for it as a design. You are giving up things like collected
> resources and standard reporting which may or may not matter you.
Thanks for these notes.

I am definitely supporting reporting -- finishing the implementation
today/Monday.

Reading the thread I mentioned before, ISTM that resources can be
trivially added with the same machinery.
> Additionally you''re building a distribution system of some sort
even if it''s
> just git and ssh where you''d need to decide how to deal with the
failure of
> the parts.
Absolutely, and that''s at the center of my design

 - It is a pull design (clients pull from a "gold" server, or through
"local proxy" servers)

 - If the pull fails, it''s a soft error; we soldier on. config changes
that have been fetched earlier still apply. importantly, config
changes that were fetched and have scheduling data have their
scheduled rollout still apply.

   For example, I make a commit with a change at midday today, and
schedule it for midnight. I publish it on the gold server. Clients
fetch it, but don''t apply it yet. Gold server goes down at 10pm --
this is a soft problem: clients will apply the changes
published&distributed earlier when their time comes.

 - Communication back to the gold server is store-and-forward, so that
midnight puppet run collects reports (and perhaps resources in the
future), and they are spooled locally until the gold server is ready
to receive them.
>         In any case I''d like to see more discussion on highly
available
> Puppet regardless of way it''s implemented.
+100



m
--
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don''t get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On 10 May 2013 19:52, Ramin K <ramin-list@badapple.net> wrote:
>
>         In any case I''d like to see more discussion on highly
available
> Puppet regardless of way it''s implemented.

We are using SRV records for running multiple puppetmasters and selecting a
site local but allowing fallback to others in case it is down.
We have 6 puppetmasters for the production environment running in this way
currently. Each normally handling 500-1000 nodes. The git repository is
push replicated to each one of them.

But only one is CA, it is backed up. If it would crash we are fine with
having a outage on installing new nodes until we have restored that part to
another node. But we have looked into some solutions for maybe making it
more resilient though.

For PuppetDB we have two service nodes and a master and hot standby for the
postgres database.

-- 
Erik Dalén

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

We have a similar setup, minus the SRV records (although that looks quire
interesting, gotta get off of 2.7). And we push SVN checkouts instead of git,
but that''s not a big difference.

I have been thinking about the CA, and how to make it more available. My first
thought is, do we have to save the generated client certs at all? I brought this
up a few weeks ago and the general answer was "there is no technical reason
to keep the certs", so I am considering deleting them immediately. Now I
don''t have to worry about backing up the puppetca!

Next, and this is where my SSL weakness  will shine, could you have all of your
HA-puppetmasters run as CAs, too, and then have multiple CA certs on trusted
list on the puppet masters? Something like this:
1. foo-east01 comes up, and gets an auto-signed vert from pm-east01.
2. pm-east01 hit by asteroid, so foo-east01 automatically fails over to
foo-west01
3. pm-west01 knows to trust the pm-east-01 signed cert.
4. We stand up a pm-east0.new1, generate a new vert for it and append said cert
to the trusted list for all clients/PMs.
5. foo-east01 starts using pm-east01.new again
6. foo-east02 comes up, gets a cert from pm-east01.new
(This is starting to feel like a certificate rotation strategy in some weird
way).

One thing I wonder is if I''ll actually be a little more secure. Instead
of having to have a single CA with a huge FW configuration (we have a lot of
independent networks across the ''net), each PM/CA has only a very
specific FW ruleset.
 
On May 14, 2013, at 7:35 AM, Erik Dalén <erik.gustav.dalen@gmail.com>
wrote:
> 
> 
> 
> On 10 May 2013 19:52, Ramin K <ramin-list@badapple.net> wrote:
> 
>         In any case I''d like to see more discussion on highly
available Puppet regardless of way it''s implemented.
> 
> We are using SRV records for running multiple puppetmasters and selecting a
site local but allowing fallback to others in case it is down.
> We have 6 puppetmasters for the production environment running in this way
currently. Each normally handling 500-1000 nodes. The git repository is push
replicated to each one of them.
> 
> But only one is CA, it is backed up. If it would crash we are fine with
having a outage on installing new nodes until we have restored that part to
another node. But we have looked into some solutions for maybe making it more
resilient though.
> 
> For PuppetDB we have two service nodes and a master and hot standby for the
postgres database.
> 
> -- 
> Erik Dalén
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Tue, May 14, 2013 at 7:35 AM, Erik Dalén <erik.gustav.dalen@gmail.com>
wrote:
> We are using SRV records for running multiple puppetmasters and selecting a
> site local but allowing fallback to others in case it is down.
> We have 6 puppetmasters for the production environment running in this way
> currently. Each normally handling 500-1000 nodes. The git repository is
push
> replicated to each one of them.
Interesting - do you use dashboard or anything similar to track the
state of the nodes?

cheers,




m
--
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don''t get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

If you don''t need to backup your puppetca, how do you carry over to a
standby puppetca server your client signed certificates and revocation list
in case of failure in the production puppetca ?


On Tue, May 14, 2013 at 8:04 AM, Mason Turner <opsmason@gmail.com> wrote:
> We have a similar setup, minus the SRV records (although that looks quire
> interesting, gotta get off of 2.7). And we push SVN checkouts instead of
> git, but that''s not a big difference.
>
> I have been thinking about the CA, and how to make it more available. My
> first thought is, do we have to save the generated client certs at all? I
> brought this up a few weeks ago and the general answer was "there is
no
> technical reason to keep the certs", so I am considering deleting them
> immediately. Now I don''t have to worry about backing up the
puppetca!
>
> Next, and this is where my SSL weakness  will shine, could you have all of
> your HA-puppetmasters run as CAs, too, and then have multiple CA certs on
> trusted list on the puppet masters? Something like this:
> 1. foo-east01 comes up, and gets an auto-signed vert from pm-east01.
> 2. pm-east01 hit by asteroid, so foo-east01 automatically fails over to
> foo-west01
> 3. pm-west01 knows to trust the pm-east-01 signed cert.
> 4. We stand up a pm-east0.new1, generate a new vert for it and append said
> cert to the trusted list for all clients/PMs.
> 5. foo-east01 starts using pm-east01.new again
> 6. foo-east02 comes up, gets a cert from pm-east01.new
> (This is starting to feel like a certificate rotation strategy in some
> weird way).
>
> One thing I wonder is if I''ll actually be a little more secure.
Instead of
> having to have a single CA with a huge FW configuration (we have a lot of
> independent networks across the ''net), each PM/CA has only a very
specific
> FW ruleset.
>
> On May 14, 2013, at 7:35 AM, Erik Dalén <erik.gustav.dalen@gmail.com>
> wrote:
>
>
>
>
> On 10 May 2013 19:52, Ramin K <ramin-list@badapple.net> wrote:
>
>>
>>         In any case I''d like to see more discussion on highly
available
>> Puppet regardless of way it''s implemented.
>
>
> We are using SRV records for running multiple puppetmasters and selecting
> a site local but allowing fallback to others in case it is down.
> We have 6 puppetmasters for the production environment running in this way
> currently. Each normally handling 500-1000 nodes. The git repository is
> push replicated to each one of them.
>
> But only one is CA, it is backed up. If it would crash we are fine with
> having a outage on installing new nodes until we have restored that part to
> another node. But we have looked into some solutions for maybe making it
> more resilient though.
>
> For PuppetDB we have two service nodes and a master and hot standby for
> the postgres database.
>
> --
> Erik Dalén
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
>
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit
>
https://groups.google.com/d/topic/puppet-users/Ze5QFJ95y3E/unsubscribe?hl=en
> .
> To unsubscribe from this group and all its topics, send an email to
> puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Hi ,

Thanks all for your knowledge sharing on mine Query .So after all I planned
to make Individual Puppet master for each locations .

Thanks Team


On Tue, May 14, 2013 at 8:56 PM, Felipe Salum <fsalum@gmail.com> wrote:
> If you don''t need to backup your puppetca, how do you carry over
to a
> standby puppetca server your client signed certificates and revocation list
> in case of failure in the production puppetca ?
>
>
> On Tue, May 14, 2013 at 8:04 AM, Mason Turner <opsmason@gmail.com>
wrote:
>
>> We have a similar setup, minus the SRV records (although that looks
quire
>> interesting, gotta get off of 2.7). And we push SVN checkouts instead
of
>> git, but that''s not a big difference.
>>
>> I have been thinking about the CA, and how to make it more available.
My
>> first thought is, do we have to save the generated client certs at all?
I
>> brought this up a few weeks ago and the general answer was "there
is no
>> technical reason to keep the certs", so I am considering deleting
them
>> immediately. Now I don''t have to worry about backing up the
puppetca!
>>
>> Next, and this is where my SSL weakness  will shine, could you have all
>> of your HA-puppetmasters run as CAs, too, and then have multiple CA
certs
>> on trusted list on the puppet masters? Something like this:
>> 1. foo-east01 comes up, and gets an auto-signed vert from pm-east01.
>> 2. pm-east01 hit by asteroid, so foo-east01 automatically fails over to
>> foo-west01
>> 3. pm-west01 knows to trust the pm-east-01 signed cert.
>> 4. We stand up a pm-east0.new1, generate a new vert for it and append
>> said cert to the trusted list for all clients/PMs.
>> 5. foo-east01 starts using pm-east01.new again
>> 6. foo-east02 comes up, gets a cert from pm-east01.new
>> (This is starting to feel like a certificate rotation strategy in some
>> weird way).
>>
>> One thing I wonder is if I''ll actually be a little more
secure. Instead
>> of having to have a single CA with a huge FW configuration (we have a
lot
>> of independent networks across the ''net), each PM/CA has only
a very
>> specific FW ruleset.
>>
>> On May 14, 2013, at 7:35 AM, Erik Dalén
<erik.gustav.dalen@gmail.com>
>> wrote:
>>
>>
>>
>>
>> On 10 May 2013 19:52, Ramin K <ramin-list@badapple.net> wrote:
>>
>>>
>>>         In any case I''d like to see more discussion on
highly available
>>> Puppet regardless of way it''s implemented.
>>
>>
>> We are using SRV records for running multiple puppetmasters and
selecting
>> a site local but allowing fallback to others in case it is down.
>> We have 6 puppetmasters for the production environment running in this
>> way currently. Each normally handling 500-1000 nodes. The git
repository is
>> push replicated to each one of them.
>>
>> But only one is CA, it is backed up. If it would crash we are fine with
>> having a outage on installing new nodes until we have restored that
part to
>> another node. But we have looked into some solutions for maybe making
it
>> more resilient though.
>>
>> For PuppetDB we have two service nodes and a master and hot standby for
>> the postgres database.
>>
>> --
>> Erik Dalén
>>
>> --
>> You received this message because you are subscribed to the Google
Groups
>> "Puppet Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send
an
>> email to puppet-users+unsubscribe@googlegroups.com.
>>
>> To post to this group, send email to puppet-users@googlegroups.com.
>> Visit this group at http://groups.google.com/group/puppet-users?hl=en.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>>
>>  --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Puppet Users" group.
>> To unsubscribe from this topic, visit
>>
https://groups.google.com/d/topic/puppet-users/Ze5QFJ95y3E/unsubscribe?hl=en
>> .
>> To unsubscribe from this group and all its topics, send an email to
>> puppet-users+unsubscribe@googlegroups.com.
>>
>> To post to this group, send email to puppet-users@googlegroups.com.
>> Visit this group at http://groups.google.com/group/puppet-users?hl=en.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.