Puppet users - Oct 2011 - Scalability, Dashboard/Foreman, and MCollective

I''ve been trying to find information, suggestions, etc. for how to
combine
scaling Puppet with the use of management tools like Dashboard/Foreman and
MCollective. Our current thinking for an initial deployment is two VMs as
Puppet Master worker nodes fronted by a pair of NetScalers. However,
I''m not
sure how to go about implementing Dashboard or Foreman and MCollective
servers in a redundant Puppet Master server environment. Can anyone speak to
their experience in this matter and/or point me to resources that discuss
such architectures?

FWIW, we''ll later implement multiple pairs of redundant Puppet servers
to
support environments we want managed separately, e.g. Production vs. Test.

Thanks,
Justin

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Oct 24, 5:38 pm, Justin Lloyd <jstn...@gmail.com>
wrote:
> I''ve been trying to find information, suggestions, etc. for how to
combine
> scaling Puppet with the use of management tools like Dashboard/Foreman and
> MCollective. Our current thinking for an initial deployment is two VMs as
> Puppet Master worker nodes fronted by a pair of NetScalers. However,
I''m not
> sure how to go about implementing Dashboard or Foreman and MCollective
> servers in a redundant Puppet Master server environment. Can anyone speak
to
> their experience in this matter and/or point me to resources that discuss
> such architectures?
>
> FWIW, we''ll later implement multiple pairs of redundant Puppet
servers to
> support environments we want managed separately, e.g. Production vs. Test.
>
> Thanks,
> Justin
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Excuse my first misclick ;)

On Oct 24, 5:38 pm, Justin Lloyd <jstn...@gmail.com>
wrote:
> I''ve been trying to find information, suggestions, etc. for how to
combine
> scaling Puppet with the use of management tools like Dashboard/Foreman and
> MCollective. Our current thinking for an initial deployment is two VMs as
> Puppet Master worker nodes fronted by a pair of NetScalers. However,
I''m not
> sure how to go about implementing Dashboard or Foreman and MCollective
> servers in a redundant Puppet Master server environment. Can anyone speak
to
> their experience in this matter and/or point me to resources that discuss
> such architectures?
Clustering Puppet Masters is relatively easy, you''ll have to think
about how you''re going to handle the shared Certificate Authority
problem though. I''ve run a cluster of PMs with a separate PM just to
act as the CA server and it worked well.

If you want to cluster Dashboard that should also be quite easy, you
just need a shared MySQL database to back onto. I''ve never used the
ENC capabilities of Dashboard so I can''t help you there, but for
Dashboard reporting, all PMs can send to any server so I''d have them
send to a Dashboard VIP on your load balancer and then you can balance
your report processing as well.

I can''t advise on The Foreman.

For MCollective, start reading about subcollectives[1] and Stomp
failover pools [2]. There''s no reason why each of your Puppet Masters
can''t also be Stomp servers. ActiveMQ has some nice abilities to send
messages between ActiveMQ servers that I started reading up on (useful
for when you start segregating dev and prod) but then I ran into
scalability issues with ActiveMQ so replaced it with RabbitMQ and
haven''t looked at what you can do with that yet.

Hope that''s a good starting point for you,

-Luke

[1] http://docs.puppetlabs.com/mcollective/reference/basic/subcollectives.html
[2]
http://docs.puppetlabs.com/mcollective/reference/plugins/connector_stomp.html
> FWIW, we''ll later implement multiple pairs of redundant Puppet
servers to
> support environments we want managed separately, e.g. Production vs. Test.
>
> Thanks,
> Justin
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

----- Original Message -----
> For MCollective, start reading about subcollectives[1] and Stomp
> failover pools [2]. There''s no reason why each of your Puppet
Masters
> can''t also be Stomp servers. ActiveMQ has some nice abilities to
send
> messages between ActiveMQ servers that I started reading up on
> (useful for when you start segregating dev and prod) but then I ran into
> scalability issues with ActiveMQ so replaced it with RabbitMQ and
> haven''t looked at what you can do with that yet.
at how many nodes did you have scalability issues with ActiveMQ? did you
do any tuning on it?

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mon, Oct 24, 2011 at 6:38 PM, Justin Lloyd <jstnlld@gmail.com>
wrote:
> I''ve been trying to find information, suggestions, etc. for how to
combine
> scaling Puppet with the use of management tools like Dashboard/Foreman and
> MCollective. Our current thinking for an initial deployment is two VMs as
> Puppet Master worker nodes fronted by a pair of NetScalers. However,
I''m not
> sure how to go about implementing Dashboard or Foreman and MCollective
> servers in a redundant Puppet Master server environment. Can anyone speak
to
> their experience in this matter and/or point me to resources that discuss
> such architectures?
>
To scale out foreman, would be very similar to any rails app, normally
you could:
1. use a load balancer across multiple foreman rails instances (e.g.
on multiple machines)
2. cluster your db

Foreman already supports both configuration (and has been implemented
by various community members with large scale deployments).

additionally, the enc script keeps a cache on each puppet master, so
in case when foreman is down, it would serve the last good enc data
that came from foreman.

hope this helps,
Ohad
> FWIW, we''ll later implement multiple pairs of redundant Puppet
servers to
> support environments we want managed separately, e.g. Production vs. Test.
>
> Thanks,
> Justin
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Oct 24, 6:20 pm, "R.I.Pienaar" <r...@devco.net>
wrote:
> ----- Original Message -----
> > For MCollective, start reading about subcollectives[1] and Stomp
> > failover pools [2]. There''s no reason why each of your Puppet
Masters
> > can''t also be Stomp servers. ActiveMQ has some nice abilities
to send
> > messages between ActiveMQ servers that I started reading up on
> > (useful for when you start segregating dev and prod) but then I ran
into
> > scalability issues with ActiveMQ so replaced it with RabbitMQ and
> > haven''t looked at what you can do with that yet.
>
> at how many nodes did you have scalability issues with ActiveMQ? did you
> do any tuning on it?
191 nodes to be exact, MCollective 1.3.1 and ActiveMQ 5.5 on a KVM VM
with 4GB RAM, 2GB allocated to ActiveMQ. Two of us spent the morning
tweaking ActiveMQ for better memory usage but weren''t achieving much.
With all nodes connected but no actual Stomp messages ever sent, JMX
was telling me the Java heap was from 900M to 1.3G used, that''s before
MCollective has even done anything. Broadcast just one message to the
estate and that would fill the heap and the broker would die. ActiveMQ
didn''t appear to be thread bound so [1] didn''t help too much
nor did
experimenting with various systemUsage values, and tips from [2] gave
little improvement.

I needed serious improvement as I was looking to connect another 300
nodes within a week. One of the developers here nodded and smiled when
I mentioned ActiveMQ memory issues and said to try RabbitMQ, which I
got working as a drop in replacement for ActiveMQ in under half an
hour with a 10th of the memory footprint. I may eventually go back to
ActiveMQ as the fine grained user access control seems a little more
feature rich than RabbitMQ does on the surface.

[1] http://www.pepperdust.org/?p=150
[2] http://activemq.apache.org/javalangoutofmemory.html

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

----- Original Message -----
> On Oct 24, 6:20 pm, "R.I.Pienaar" <r...@devco.net> wrote:
> > ----- Original Message -----
> > > For MCollective, start reading about subcollectives[1] and Stomp
> > > failover pools [2]. There''s no reason why each of your
Puppet
> > > Masters
> > > can''t also be Stomp servers. ActiveMQ has some nice
abilities to
> > > send
> > > messages between ActiveMQ servers that I started reading up on
> > > (useful for when you start segregating dev and prod) but then I
> > > ran into
> > > scalability issues with ActiveMQ so replaced it with RabbitMQ and
> > > haven''t looked at what you can do with that yet.
> >
> > at how many nodes did you have scalability issues with ActiveMQ?
> > did you
> > do any tuning on it?
> 
> 191 nodes to be exact, MCollective 1.3.1 and ActiveMQ 5.5 on a KVM VM
> with 4GB RAM, 2GB allocated to ActiveMQ. Two of us spent the morning
> tweaking ActiveMQ for better memory usage but weren''t achieving
much.
> With all nodes connected but no actual Stomp messages ever sent, JMX
> was telling me the Java heap was from 900M to 1.3G used, that''s
> before MCollective has even done anything. Broadcast just one message to
the
> estate and that would fill the heap and the broker would die.
> ActiveMQ didn''t appear to be thread bound so [1] didn''t
help too much nor did
> experimenting with various systemUsage values, and tips from [2] gave
> little improvement.
> 
> I needed serious improvement as I was looking to connect another 300
> nodes within a week. One of the developers here nodded and smiled
> when I mentioned ActiveMQ memory issues and said to try RabbitMQ, which I
> got working as a drop in replacement for ActiveMQ in under half an
> hour with a 10th of the memory footprint. I may eventually go back to
> ActiveMQ as the fine grained user access control seems a little more
> feature rich than RabbitMQ does on the surface.
Odd, that sounds suspiciously like some kind of messaging loop or something.
Guess without recreating it''s hard to say.  But def have bigger
networks than
that and they just work - physical kit though, ActiveMQ seems quite unhappy 
about dodgy clocks like you would find on VMWare instances.

It''s the old painful story of memory tuning on Java though, its too
complex
but the jvm does at least give you the data you need.

the symptom of using all memory prior to anything being done is normal though

-- 
R.I.Pienaar

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.