Antonio Xanxess
2011-Dec-16 12:56 UTC
[Puppet Users] Puppet capacity with apache+passenger, server saturated
Hi everyone, I have a problem with my Puppet server, the server is overloaded and processes need to add half of my farm nodes yet! The server is hosted on a blade with the following characteristics: Processors: 8 cores Memory Ram: 12 Gb Operating System: Debian Squeeze Version of parcel: 2.7.1-1 ~ bpo60 puppet 1 puppet-common 2.7.1-1 ~ bpo60 1 2.7.1-1 ~ bpo60 1 puppetmaster puppetmaster 2.7.1-1 ~ bpo60-1 common puppet vim 2.7.1-1 ~ bpo60-1 The number of nodes that I have now is 1927 and I''m in the middle of the deployment process. The nodes do not use the demon puppet, but I created a cron task that runs dispersed to run in one hour intervals. With these data the server is currently using on average 4 cores, but the RAM is used almost entirely (it has 1 GB free of 12) I would like to know some technical expert for the optimization of services, anything you can think of will help me because I am a bit blocked .... I''ve been thinking about creating another puppetmaster and use a load balancer, but in my case would have security problems because the connections are behind the balancer and servers unencrypted would be in different physical locations, having to go unencrypted information by the network. Thank you and if you need some data from the server will put. A greeting! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Brice Figureau
2011-Dec-16 14:36 UTC
Re: [Puppet Users] Puppet capacity with apache+passenger, server saturated
On Fri, 2011-12-16 at 04:56 -0800, Antonio Xanxess wrote:> I have a problem with my Puppet server, the server is overloaded and > processes need to add half of my farm nodes yet! > The server is hosted on a blade with the following characteristics: > Processors: 8 cores > Memory Ram: 12 Gb > Operating System: Debian Squeeze > Version of parcel: > 2.7.1-1 ~ bpo60 puppet 1 > puppet-common 2.7.1-1 ~ bpo60 1 > 2.7.1-1 ~ bpo60 1 puppetmaster > puppetmaster 2.7.1-1 ~ bpo60-1 common > puppet vim 2.7.1-1 ~ bpo60-1 > > The number of nodes that I have now is 1927 and I''m in the middle of > the deployment process. The nodes do not use the demon puppet, but I > created a cron task that runs dispersed to run in one hour intervals. > With these data the server is currently using on average 4 cores, but > the RAM is used almost entirely (it has 1 GB free of 12)If your setup is consuming so much ram, that means your concurrency is too high. You might also benefit of using Ruby Enterprise Edition, because passenger might be able to share more memory between those ruby processes. Puppetmasters are mostly CPU bound. Since your server has 8 cores, having more than 8, 12 or 16 puppetmaster processes at one will not give you more horse power (yeah CPU is a finite quantity). What is your current concurrency? If your problem is that your CPU is constantly at 100% because of the puppet runs (ie too many clients checking-in at the same time), then I''m afraid you''ll either: * need another server * increase the time between your client check in (every 60min in lieu of 30 for instance) * use of Mcollective Puppet Controller [1] * my experimental catalog caching [2] You can do some math to see if your near the limit: if you have 1927 nodes each checking-in every 30 minutes, that means you (if spread evenly) 1927/30=64 nodes checking in every minute. That''s more than 1 node coming in per second. Since you have a concurrency of 8 (because of your 8 core) at max, you can accomodate 8 nodes at the same time. Since 64/8 = 8, you can accomodate 8 client per minute, which means each client must be serviced in less than 60/8=7.5s or you''ll run into trouble. And you can even do better than math, you can benchmark your master (better a test server with the same capacity) [3]> I would like to know some technical expert for the optimization of > services, anything you can think of will help me because I am a bit > blocked .... > I''ve been thinking about creating another puppetmaster and use a load > balancer, but in my case would have security problems because the > connections are behind the balancer and servers unencrypted would be > in different physical locations, having to go unencrypted information > by the network.You can certainly load balance SSL connections without having to decrypt the content (ie a pure layer 3-4 load-balancer). I''m sure haproxy would be able to do that. If you go this route, make sure you have a standalone CA and point your clients to this standalone CA (that''s way simpler than trying to replicate your CA between your masters). [1]: http://www.devco.net/archives/2010/03/17/scheduling_puppet_with_mcollective.php [2]: http://www.masterzen.fr/2010/03/21/more-puppet-offloading/ [3]: http://www.masterzen.fr/2010/10/18/benchmarking-puppetmaster-stacks/ -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Antonio Xanxess
2011-Dec-20 09:35 UTC
[Puppet Users] Re: Puppet capacity with apache+passenger, server saturated
Hi Brice,
Thanks for the reply. I''ve been looking at what you sent me and now I
think I meet all these requirements.
Currently, each node accesses the settings every hour, which means
that from 1927 to 1960 = 32 nodes checking every minute.
I use Passenger with apache, maybe it''s the best solution I''ve
also
read that Nginx is lighter than apache, it would be a solution?
I recently had a problem observed in the apache logs, in which he said
he had to increase the MaxClients variable, so I decided to upload it.
I attached the module worker (currently I have installed apache-mpm-
worker) if maybe one of these parameters are wrong.
<IfModule mpm_worker_module>
StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxClients 400
MaxRequestsPerChild 0
</ IfModule>
With this configuration, the service works perfectly, but I think that
has increased exponentially, my consumption of RAM, I leave the data
extracted from Munin:
apps: 10Gb
page_tables: 48.91 Mb
Swap_Cache: 7.24 Mb
Slab_Cache: 110.55 Mb
Cache: 565.15 Mb
Buffers: 62.67 Mb
Unused: 243.82 Mb
Swap: 29.40 Mb
Inactive: 1.68 Gb
Committed: 12.91 GB
Active: 9.64 Gb
Vmalloc_Used: 39.53 Mb
Mapped: 13.68 Mb
As a preventive measure to continue the deployment of the application
I will dilate much execution interval to every two hours scattered
through a seed (as I have done with cron minutes to the dispersion of
nodes), also occurred to me to use the variable use_cached_catalog,
but I doubt arises, if I add this variable nodes use the catalog
caching, meaning that if you change the manifest to the server nodes
are not updated? There would be some way to tell the node to use its
cached catalog until they have updates on the server?
Thank you for everything! ;)
Best regards!
On 16 dic, 15:36, Brice Figureau <brice-pup...@daysofwonder.com>
wrote:> On Fri, 2011-12-16 at 04:56 -0800, Antonio Xanxess wrote:
> > I have a problem with my Puppet server, the server is overloaded and
> > processes need to add half of my farm nodes yet!
> > The server is hosted on a blade with the following characteristics:
> > Processors: 8 cores
> > Memory Ram: 12 Gb
> > Operating System: Debian Squeeze
> > Version of parcel:
> > 2.7.1-1 ~ bpo60 puppet 1
> > puppet-common 2.7.1-1 ~ bpo60 1
> > 2.7.1-1 ~ bpo60 1 puppetmaster
> > puppetmaster 2.7.1-1 ~ bpo60-1 common
> > puppet vim 2.7.1-1 ~ bpo60-1
>
> > The number of nodes that I have now is 1927 and I''m in the
middle of
> > the deployment process. The nodes do not use the demon puppet, but I
> > created a cron task that runs dispersed to run in one hour intervals.
> > With these data the server is currently using on average 4 cores, but
> > the RAM is used almost entirely (it has 1 GB free of 12)
>
> If your setup is consuming so much ram, that means your concurrency is
> too high. You might also benefit of using Ruby Enterprise Edition,
> because passenger might be able to share more memory between those ruby
> processes.
>
> Puppetmasters are mostly CPU bound. Since your server has 8 cores,
> having more than 8, 12 or 16 puppetmaster processes at one will not give
> you more horse power (yeah CPU is a finite quantity).
>
> What is your current concurrency?
>
> If your problem is that your CPU is constantly at 100% because of the
> puppet runs (ie too many clients checking-in at the same time), then
I''m
> afraid you''ll either:
>
> * need another server
> * increase the time between your client check in (every 60min in lieu of
> 30 for instance)
> * use of Mcollective Puppet Controller [1]
> * my experimental catalog caching [2]
>
> You can do some math to see if your near the limit: if you have 1927
> nodes each checking-in every 30 minutes, that means you (if spread
> evenly) 1927/30=64 nodes checking in every minute.
> That''s more than 1 node coming in per second.
> Since you have a concurrency of 8 (because of your 8 core) at max, you
> can accomodate 8 nodes at the same time.
> Since 64/8 = 8, you can accomodate 8 client per minute, which means each
> client must be serviced in less than 60/8=7.5s or you''ll run into
> trouble.
>
> And you can even do better than math, you can benchmark your master
> (better a test server with the same capacity) [3]
>
> > I would like to know some technical expert for the optimization of
> > services, anything you can think of will help me because I am a bit
> > blocked ....
> > I''ve been thinking about creating another puppetmaster and
use a load
> > balancer, but in my case would have security problems because the
> > connections are behind the balancer and servers unencrypted would be
> > in different physical locations, having to go unencrypted information
> > by the network.
>
> You can certainly load balance SSL connections without having to decrypt
> the content (ie a pure layer 3-4 load-balancer). I''m sure haproxy
would
> be able to do that.
> If you go this route, make sure you have a standalone CA and point your
> clients to this standalone CA (that''s way simpler than trying to
> replicate your CA between your masters).
>
>
[1]:http://www.devco.net/archives/2010/03/17/scheduling_puppet_with_mcoll...
> [2]:http://www.masterzen.fr/2010/03/21/more-puppet-offloading/
> [3]:http://www.masterzen.fr/2010/10/18/benchmarking-puppetmaster-stacks/
> --
> Brice Figureau
> Follow the latest Puppet Community evolutions onwww.planetpuppet.org!
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Trevor Vaughan
2011-Dec-20 13:46 UTC
Re: [Puppet Users] Puppet capacity with apache+passenger, server saturated
One other thing to look at is how many nodes are hitting your server at once. If you''re not using the ip_to_cron function from the Cron Patterns page, you might want to take a look at it. http://projects.puppetlabs.com/projects/1/wiki/Cron_Patterns Some people are also using MCollective to to a more controlled call-in of nodes. Trevor On Fri, Dec 16, 2011 at 7:56 AM, Antonio Xanxess <antonio.sanchez.aguilar@gmail.com> wrote:> Hi everyone, > > I have a problem with my Puppet server, the server is overloaded and > processes need to add half of my farm nodes yet! > The server is hosted on a blade with the following characteristics: > Processors: 8 cores > Memory Ram: 12 Gb > Operating System: Debian Squeeze > Version of parcel: > 2.7.1-1 ~ bpo60 puppet 1 > puppet-common 2.7.1-1 ~ bpo60 1 > 2.7.1-1 ~ bpo60 1 puppetmaster > puppetmaster 2.7.1-1 ~ bpo60-1 common > puppet vim 2.7.1-1 ~ bpo60-1 > > The number of nodes that I have now is 1927 and I''m in the middle of > the deployment process. The nodes do not use the demon puppet, but I > created a cron task that runs dispersed to run in one hour intervals. > With these data the server is currently using on average 4 cores, but > the RAM is used almost entirely (it has 1 GB free of 12) > I would like to know some technical expert for the optimization of > services, anything you can think of will help me because I am a bit > blocked .... > I''ve been thinking about creating another puppetmaster and use a load > balancer, but in my case would have security problems because the > connections are behind the balancer and servers unencrypted would be > in different physical locations, having to go unencrypted information > by the network. > > Thank you and if you need some data from the server will put. > > A greeting! > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >-- Trevor Vaughan Vice President, Onyx Point, Inc (410) 541-6699 tvaughan@onyxpoint.com -- This account not approved for unencrypted proprietary information -- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Jeffrey Watts
2011-Dec-20 18:26 UTC
Re: [Puppet Users] Puppet capacity with apache+passenger, server saturated
Here''s how I do it, which IMHO works really well in most situations and
is
much simpler. No external scripts, no custom functions. We take the IP
address of the system, strip out the periods, and modulo 30 it:
$cron_time1= generate(''/usr/bin/env'',
''sh'', ''-c'', "printf $(($(echo
$::ipaddress | awk -F . \''{print \$1+\$2+\$3+\$4}\'') %
30))")
$cron_time2 = $cron_time1 + 30
Now, that''s if you want to have your systems check in twice per hour.
You
can adjust it accordingly if you want quicker or longer check-in intervals.
Jeffrey.
On Tue, Dec 20, 2011 at 7:46 AM, Trevor Vaughan
<tvaughan@onyxpoint.com>wrote:
> One other thing to look at is how many nodes are hitting your server at
> once.
>
> If you''re not using the ip_to_cron function from the Cron Patterns
> page, you might want to take a look at it.
>
> http://projects.puppetlabs.com/projects/1/wiki/Cron_Patterns
>
> Some people are also using MCollective to to a more controlled call-in of
> nodes.
>
> Trevor
>
> On Fri, Dec 16, 2011 at 7:56 AM, Antonio Xanxess
> <antonio.sanchez.aguilar@gmail.com> wrote:
> > Hi everyone,
> >
> > I have a problem with my Puppet server, the server is overloaded and
> > processes need to add half of my farm nodes yet!
> > The server is hosted on a blade with the following characteristics:
> > Processors: 8 cores
> > Memory Ram: 12 Gb
> > Operating System: Debian Squeeze
> > Version of parcel:
> > 2.7.1-1 ~ bpo60 puppet 1
> > puppet-common 2.7.1-1 ~ bpo60 1
> > 2.7.1-1 ~ bpo60 1 puppetmaster
> > puppetmaster 2.7.1-1 ~ bpo60-1 common
> > puppet vim 2.7.1-1 ~ bpo60-1
> >
> > The number of nodes that I have now is 1927 and I''m in the
middle of
> > the deployment process. The nodes do not use the demon puppet, but I
> > created a cron task that runs dispersed to run in one hour intervals.
> > With these data the server is currently using on average 4 cores, but
> > the RAM is used almost entirely (it has 1 GB free of 12)
> > I would like to know some technical expert for the optimization of
> > services, anything you can think of will help me because I am a bit
> > blocked ....
> > I''ve been thinking about creating another puppetmaster and
use a load
> > balancer, but in my case would have security problems because the
> > connections are behind the balancer and servers unencrypted would be
> > in different physical locations, having to go unencrypted information
> > by the network.
> >
> > Thank you and if you need some data from the server will put.
> >
> > A greeting!
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> > To post to this group, send email to puppet-users@googlegroups.com.
> > To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> > For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
> >
>
>
>
> --
> Trevor Vaughan
> Vice President, Onyx Point, Inc
> (410) 541-6699
> tvaughan@onyxpoint.com
>
> -- This account not approved for unencrypted proprietary information --
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Antonio Xanxess
2011-Dec-21 13:01 UTC
[Puppet Users] Re: Puppet capacity with apache+passenger, server saturated
Hello everyone,
Trevor and Jeffrey Thanks for the information but I have included
puppet task cron dispersed, what I want is to optimize the service as
much as possible and discharged the RAM since I have yet to get around
more than 2000 nodes.
I included in the message above two questions I''d like to clarify:
1 - The configuration of Apache, I think I have currently values may
be affecting the service, I attached it again:
<IfModule mpm_worker_module>
StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxClients 400
MaxRequestsPerChild 0
</ IfModule>
2 - Use variable puppet.conf used_cached_catalog in the file, I have
my doubts about it, I ask again: meaning that if you change the
manifest to the server nodes are not updated? There would be some way
to tell the node to use its cached catalog until they have updates on
the server?
Thank you for everything!
Best regards!
On 20 dic, 19:26, Jeffrey Watts <jeffrey.w.wa...@gmail.com>
wrote:> Here''s how I do it, which IMHO works really well in most
situations and is
> much simpler. No external scripts, no custom functions. We take the IP
> address of the system, strip out the periods, and modulo 30 it:
>
> $cron_time1= generate(''/usr/bin/env'',
''sh'', ''-c'', "printf $(($(echo
> $::ipaddress | awk -F . \''{print \$1+\$2+\$3+\$4}\'') %
30))")
> $cron_time2 = $cron_time1 + 30
>
> Now, that''s if you want to have your systems check in twice per
hour. You
> can adjust it accordingly if you want quicker or longer check-in intervals.
>
> Jeffrey.
>
> On Tue, Dec 20, 2011 at 7:46 AM, Trevor Vaughan
<tvaug...@onyxpoint.com>wrote:
>
>
>
>
>
>
>
> > One other thing to look at is how many nodes are hitting your server
at
> > once.
>
> > If you''re not using the ip_to_cron function from the Cron
Patterns
> > page, you might want to take a look at it.
>
> >http://projects.puppetlabs.com/projects/1/wiki/Cron_Patterns
>
> > Some people are also using MCollective to to a more controlled call-in
of
> > nodes.
>
> > Trevor
>
> > On Fri, Dec 16, 2011 at 7:56 AM, Antonio Xanxess
> > <antonio.sanchez.agui...@gmail.com> wrote:
> > > Hi everyone,
>
> > > I have a problem with my Puppet server, the server is overloaded
and
> > > processes need to add half of my farm nodes yet!
> > > The server is hosted on a blade with the following
characteristics:
> > > Processors: 8 cores
> > > Memory Ram: 12 Gb
> > > Operating System: Debian Squeeze
> > > Version of parcel:
> > > 2.7.1-1 ~ bpo60 puppet 1
> > > puppet-common 2.7.1-1 ~ bpo60 1
> > > 2.7.1-1 ~ bpo60 1 puppetmaster
> > > puppetmaster 2.7.1-1 ~ bpo60-1 common
> > > puppet vim 2.7.1-1 ~ bpo60-1
>
> > > The number of nodes that I have now is 1927 and I''m in
the middle of
> > > the deployment process. The nodes do not use the demon puppet,
but I
> > > created a cron task that runs dispersed to run in one hour
intervals.
> > > With these data the server is currently using on average 4 cores,
but
> > > the RAM is used almost entirely (it has 1 GB free of 12)
> > > I would like to know some technical expert for the optimization
of
> > > services, anything you can think of will help me because I am a
bit
> > > blocked ....
> > > I''ve been thinking about creating another puppetmaster
and use a load
> > > balancer, but in my case would have security problems because the
> > > connections are behind the balancer and servers unencrypted would
be
> > > in different physical locations, having to go unencrypted
information
> > > by the network.
>
> > > Thank you and if you need some data from the server will put.
>
> > > A greeting!
>
> > > --
> > > You received this message because you are subscribed to the
Google
> > Groups "Puppet Users" group.
> > > To post to this group, send email to
puppet-users@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > puppet-users+unsubscribe@googlegroups.com.
> > > For more options, visit this group at
> >http://groups.google.com/group/puppet-users?hl=en.
>
> > --
> > Trevor Vaughan
> > Vice President, Onyx Point, Inc
> > (410) 541-6699
> > tvaug...@onyxpoint.com
>
> > -- This account not approved for unencrypted proprietary information
--
>
> > --
> > You received this message because you are subscribed to the Google
Groups
> > "Puppet Users" group.
> > To post to this group, send email to puppet-users@googlegroups.com.
> > To unsubscribe from this group, send email to
> > puppet-users+unsubscribe@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/puppet-users?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Gabriel Filion
2011-Dec-21 13:56 UTC
Re: [Puppet Users] Re: Puppet capacity with apache+passenger, server saturated
Hello, On 11-12-21 08:01 AM, Antonio Xanxess wrote:> MinSpareThreads 25 > MaxSpareThreads 75> MaxRequestsPerChild 0How much RAM does each Apache process take? Does Apache take more and more memory up to the point you mentioned (11Gb taken)? With luck, if you set MaxRequestsPerChild to something like 1000 (or maybe a little lower) your processes will be able to clear out their memory from time to time.. It''s possible that it doesn''t really help that much, but it could give you a little more space. Also, depending on how much clients hit your server at peak concurrency, you could lower MaxSpareThreads a little to have Apache kill more loose processes (and thus regain their memory faster). -- Gabriel Filion -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Ramin K
2011-Dec-22 00:50 UTC
[Puppet Users] Re: Puppet capacity with apache+passenger, server saturated
Apache or any other http server is usually not the problem in a Passenger based Puppet master because it''s only peripherally involved in the actual work. Here''s roughly how a client receives a catalog. Client runs, makes a connections to Apache on the Puppet master Puppet master Apache receives, negotiates ssl, etc. Client gets/posts or whatever it does these days. Apache looks at request and Puppet vhost and says "This is Rack request" Apache hands the whole thing to mod_passenger Passenger looks at request and says "Do I have a Rack processes running?" If it does and they aren''t busy, it passes the request to one of the Puppet Rack processes If it does not or the current one is busy, it spawns a new Rack process to take the request Passenger will spawn Rack processes to the limit you''ve set or until it runs out of RAM. On my Puppet master I''m running four Rack processes at 200MB each, a 150MB one for the Dashboard, and Apache is using 300MB. Because my server doesn''t have much CPU I limit the total number of Rack processes to six. In any case what we should look at is the passenger.conf and any Passenger settings in the Puppet master vhost. Ramin On Dec 21, 5:01 am, Antonio Xanxess <antonio.sanchez.agui...@gmail.com> wrote:> Hello everyone, > > Trevor and Jeffrey Thanks for the information but I have included > puppet task cron dispersed, what I want is to optimize the service as > much as possible and discharged the RAM since I have yet to get around > more than 2000 nodes. > I included in the message above two questions I''d like to clarify: > 1 - The configuration of Apache, I think I have currently values may > be affecting the service, I attached it again: > > <IfModule mpm_worker_module> > StartServers 2 > MinSpareThreads 25 > MaxSpareThreads 75 > ThreadLimit 64 > ThreadsPerChild 25 > MaxClients 400 > MaxRequestsPerChild 0 > </ IfModule> > > 2 - Use variable puppet.conf used_cached_catalog in the file, I have > my doubts about it, I ask again: meaning that if you change the > manifest to the server nodes are not updated? There would be some way > to tell the node to use its cached catalog until they have updates on > the server? > > Thank you for everything! > > Best regards! > > On 20 dic, 19:26, Jeffrey Watts <jeffrey.w.wa...@gmail.com> wrote: > > > > > > > > > Here''s how I do it, which IMHO works really well in most situations and is > > much simpler. No external scripts, no custom functions. We take the IP > > address of the system, strip out the periods, and modulo 30 it: > > > $cron_time1= generate(''/usr/bin/env'', ''sh'', ''-c'', "printf $(($(echo > > $::ipaddress | awk -F . \''{print \$1+\$2+\$3+\$4}\'') % 30))") > > $cron_time2 = $cron_time1 + 30 > > > Now, that''s if you want to have your systems check in twice per hour. You > > can adjust it accordingly if you want quicker or longer check-in intervals. > > > Jeffrey. > > > On Tue, Dec 20, 2011 at 7:46 AM, Trevor Vaughan <tvaug...@onyxpoint.com>wrote: > > > > One other thing to look at is how many nodes are hitting your server at > > > once. > > > > If you''re not using the ip_to_cron function from the Cron Patterns > > > page, you might want to take a look at it. > > > >http://projects.puppetlabs.com/projects/1/wiki/Cron_Patterns > > > > Some people are also using MCollective to to a more controlled call-in of > > > nodes. > > > > Trevor > > > > On Fri, Dec 16, 2011 at 7:56 AM, Antonio Xanxess > > > <antonio.sanchez.agui...@gmail.com> wrote: > > > > Hi everyone, > > > > > I have a problem with my Puppet server, the server is overloaded and > > > > processes need to add half of my farm nodes yet! > > > > The server is hosted on a blade with the following characteristics: > > > > Processors: 8 cores > > > > Memory Ram: 12 Gb > > > > Operating System: Debian Squeeze > > > > Version of parcel: > > > > 2.7.1-1 ~ bpo60 puppet 1 > > > > puppet-common 2.7.1-1 ~ bpo60 1 > > > > 2.7.1-1 ~ bpo60 1 puppetmaster > > > > puppetmaster 2.7.1-1 ~ bpo60-1 common > > > > puppet vim 2.7.1-1 ~ bpo60-1 > > > > > The number of nodes that I have now is 1927 and I''m in the middle of > > > > the deployment process. The nodes do not use the demon puppet, but I > > > > created a cron task that runs dispersed to run in one hour intervals. > > > > With these data the server is currently using on average 4 cores, but > > > > the RAM is used almost entirely (it has 1 GB free of 12) > > > > I would like to know some technical expert for the optimization of > > > > services, anything you can think of will help me because I am a bit > > > > blocked .... > > > > I''ve been thinking about creating another puppetmaster and use a load > > > > balancer, but in my case would have security problems because the > > > > connections are behind the balancer and servers unencrypted would be > > > > in different physical locations, having to go unencrypted information > > > > by the network. > > > > > Thank you and if you need some data from the server will put. > > > > > A greeting! > > > > > -- > > > > You received this message because you are subscribed to the Google > > > Groups "Puppet Users" group. > > > > To post to this group, send email to puppet-users@googlegroups.com. > > > > To unsubscribe from this group, send email to > > > puppet-users+unsubscribe@googlegroups.com. > > > > For more options, visit this group at > > >http://groups.google.com/group/puppet-users?hl=en. > > > > -- > > > Trevor Vaughan > > > Vice President, Onyx Point, Inc > > > (410) 541-6699 > > > tvaug...@onyxpoint.com > > > > -- This account not approved for unencrypted proprietary information -- > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "Puppet Users" group. > > > To post to this group, send email to puppet-users@googlegroups.com. > > > To unsubscribe from this group, send email to > > > puppet-users+unsubscribe@googlegroups.com. > > > For more options, visit this group at > > >http://groups.google.com/group/puppet-users?hl=en.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Antonio Xanxess
2012-Jan-11 10:09 UTC
[Puppet Users] Re: Puppet capacity with apache+passenger, server saturated
Hello everyone, Ramin and Gabriel Thanks for the answer and feel the delay in replying. Solve the problem of abusive use of memory by modifying some parameters of passenger, shortening the lifetime of instances of puppetmaster and performing a triple nodes dispersion: Dispersion time (the nodes spend their settings every 2 hours), dispersion in minutes (establishing a random minute from a seed file) and dispersion in seconds (by performing a random sleep before calling puppetd), so I get the machine down their consumption noticeably. Again thank you very much to all and happy new year! A greeting! On 22 dic 2011, 01:50, Ramin K <ramin.khat...@gmail.com> wrote:> Apache or any other http server is usually not the problem in a > Passenger based Puppet master because it''s only peripherally involved > in the actual work. Here''s roughly how a client receives a catalog. > > Client runs, makes a connections to Apache on the Puppet master > Puppet master Apache receives, negotiates ssl, etc. > Client gets/posts or whatever it does these days. > > Apache looks at request and Puppet vhost and says "This is Rack > request" > Apache hands the whole thing to mod_passenger > Passenger looks at request and says "Do I have a Rack processes > running?" > If it does and they aren''t busy, it passes the request to one of the > Puppet Rack processes > If it does not or the current one is busy, it spawns a new Rack > process to take the request > > Passenger will spawn Rack processes to the limit you''ve set or until > it runs out of RAM. On my Puppet master I''m running four Rack > processes at 200MB each, a 150MB one for the Dashboard, and Apache is > using 300MB. Because my server doesn''t have much CPU I limit the total > number of Rack processes to six. > > In any case what we should look at is the passenger.conf and any > Passenger settings in the Puppet master vhost. > > Ramin > > On Dec 21, 5:01 am, Antonio Xanxess > > > > > > > > <antonio.sanchez.agui...@gmail.com> wrote: > > Hello everyone, > > > Trevor and Jeffrey Thanks for the information but I have included > > puppet task cron dispersed, what I want is to optimize the service as > > much as possible and discharged the RAM since I have yet to get around > > more than 2000 nodes. > > I included in the message above two questions I''d like to clarify: > > 1 - The configuration of Apache, I think I have currently values may > > be affecting the service, I attached it again: > > > <IfModule mpm_worker_module> > > StartServers 2 > > MinSpareThreads 25 > > MaxSpareThreads 75 > > ThreadLimit 64 > > ThreadsPerChild 25 > > MaxClients 400 > > MaxRequestsPerChild 0 > > </ IfModule> > > > 2 - Use variable puppet.conf used_cached_catalog in the file, I have > > my doubts about it, I ask again: meaning that if you change the > > manifest to the server nodes are not updated? There would be some way > > to tell the node to use its cached catalog until they have updates on > > the server? > > > Thank you for everything! > > > Best regards! > > > On 20 dic, 19:26, Jeffrey Watts <jeffrey.w.wa...@gmail.com> wrote: > > > > Here''s how I do it, which IMHO works really well in most situations and is > > > much simpler. No external scripts, no custom functions. We take the IP > > > address of the system, strip out the periods, and modulo 30 it: > > > > $cron_time1= generate(''/usr/bin/env'', ''sh'', ''-c'', "printf $(($(echo > > > $::ipaddress | awk -F . \''{print \$1+\$2+\$3+\$4}\'') % 30))") > > > $cron_time2 = $cron_time1 + 30 > > > > Now, that''s if you want to have your systems check in twice per hour. You > > > can adjust it accordingly if you want quicker or longer check-in intervals. > > > > Jeffrey. > > > > On Tue, Dec 20, 2011 at 7:46 AM, Trevor Vaughan <tvaug...@onyxpoint.com>wrote: > > > > > One other thing to look at is how many nodes are hitting your server at > > > > once. > > > > > If you''re not using the ip_to_cron function from the Cron Patterns > > > > page, you might want to take a look at it. > > > > >http://projects.puppetlabs.com/projects/1/wiki/Cron_Patterns > > > > > Some people are also using MCollective to to a more controlled call-in of > > > > nodes. > > > > > Trevor > > > > > On Fri, Dec 16, 2011 at 7:56 AM, Antonio Xanxess > > > > <antonio.sanchez.agui...@gmail.com> wrote: > > > > > Hi everyone, > > > > > > I have a problem with my Puppet server, the server is overloaded and > > > > > processes need to add half of my farm nodes yet! > > > > > The server is hosted on a blade with the following characteristics: > > > > > Processors: 8 cores > > > > > Memory Ram: 12 Gb > > > > > Operating System: Debian Squeeze > > > > > Version of parcel: > > > > > 2.7.1-1 ~ bpo60 puppet 1 > > > > > puppet-common 2.7.1-1 ~ bpo60 1 > > > > > 2.7.1-1 ~ bpo60 1 puppetmaster > > > > > puppetmaster 2.7.1-1 ~ bpo60-1 common > > > > > puppet vim 2.7.1-1 ~ bpo60-1 > > > > > > The number of nodes that I have now is 1927 and I''m in the middle of > > > > > the deployment process. The nodes do not use the demon puppet, but I > > > > > created a cron task that runs dispersed to run in one hour intervals. > > > > > With these data the server is currently using on average 4 cores, but > > > > > the RAM is used almost entirely (it has 1 GB free of 12) > > > > > I would like to know some technical expert for the optimization of > > > > > services, anything you can think of will help me because I am a bit > > > > > blocked .... > > > > > I''ve been thinking about creating another puppetmaster and use a load > > > > > balancer, but in my case would have security problems because the > > > > > connections are behind the balancer and servers unencrypted would be > > > > > in different physical locations, having to go unencrypted information > > > > > by the network. > > > > > > Thank you and if you need some data from the server will put. > > > > > > A greeting! > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > Groups "Puppet Users" group. > > > > > To post to this group, send email to puppet-users@googlegroups.com. > > > > > To unsubscribe from this group, send email to > > > > puppet-users+unsubscribe@googlegroups.com. > > > > > For more options, visit this group at > > > >http://groups.google.com/group/puppet-users?hl=en. > > > > > -- > > > > Trevor Vaughan > > > > Vice President, Onyx Point, Inc > > > > (410) 541-6699 > > > > tvaug...@onyxpoint.com > > > > > -- This account not approved for unencrypted proprietary information -- > > > > > -- > > > > You received this message because you are subscribed to the Google Groups > > > > "Puppet Users" group. > > > > To post to this group, send email to puppet-users@googlegroups.com. > > > > To unsubscribe from this group, send email to > > > > puppet-users+unsubscribe@googlegroups.com. > > > > For more options, visit this group at > > > >http://groups.google.com/group/puppet-users?hl=en.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Daniel Pittman
2012-Jan-12 22:39 UTC
Re: [Puppet Users] Re: Puppet capacity with apache+passenger, server saturated
On Wed, Jan 11, 2012 at 02:09, Antonio Xanxess <antonio.sanchez.aguilar@gmail.com> wrote:> Ramin and Gabriel Thanks for the answer and feel the delay in > replying. > Solve the problem of abusive use of memory by modifying some > parameters of passenger, shortening the lifetime of instances of > puppetmaster and performing a triple nodes dispersion: Dispersion time > (the nodes spend their settings every 2 hours), dispersion in minutes > (establishing a random minute from a seed file) and dispersion in > seconds (by performing a random sleep before calling puppetd), so I > get the machine down their consumption noticeably.FWIW, the `puppetcommander` feature that MCollective supports is an awesome way to do this - it gives you a central point responsible for making sure that load is spread about your network as efficiently as possible. -- Daniel Pittman ⎋ Puppet Labs Developer – http://puppetlabs.com ♲ Made with 100 percent post-consumer electrons -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.