Eric
2012-May-15 21:06 UTC
[Puppet Users] Connection reset by peer errors when performing puppet agent configuration run
I am having an issue with my Puppet agents receiving "connection reset by
peer" errors. I only have 4 puppet agent systems in my environment right
now. I have two Puppet Master servers running Apache HTTPD with
mod_passenger. I have disabled the "puppetmaster" service and am only
running HTTPD. See below for my configuration. Occasionally throughout the
day I will see the following error when a Puppet agent tries to perform a
configuration run:
err:
/Stage[main]/Apr-tag::Config/File[/usr/local/scripts/nrpe/o2_log_check.sh]:
Could not evaluate: Connection reset by peer Could not retrieve file
metadata for
puppet:///modules/apr-tag/usr/local/scripts/nrpe/o2_log_check.sh:
Connection reset by peer at
/etc/puppet/modules/apr-tag/manifests/init.pp:249
The file that it can not retrieve changes, but the error remains the same.
I see the following error message in Apache''s error_log:
[ pid=24204 thr=139789832615904 file=ext/apache2/Hooks.cpp:789
time=2012-05-11 14:35:33.895 ]: Either the vistor clicked on the
''Stop''
button in the web browser, or the visitor''s connection has stalled and
couldn''t receive the data that Apache is sending to it. As a result,
you
will probably see a ''Broken Pipe'' error in this log file.
Please ignore it,
this is normal. You might also want to increase Apache''s TimeOut
configuration option if you experience this problem often.
I only have 4 puppet clients, I don''t see why I am encountering this
type
of performance issue. I get the same behavior when I run the
"puppetmaster"
service without HTTPD, so it appears that putting HTTPD in front of
"puppetmaster" is not helping me very much. Does anyone have any
suggestions on how I can resolve this? I am running the following versions
of the relevant software packages:
CentOS Linux release 6.0 (Final)
puppet-server-2.7.14-1.el6.noarch
puppet-2.7.14-1.el6.noarch
ruby-1.8.7.299-7.el6_1.1.x86_64
httpd-2.2.15-9.el6.centos.3.x86_64
rubygem-passenger-native-3.0.9-1.el6.x86_64
mod_passenger-3.0.9-1.el6.x86_64
rubygem-passenger-3.0.9-1.el6.x86_64
rubygem-passenger-native-libs-3.0.9-1.el6_1.8.7.299.x86_64
--------------------------------------------------------------------------------
/etc/httpd/conf.d/passenger.conf:
LoadModule passenger_module /usr/lib64/httpd/modules/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.9
PassengerRuby /usr/bin/ruby
# I have tried tweaking the below settings higher and lower but it
hasn''t
helped
PassengerHighPerformance on
PassengerUseGlobalQueue on
PassengerMaxRequests 99999
PassengerPoolIdleTime 60
PassengerMaxPoolSize 64
PassengerMinInstances 16
PassengerMaxInstancesPerApp 99999
--------------------------------------------------------------------------------
/etc/httpd/conf.d/puppetmaster.conf:
Listen 8140
<VirtualHost *:8140>
SSLEngine on
SSlProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
SSLCertificateFile /var/lib/puppet/ssl/certs/puppet-prod-sjc.agkn.net.pem
SSLCertificateKeyFile
/var/lib/puppet/ssl/private_keys/puppet-prod-sjc.agkn.net.pem
SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem
SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
SSLVerifyClient optional
SSLVerifyDepth 1
SSLOptions +StdEnvVars
RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
LogFormat "%h %l %u %t \"%r\" %>s %b %D
\"%{Referer}i\" \"%{User-Agent}i\""
puppet
CustomLog /var/log/httpd/puppetmaster.log puppet
RackAutoDetect On
DocumentRoot /etc/puppet/rack/public/
<Directory /etc/puppet/rack/puppetmaster/>
Options None
AllowOverride None
Order allow,deny
allow from all
</Directory>
</VirtualHost>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Ramin K
2012-May-15 21:30 UTC
Re: [Puppet Users] Connection reset by peer errors when performing puppet agent configuration run
On 5/15/2012 2:06 PM, Eric wrote:> PassengerHighPerformance on > PassengerUseGlobalQueue on > PassengerMaxRequests 99999 > PassengerPoolIdleTime 60 > PassengerMaxPoolSize 64 > PassengerMinInstances 16 > PassengerMaxInstancesPerApp 99999Your setting are as they say, not even wrong. Try the following in their place and restart Apache. Also you should be running Apache with the worker-mpm enabled if you haven''t done that yet. Keep in mind that the Apache/Passenger/Rack/Puppet stack needs roughly a minute from a restart to be ready to take traffic. PassengerMaxPoolSize 6 PassengerMinInstances 1 PassengerMaxRequests 10000 PassengerPoolIdleTime 0 #(or 2000 if this is a shared machine) PassengerStatThrottleRate 30 Assuming things work normally you can add these back in. PassengerHighPerformance on PassengerUseGlobalQueue on As you add more clients your limiting factors will be that each concurrent instance uses roughly 200MB of RAM and you should have no more than four instances per core (give or take) assuming this is a dedicated Puppet machine. If you run Puppet Dashaboard, Mysql, etc on the same box you may need to lower the max instances or add more machines. Ramin -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Campee
2012-May-15 21:49 UTC
[Puppet Users] Re: Connection reset by peer errors when performing puppet agent configuration run
On May 15, 2:30 pm, Ramin K <ramin-l...@badapple.net> wrote:> On 5/15/2012 2:06 PM, Eric wrote: > > > PassengerHighPerformance on > > PassengerUseGlobalQueue on > > PassengerMaxRequests 99999 > > PassengerPoolIdleTime 60 > > PassengerMaxPoolSize 64 > > PassengerMinInstances 16 > > PassengerMaxInstancesPerApp 99999 > > Your setting are as they say, not even wrong. > > Try the following in their place and restart Apache. Also you should be > running Apache with the worker-mpm enabled if you haven''t done that yet. > Keep in mind that the Apache/Passenger/Rack/Puppet stack needs roughly a > minute from a restart to be ready to take traffic. > > PassengerMaxPoolSize 6 > PassengerMinInstances 1 > PassengerMaxRequests 10000 > PassengerPoolIdleTime 0 #(or 2000 if this is a shared machine) > PassengerStatThrottleRate 30 > > Assuming things work normally you can add these back in. > PassengerHighPerformance on > PassengerUseGlobalQueue on > > As you add more clients your limiting factors will be that each > concurrent instance uses roughly 200MB of RAM and you should have no > more than four instances per core (give or take) assuming this is a > dedicated Puppet machine. If you run Puppet Dashaboard, Mysql, etc on > the same box you may need to lower the max instances or add more machines. > > RaminI made the changes that you suggested, restarted Apache, and I am still getting the "connection reset by peer" error 25% of the time. # cat /etc/httpd/conf.d/passenger.conf LoadModule passenger_module /usr/lib64/httpd/modules/mod_passenger.so PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.9 PassengerRuby /usr/bin/ruby PassengerMaxPoolSize 6 PassengerMinInstances 1 PassengerMaxRequests 10000 PassengerPoolIdleTime 0 PassengerStatThrottleRate 30 # grep -i worker /etc/sysconfig/httpd HTTPD=/usr/sbin/httpd.worker -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.