Hi all,
I''ve followed http://reductivelabs.com/trac/puppet/wiki/UsingMongrel
for configuring my puppet with mongrel.
Al seems to work fine, except that, after a reinstall of 40 nodes atone
time, I got many kind of errors like:
-------------------------------------------------------------------------
err: Could not request certificate: Certificate retrieval failed: .tmp
file already exists for /var/lib/puppet/ssl/ca/serial; Aborting locked
write. Check the .tmp file and delete if appropriate
-------------------------------------------------------------------------
info: Creating a new certificate request for td029.pic.es
info: Creating a new SSL key
at /var/lib/puppet/ssl/private_keys/td029.pic.es.pem notice: Got signed
certificate err: Connection timeout calling puppetmaster.getconfig:
execution expired err: Could not retrieve catalog: Connection Timeout
warning: Not using cache on failed catalog
-------------------------------------------------------------------------
info: Creating a new certificate request for td035.pic.es
info: Creating a new SSL key
at /var/lib/puppet/ssl/private_keys/td035.pic.es.pem err: Could not
request certificate: Certificate retrieval failed: .tmp file already
exists for /var/lib/puppet/ssl/ca/serial; Aborting locked write. Check
the .tmp file and delete if appropriate
Not sure if there are more... but seems that puppetmasterd is getting
crazy...
I took the second conf example:
# grep . /etc/httpd/conf.d/mongrel.conf
Listen 8140
ProxyRequests Off
LoadModule ssl_module modules/mod_ssl.so
<Proxy balancer://puppetmaster>
BalancerMember http://127.0.0.1:18140
</Proxy>
<VirtualHost *:8140>
SSLEngine on
SSLCipherSuite SSLv2:-LOW:-EXPORT:RC4+RSA
SSLCertificateFile /var/lib/puppet/ssl/certs/gridinstall.pic.es.pem
SSLCertificateKeyFile
/var/lib/puppet/ssl/private_keys/gridinstall.pic.es.pem
SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
SSLVerifyClient optional
SSLVerifyDepth 1
SSLOptions +StdEnvVars
RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
<Location />
SetHandler balancer-manager
Order allow,deny
Allow from all
</Location>
ProxyPass / balancer://puppetmaster:8140/
ProxyPassReverse / balancer://puppetmaster:8140/
ProxyPreserveHost on
</VirtualHost>
]# grep -v "#" /etc/sysconfig/puppetmaster|grep .
PUPPETMASTER_MANIFEST=/etc/puppet/manifests/site.pp
PUPPETMASTER_LOG=/var/log/puppet/puppetmaster.log
PUPPETMASTER_PORTS=( 18140 18141 18142 18143 )
PUPPETMASTER_EXTRA_OPTS="--servertype=mongrel"
And puppet is running as:
puppet 2848 12.8 3.1 147100 131852 ? Rsl 11:52 1:56 /usr/bin/ruby
/usr/sbin/puppetmasterd --manifest=/etc/puppet/manifests/site.pp
--servertype=mongrel --logdest=/var/log/puppet/puppetmaster.log
--servertype=mongrel --masterport=18140
--pidfile=/var/run/puppet/puppetmaster.18140.pid
puppet 2866 0.0 0.5 37204 21260 ? Ssl 11:52 0:00 /usr/bin/ruby
/usr/sbin/puppetmasterd --manifest=/etc/puppet/manifests/site.pp
--servertype=mongrel --logdest=/var/log/puppet/puppetmaster.log
--servertype=mongrel --masterport=18141
--pidfile=/var/run/puppet/puppetmaster.18141.pid
puppet 2884 0.0 0.5 37196 21256 ? Ssl 11:52 0:00 /usr/bin/ruby
/usr/sbin/puppetmasterd --manifest=/etc/puppet/manifests/site.pp
--servertype=mongrel --logdest=/var/log/puppet/puppetmaster.log
--servertype=mongrel --masterport=18142
--pidfile=/var/run/puppet/puppetmaster.18142.pid
puppet 2902 0.0 0.5 37196 21260 ? Ssl 11:52 0:00 /usr/bin/ruby
/usr/sbin/puppetmasterd --manifest=/etc/puppet/manifests/site.pp
--servertype=mongrel --logdest=/var/log/puppet/puppetmaster.log
--servertype=mongrel --masterport=18143
--pidfile=/var/run/puppet/puppetmaster.18143.pid
root 3042 0.0 0.0 3912 664 pts/0 R+ 12:07 0:00 grep puppet
(notice that only one server has used some cpu time, the first one).
Anyone could help me to find the main reason why puppet is behaving
lke this?
TIA,
Arnau
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---
I forgot to add a netstat: [root@gridinstall ~]# netstat -puta|grep ruby tcp 0 0 localhost.localdomain:18140 *:* LISTEN 2848/ruby tcp 0 0 localhost.localdomain:18141 *:* LISTEN 2866/ruby tcp 0 0 localhost.localdomain:18142 *:* LISTEN 2884/ruby tcp 0 0 localhost.localdomain:18143 *:* LISTEN 2902/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32966 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32967 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32964 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32965 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32962 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32963 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32960 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32961 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32970 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32971 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32968 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32969 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32926 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32934 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32935 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32930 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32928 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32943 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32940 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32941 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32938 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32939 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32936 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32937 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32950 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32948 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32946 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32947 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32945 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32958 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32959 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32957 ESTABLISHED 2848/ruby tcp 0 0 localhost.localdomain:18140 localhost.localdomain:32953 ESTABLISHED 2848/ruby All connections go to por 18140.... TIA, Arnau --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I''ve cahnged to the first apache conf exmaple and seems that now it''s balancing: [root@gridinstall conf.d]# netstat -puta|grep ruby|grep -c 18143 12 [root@gridinstall conf.d]# netstat -puta|grep ruby|grep -c 18142 10 [root@gridinstall conf.d]# netstat -puta|grep ruby|grep -c 18141 21 [root@gridinstall conf.d]# netstat -puta|grep ruby|grep -c 18140 17 But now I have many : err: Connection timeout calling puppetmaster.getconfig: execution expired err: Could not retrieve catalog: Connection Timeout warning: Not using cache on failed catalog in clients and: # cat balancer_error.log [Tue Feb 10 12:30:17 2009] [error] [client 193.109.173.11] proxy: error reading status line from remote server 127.0.0.1 [Tue Feb 10 12:30:17 2009] [error] [client 193.109.173.11] proxy: Error reading from remote server returned by /RPC2 [...] [Tue Feb 10 12:31:25 2009] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:18140 (127.0.0.1) failed [Tue Feb 10 12:31:25 2009] [error] ap_proxy_connect_backend disabling worker for (127.0.0.1) [...] [Tue Feb 10 12:31:32 2009] [error] proxy: BALANCER: (balancer://puppetmaster). All workers are in error state [Tue Feb 10 12:31:32 2009] [error] proxy: BALANCER: (balancer://puppetmaster). All workers are in error state Anyone could help me? Cheers, Arnau --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Feb 10, 2009, at 5:34 AM, Arnau Bria wrote:> > Hi all, > > I''ve followed http://reductivelabs.com/trac/puppet/wiki/UsingMongrel > for configuring my puppet with mongrel. > > Al seems to work fine, except that, after a reinstall of 40 nodes > atone > time, I got many kind of errors like: > > ------------------------------------------------------------------------- > err: Could not request certificate: Certificate retrieval failed: .tmp > file already exists for /var/lib/puppet/ssl/ca/serial; Aborting locked > write. Check the .tmp file and delete if appropriateThis looks like contention between multiple processes for that serial file, which shouldn''t happen very often although is reasonable once and a while.> > ------------------------------------------------------------------------- > info: Creating a new certificate request for td029.pic.es > info: Creating a new SSL key > at /var/lib/puppet/ssl/private_keys/td029.pic.es.pem notice: Got > signed > certificate err: Connection timeout calling puppetmaster.getconfig: > execution expired err: Could not retrieve catalog: Connection Timeout > warning: Not using cache on failed catalogNot sure on this one; I''ve seen it, but usually it''s actually a timeout issue. Maybe your server is waiting on a lock for that serial file? -- If you would be a real seeker after truth, it is necessary that at least once in your life you doubt, as far as possible, all things. -- Rene Descartes --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---