Anybody else experience the need to restart puppetmasterd before a node can register itself. I''ve been cleaning up my installation here on client nodes and every once and a while I find myself needing to restart the puppetmaster daemon. A client node attempts to connect and never does. A restart fixes things. There doesn''t seem to be anything useful in the error messages except for an ''accept'' error from OpenSSL. Cheers, Ryan
On May 18, 2007, at 12:28 PM, Ryan Dooley wrote:> Anybody else experience the need to restart puppetmasterd before a > node can > register itself. > > I''ve been cleaning up my installation here on client nodes and > every once > and a while I find myself needing to restart the puppetmaster > daemon. A > client node attempts to connect and never does. A restart fixes > things. > > There doesn''t seem to be anything useful in the error messages > except for an > ''accept'' error from OpenSSL.Can you describe the problem in more detail? When you say a client attempts to connect and never does, do you mean the connection never returns, it encounters an error, or what? -- As a general rule, don''t solve puzzles that open portals to Hell. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
> Can you describe the problem in more detail? When you say a client > attempts to connect and never does, do you mean the connection never > returns, it encounters an error, or what?Sure (I should have probably provided more detail to begin with but you know... :-) The client starts up like this: /foo/bin/puppetd -o --server puppetmaster --authconfig /foo/conf/puppet --ssldir /foo/conf/puppet/ssl --vardir /foo/var/lib/puppet --classfile /foo/conf/puppet/classes.txt --rundir /foo/var/run/puppet The client establishes a connection to puppetmaster [root@client]~# netstat |grep puppetmaster tcp 0 126 client:54164 puppetmaster:8140 ESTABLISHED The master seems to be stuck in SYN_RECV most of the time for this host: (puppetmaster)[~]% netstat | grep client tcp 0 0 puppetmaster:8140 client:54164 SYN_RECV The clients logs contain: May 18 20:00:31 client puppetd[21000]: Could not call puppetmaster.getconfig: #<Errno::ECONNRESET: Connection reset by peer> May 18 20:00:31 client puppetd[21000]: Could not retrieve configuration: Connection reset by peer May 18 20:00:31 client puppetd[21000]: Could not get config; using cached copy The server logs don''t really show anything at all other than previous connection attempts. Attaching to the running process with strace doesn''t show much either. As soon as I restart the puppemaster process, puppet does the right thing. I''ll run the master with debugging turned on so I can grab what''s going on for the next time I see this happening. Cheers, Ryan
Okay, more on this puppetmaster weirdness front: The client starts up after installation due to another package (monit) finding a monitrc and starting the service. That part on my end is busted because monit starts up puppet incorrectly (not sure why, all it''s doing is calling a custom init.d script). This results in a SSL file being stored in the traditional place (/etc/puppet/ssl/...) By hand I have to kill the running puppet (monit at this point is looking at the wrong place for a pid file for the running puppetd on the client). As soon as I do that, the puppetmaster stops responding to clients (with --debug and --verbose) the master just sits there spinning away. No other clients are allowed to connect (or what was witnessed for about 10 minutes). As soon as the puppetmaster is restarted, things go back to normal. Clients connect again and life is good. Half my fix will be to make sure that my custom packages don''t require monit right away and I''ll let puppet install monit as needed :-) Cheers, Ryan