Greetings all, My deployment of puppet includes a monitrc to make sure that puppetd stays up and running. It seems like (across all of my hosts) that pupeptd will end up dying and restarted by monit. I''m running puppetd under a screen session now on one particular host with --debug --verbose to see if any thing comes out on console but until it happens again, any anybody else experienced puppetd dying randomly? Cheers, Ryan
> My deployment of puppet includes a monitrc to make sure that puppetdstays> up and running. It seems like (across all of my hosts) that pupeptdwill> end up dying and restarted by monit. I''m running puppetd under ascreen> session now on one particular host with --debug --verbose to see ifany> thing comes out on console but until it happens again, any anybodyelse> experienced puppetd dying randomly?I run the same setup (monit monitoring puppet) and I''d estimate that I get one puppetd restart every couple of days. That''s from about 30 servers. James ********************************************************************************* Important Note This email (including any attachments) contains information which is confidential and may be subject to legal privilege. If you are not the intended recipient you must not use, distribute or copy this email. If you have received this email in error please notify the sender immediately and delete this email. Any views expressed in this email are not necessarily the views of AXA-Tech Australia. Thank you. **********************************************************************************
Ryan Dooley <rd@powerset.com> writes:> Greetings all,> My deployment of puppet includes a monitrc to make sure that puppetd > stays up and running. It seems like (across all of my hosts) that > pupeptd will end up dying and restarted by monit. I''m running puppetd > under a screen session now on one particular host with --debug --verbose > to see if any thing comes out on console but until it happens again, any > anybody else experienced puppetd dying randomly?Yes, we''ve been trying to track down occasional problems with puppetd just silently exiting. It seems relatively rare, which makes it hard to track down, but it seems to be more common (although still not reproducible reliably) if the puppet master isn''t responding for some reason. -- Russ Allbery (rra@stanford.edu) <http://www.eyrie.org/~eagle/>
On 10/11/07 8:15 PM, "Russ Allbery" <rra@stanford.edu> wrote: Ryan Dooley <rd@powerset.com> writes:> Greetings all,> My deployment of puppet includes a monitrc to make sure that puppetd > stays up and running. It seems like (across all of my hosts) that > pupeptd will end up dying and restarted by monit. I''m running puppetd > under a screen session now on one particular host with --debug --verbose > to see if any thing comes out on console but until it happens again, any > anybody else experienced puppetd dying randomly?Yes, we''ve been trying to track down occasional problems with puppetd just silently exiting. It seems relatively rare, which makes it hard to track down, but it seems to be more common (although still not reproducible reliably) if the puppet master isn''t responding for some reason. That''s it exactly it seems, I just caught this in that screen session: info: Config is up to date notice: Starting configuration run notice: //base/Exec[set-password]/returns: executed successfully /usr/lib/ruby/1.8/timeout.rb:54:in `open'': execution expired (Timeout::Error) from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'' from /usr/lib/ruby/1.8/timeout.rb:56:in `timeout'' from /usr/lib/ruby/1.8/timeout.rb:76:in `timeout'' from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'' from /usr/lib/ruby/1.8/net/http.rb:553:in `do_start'' from /usr/lib/ruby/1.8/net/http.rb:542:in `start'' from /usr/lib/ruby/1.8/net/http.rb:1032:in `request'' from /usr/lib/ruby/1.8/net/http.rb:989:in `post2'' ... 46 levels... from /foo/lib/gems/puppet-0.23.2/lib/puppet.rb:361:in `start'' from /foo/lib/gems/puppet-0.23.2/bin/puppetd:447 from /foo/bin/puppetd:19:in `load'' from /foo/bin/puppetd:19 On the puppet master I do have (in my masterhttp.log): [2007-10-12 02:56:04] ERROR OpenSSL::SSL::SSLError: /usr/lib/ruby/1.8/openssl/ssl.rb:122:in `accept'' [2007-10-12 02:58:04] ERROR OpenSSL::SSL::SSLError: /usr/lib/ruby/1.8/openssl/ssl.rb:122:in `accept'' [2007-10-12 03:00:04] ERROR OpenSSL::SSL::SSLError: /usr/lib/ruby/1.8/openssl/ssl.rb:122:in `accept'' My install base isn''t that big (yet) but I do have my interval set to just 15 minutes (instead of the default 30). I wonder if it would be possible to have puppetd set a random interval. I suppose it''s possible that I have too many hosts connecting at the same time. I suppose the quick fix is probably to rescue from the Timeout::Error (probably not the best place for it but...) if block begin EventLoop.current.run rescue Timeout::Error end end ... which I''m running now (guess I need to make the time to dive in to the code :-) I have a single puppetmaster at the moment, 5 mongrel and 1 webrick (for the ca). The mongrels are proxied to by apache. Cheers, Ryan _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday 12 October 2007, Ryan Dooley wrote:> My install base isn''t that big (yet) but I do have my interval set to just > 15 minutes (instead of the default 30). > > I wonder if it would be possible to have puppetd set a random interval. I > suppose it''s possible that I have too many hosts connecting at the same > time.Use the splay* options. Regards, David - -- The primary freedom of open source is not the freedom from cost, but the free- dom to shape software to do what you want. This freedom is /never/ exercised without cost, but is available /at all/ only by accepting the very different costs associated with open source, costs not in money, but in time and effort. - -- http://www.schierer.org/~luke/log/20070710-1129/on-forks-and-forking -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFHDynu/Pp1N6Uzh0URApHQAJ9e4e2wo0eZCthfP0rj6fM6DMEYYgCgnbvW BFlmK9EYkoLVSsY/CtQDPiQ=+ttC -----END PGP SIGNATURE-----
On Thu, Oct 11, 2007 at 08:36:55PM -0700, Ryan Dooley wrote:> On 10/11/07 8:15 PM, "Russ Allbery" <rra@stanford.edu> wrote: > > Ryan Dooley <rd@powerset.com> writes: > > > Greetings all, > > > My deployment of puppet includes a monitrc to make sure that puppetd > > stays up and running. It seems like (across all of my hosts) that > > pupeptd will end up dying and restarted by monit. I''m running puppetd > > under a screen session now on one particular host with --debug --verbose > > to see if any thing comes out on console but until it happens again, any > > anybody else experienced puppetd dying randomly? > > Yes, we''ve been trying to track down occasional problems with puppetd just > silently exiting. It seems relatively rare, which makes it hard to track > down, but it seems to be more common (although still not reproducible > reliably) if the puppet master isn''t responding for some reason. > > > That''s it exactly it seems, I just caught this in that screen session: > > > info: Config is up to date > notice: Starting configuration run > notice: //base/Exec[set-password]/returns: executed successfully > /usr/lib/ruby/1.8/timeout.rb:54:in `open'': execution expired (Timeout::Error) > from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'' > from /usr/lib/ruby/1.8/timeout.rb:56:in `timeout'' > from /usr/lib/ruby/1.8/timeout.rb:76:in `timeout'' > from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'' > from /usr/lib/ruby/1.8/net/http.rb:553:in `do_start'' > from /usr/lib/ruby/1.8/net/http.rb:542:in `start'' > from /usr/lib/ruby/1.8/net/http.rb:1032:in `request'' > from /usr/lib/ruby/1.8/net/http.rb:989:in `post2'' > ... 46 levels... > from /foo/lib/gems/puppet-0.23.2/lib/puppet.rb:361:in `start'' > from /foo/lib/gems/puppet-0.23.2/bin/puppetd:447 > from /foo/bin/puppetd:19:in `load'' > from /foo/bin/puppetd:19see http://reductivelabs.com/trac/puppet/ticket/685 -- Marcin Owsiany Web Systems Integrator - Guardian Unlimited ------------------------------------------------------------------ Visit Guardian Unlimited - the UK''s most popular newspaper website http://guardian.co.uk http://observer.co.uk ------------------------------------------------------------------ The Newspaper Marketing Agency Opening Up Newspapers http://www.nmauk.co.uk ------------------------------------------------------------------ This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News & Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software. Guardian News & Media Limited A member of Guardian Media Group PLC Registered Office Number 1 Scott Place, Manchester M3 3GG Registered in England Number 908396
Most excellent :-) Cheers, Ryan ________________________________________ From: puppet-users-bounces@madstop.com [puppet-users-bounces@madstop.com] On Behalf Of Marcin Owsiany [marcin.owsiany@guardian.co.uk] Sent: Friday, October 12, 2007 1:10 AM To: puppet-users@madstop.com Subject: Re: [Puppet-users] puppetd dies? On Thu, Oct 11, 2007 at 08:36:55PM -0700, Ryan Dooley wrote:> On 10/11/07 8:15 PM, "Russ Allbery" <rra@stanford.edu> wrote: > > Ryan Dooley <rd@powerset.com> writes: > > > Greetings all, > > > My deployment of puppet includes a monitrc to make sure that puppetd > > stays up and running. It seems like (across all of my hosts) that > > pupeptd will end up dying and restarted by monit. I''m running puppetd > > under a screen session now on one particular host with --debug --verbose > > to see if any thing comes out on console but until it happens again, any > > anybody else experienced puppetd dying randomly? > > Yes, we''ve been trying to track down occasional problems with puppetd just > silently exiting. It seems relatively rare, which makes it hard to track > down, but it seems to be more common (although still not reproducible > reliably) if the puppet master isn''t responding for some reason. > > > That''s it exactly it seems, I just caught this in that screen session: > > > info: Config is up to date > notice: Starting configuration run > notice: //base/Exec[set-password]/returns: executed successfully > /usr/lib/ruby/1.8/timeout.rb:54:in `open'': execution expired (Timeout::Error) > from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'' > from /usr/lib/ruby/1.8/timeout.rb:56:in `timeout'' > from /usr/lib/ruby/1.8/timeout.rb:76:in `timeout'' > from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'' > from /usr/lib/ruby/1.8/net/http.rb:553:in `do_start'' > from /usr/lib/ruby/1.8/net/http.rb:542:in `start'' > from /usr/lib/ruby/1.8/net/http.rb:1032:in `request'' > from /usr/lib/ruby/1.8/net/http.rb:989:in `post2'' > ... 46 levels... > from /foo/lib/gems/puppet-0.23.2/lib/puppet.rb:361:in `start'' > from /foo/lib/gems/puppet-0.23.2/bin/puppetd:447 > from /foo/bin/puppetd:19:in `load'' > from /foo/bin/puppetd:19see http://reductivelabs.com/trac/puppet/ticket/685 -- Marcin Owsiany Web Systems Integrator - Guardian Unlimited ------------------------------------------------------------------ Visit Guardian Unlimited - the UK''s most popular newspaper website http://guardian.co.uk http://observer.co.uk ------------------------------------------------------------------ The Newspaper Marketing Agency Opening Up Newspapers http://www.nmauk.co.uk ------------------------------------------------------------------ This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News & Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software. Guardian News & Media Limited A member of Guardian Media Group PLC Registered Office Number 1 Scott Place, Manchester M3 3GG Registered in England Number 908396 _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On 10/12/07 1:01 AM, "David Schmitt" <david@schmitt.edv-bus.at> wrote: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday 12 October 2007, Ryan Dooley wrote:> My install base isn''t that big (yet) but I do have my interval set to just > 15 minutes (instead of the default 30). > > I wonder if it would be possible to have puppetd set a random interval. I > suppose it''s possible that I have too many hosts connecting at the same > time.Use the splay* options. Nice. Thanks for pointing that out :-) Cheers, Ryan _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users