Jose Cambronero
2014-Mar-26 17:53 UTC
[Puppet Users] Puppet Agent run failing in puppet master server: Connection reset by peer, rbuf_fill
Our puppet master server is running version 3.0.2, using Apache/Passenger and has been handling about 10,000 resources just fine. Initially we had issues with the manifest timing out, but we added the "configtimeout" setting at the agent level, and set it to 15m. After making that change, there were no further errors. Last night we added about 10,000 additional managed resources and ever since, the puppet agent runs have been failing constantly. What's even more interesting is that the agent run fails exactly after 5 minutes of starting. I wonder if there's any other timeout setting that should be changed for this amount of resources? The agent run with 10,000 resources was taking ~10 minutes. Here's the information displayed when using puppet agent -t --debug --trace: Error: Could not retrieve catalog from remote server: Connection reset by peer /usr/lib/ruby/1.8/net/protocol.rb:135:in `sysread' /usr/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill' /usr/lib/ruby/1.8/timeout.rb:67:in `timeout' /usr/lib/ruby/1.8/timeout.rb:101:in `timeout' /usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill' /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil' /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline' /usr/lib/ruby/1.8/net/http.rb:2028:in `read_status_line' /usr/lib/ruby/1.8/net/http.rb:2017:in `read_new' /usr/lib/ruby/1.8/net/http.rb:1051:in `request' /usr/lib/ruby/1.8/net/http.rb:1037:in `request' /usr/lib/ruby/1.8/net/http.rb:543:in `start' /usr/lib/ruby/1.8/net/http.rb:1035:in `request' /usr/lib/ruby/1.8/net/http.rb:845:in `post' /usr/lib/ruby/vendor_ruby/puppet/network/http/connection.rb:63:in `send' /usr/lib/ruby/vendor_ruby/puppet/network/http/connection.rb:63:in `request' /usr/lib/ruby/vendor_ruby/puppet/network/http/connection.rb:31:in `post' /usr/lib/ruby/vendor_ruby/puppet/indirector/rest.rb:105:in `send' /usr/lib/ruby/vendor_ruby/puppet/indirector/rest.rb:105:in `http_request' /usr/lib/ruby/vendor_ruby/puppet/indirector/rest.rb:88:in `http_post' /usr/lib/ruby/vendor_ruby/puppet/indirector/rest.rb:116:in `find' /usr/lib/ruby/vendor_ruby/puppet/indirector/rest.rb:177:in `do_request' /usr/lib/ruby/vendor_ruby/puppet/indirector/request.rb:216:in `do_request' /usr/lib/ruby/vendor_ruby/puppet/indirector/rest.rb:177:in `do_request' /usr/lib/ruby/vendor_ruby/puppet/indirector/rest.rb:112:in `find' /usr/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:191:in `find' /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:243:in `retrieve_new_catalog' /usr/lib/ruby/vendor_ruby/puppet/util.rb:344:in `thinmark' /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime' /usr/lib/ruby/vendor_ruby/puppet/util.rb:343:in `thinmark' /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:242:in `retrieve_new_catalog' /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:67:in `retrieve_catalog' /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:107:in `prepare_and_retrieve_catalog' /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:159:in `run' /usr/lib/ruby/vendor_ruby/puppet/agent.rb:45:in `run' /usr/lib/ruby/vendor_ruby/puppet/agent/locker.rb:20:in `lock' /usr/lib/ruby/vendor_ruby/puppet/agent.rb:45:in `run' /usr/lib/ruby/1.8/sync.rb:230:in `synchronize' /usr/lib/ruby/vendor_ruby/puppet/agent.rb:45:in `run' /usr/lib/ruby/vendor_ruby/puppet/agent.rb:119:in `with_client' /usr/lib/ruby/vendor_ruby/puppet/agent.rb:42:in `run' /usr/lib/ruby/vendor_ruby/puppet/agent.rb:84:in `run_in_fork' /usr/lib/ruby/vendor_ruby/puppet/agent.rb:41:in `run' /usr/lib/ruby/vendor_ruby/puppet/application.rb:175:in `call' /usr/lib/ruby/vendor_ruby/puppet/application.rb:175:in `controlled_run' /usr/lib/ruby/vendor_ruby/puppet/agent.rb:39:in `run' /usr/lib/ruby/vendor_ruby/puppet/application/agent.rb:338:in `onetime' /usr/lib/ruby/vendor_ruby/puppet/application/agent.rb:312:in `run_command' /usr/lib/ruby/vendor_ruby/puppet/application.rb:346:in `run' /usr/lib/ruby/vendor_ruby/puppet/application.rb:438:in `plugin_hook' /usr/lib/ruby/vendor_ruby/puppet/application.rb:346:in `run' /usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail' /usr/lib/ruby/vendor_ruby/puppet/application.rb:346:in `run' /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:87:in `execute' /usr/bin/puppet:4 Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run Thanks, Jose -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/b200434e-4834-4819-a9bb-e0267570b020%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.