Jason Antman
2012-Sep-14 14:40 UTC
[Puppet Users] assistance needed - err: Could not retrieve catalog from remote server: execution expired
Hi, I have a puppet infrastructure running 2.6.14 under Passenger with about 30 nodes. I''m just spinning up a new Cent 6.3, puppet 2.7.19 master. Right now I have the master built and configured using the same configuration as my 2.6.14 setup, with a few tweaks for the new version. Puppet on the master built itself fine. I have puppetd stopped on the master. I tried pointing one node at it, but I''m getting: err: Could not retrieve catalog from remote server: execution expired I disabled passenger and ran through the builtin webrick, and confirmed that only one node is talking to the master. No change. No helpful information in the master log. I''m running the master with: /usr/sbin/puppetmasterd --no-daemonize --debug --logdest console --verbose --trace and some relevant puppet.conf portions on the master: storeconfigs = true thin_storeconfigs = true reports = log,puppet_dashboard reportfrom = root syslogfacility = user # Remote Inventory Service facts_terminus = rest inventory_server = tv-puppetma1.techtarget.com # fqdn of master inventory_port = 8140 The exact behavior I''m seeing is that when I run puppetd on the node, all works well up to seeing "debug: catalog supports formats: marshal pson raw yaml; using pson" on the node and "debug: facts supports formats: b64_zlib_yaml pson raw yaml; using pson" on the master. At that point, it hangs for 2 minutes (120 seconds exactly), with no output on either the node or master. puppetmasterd is using ~25% CPU and ~14 memory. After that time, I see the below stack trace on the master, followed by a whole slew of the same for the facts, and "err: Could not retrieve catalog from remote server: execution expired" on the node. Any ideas? I''m going to try adding resources to the master VM, but I find it hard to believe that this is just caused by compiling a single catalog on the master... Any advice or insight would be greatly appreciated. Thanks, Jason Antman Trace from master: /usr/lib/ruby/1.8/timeout.rb:64:in `rbuf_fill'' /usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'' /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'' /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'' /usr/lib/ruby/1.8/net/http.rb:2028:in `read_status_line'' /usr/lib/ruby/1.8/net/http.rb:2017:in `read_new'' /usr/lib/ruby/1.8/net/http.rb:1051:in `request'' /usr/lib/ruby/1.8/net/http.rb:1037:in `request'' /usr/lib/ruby/1.8/net/http.rb:543:in `start'' /usr/lib/ruby/1.8/net/http.rb:1035:in `request'' /usr/lib/ruby/1.8/net/http.rb:857:in `put'' /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:98:in `send'' /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:98:in `http_request'' /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:76:in `http_put'' /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:158:in `save'' /usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:271:in `save'' /usr/lib/ruby/site_ruby/1.8/puppet/node/facts.rb:20:in `save'' /usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:26:in `extract_facts_from_request'' /usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:31:in `find'' /usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:195:in `find'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:109:in `do_find'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `send'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `process'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick/rest.rb:24:in `service'' /usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'' /usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:45:in `listen'' /usr/lib/ruby/1.8/webrick/server.rb:173:in `call'' /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'' /usr/lib/ruby/1.8/webrick/server.rb:162:in `start'' /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'' /usr/lib/ruby/1.8/webrick/server.rb:95:in `start'' /usr/lib/ruby/1.8/webrick/server.rb:92:in `each'' /usr/lib/ruby/1.8/webrick/server.rb:92:in `start'' /usr/lib/ruby/1.8/webrick/server.rb:23:in `start'' /usr/lib/ruby/1.8/webrick/server.rb:82:in `start'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `initialize'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `listen'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `synchronize'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `listen'' /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:126:in `listen'' /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:141:in `start'' /usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:124:in `start'' /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:202:in `main'' /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:146:in `run_command'' /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'' /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:416:in `hook'' /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'' /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:407:in `exit_on_fail'' /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'' /usr/sbin/puppetmasterd:4 err: /usr/lib/ruby/1.8/timeout.rb:64:in `rbuf_fill'' /usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'' /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'' /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'' /usr/lib/ruby/1.8/net/http.rb:2028:in `read_status_line'' /usr/lib/ruby/1.8/net/http.rb:2017:in `read_new'' /usr/lib/ruby/1.8/net/http.rb:1051:in `request'' /usr/lib/ruby/1.8/net/http.rb:1037:in `request'' /usr/lib/ruby/1.8/net/http.rb:543:in `start'' /usr/lib/ruby/1.8/net/http.rb:1035:in `request'' /usr/lib/ruby/1.8/net/http.rb:857:in `put'' /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:98:in `send'' /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:98:in `http_request'' /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:76:in `http_put'' /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:158:in `save'' /usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:271:in `save'' /usr/lib/ruby/site_ruby/1.8/puppet/node/facts.rb:20:in `save'' /usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:26:in `extract_facts_from_request'' /usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:31:in `find'' /usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:195:in `find'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:109:in `do_find'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `send'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `process'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick/rest.rb:24:in `service'' /usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'' /usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:45:in `listen'' /usr/lib/ruby/1.8/webrick/server.rb:173:in `call'' /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'' /usr/lib/ruby/1.8/webrick/server.rb:162:in `start'' /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'' /usr/lib/ruby/1.8/webrick/server.rb:95:in `start'' /usr/lib/ruby/1.8/webrick/server.rb:92:in `each'' /usr/lib/ruby/1.8/webrick/server.rb:92:in `start'' /usr/lib/ruby/1.8/webrick/server.rb:23:in `start'' /usr/lib/ruby/1.8/webrick/server.rb:82:in `start'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `initialize'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `listen'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `synchronize'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `listen'' /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:126:in `listen'' /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:141:in `start'' /usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:124:in `start'' /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:202:in `main'' /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:146:in `run_command'' /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'' /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:416:in `hook'' /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'' /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:407:in `exit_on_fail'' /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'' /usr/sbin/puppetmasterd:4 err: execution expired -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/MmfNpa2LnIoJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Jason Antman
2012-Sep-14 15:25 UTC
[Puppet Users] Re: assistance needed - err: Could not retrieve catalog from remote server: execution expired
Commented out storeconfigs/thin_storeconfigs as we were only using them to get facts in a DB and for Dashboard. Also commented out the Inventory Service stuff. Looks like that did it. Now I just need to figure out the real fix... On Friday, September 14, 2012 10:40:16 AM UTC-4, Jason Antman wrote:> > Hi, > > I have a puppet infrastructure running 2.6.14 under Passenger with about > 30 nodes. I''m just spinning up a new Cent 6.3, puppet 2.7.19 master. Right > now I have the master built and configured using the same configuration as > my 2.6.14 setup, with a few tweaks for the new version. Puppet on the > master built itself fine. I have puppetd stopped on the master. I tried > pointing one node at it, but I''m getting: > err: Could not retrieve catalog from remote server: execution expired > > I disabled passenger and ran through the builtin webrick, and confirmed > that only one node is talking to the master. No change. No helpful > information in the master log. I''m running the master with: > /usr/sbin/puppetmasterd --no-daemonize --debug --logdest console --verbose > --trace > > and some relevant puppet.conf portions on the master: > storeconfigs = true > thin_storeconfigs = true > reports = log,puppet_dashboard > reportfrom = root > syslogfacility = user > # Remote Inventory Service > facts_terminus = rest > inventory_server = tv-puppetma1.techtarget.com # fqdn of master > inventory_port = 8140 > > The exact behavior I''m seeing is that when I run puppetd on the node, all > works well up to seeing "debug: catalog supports formats: marshal pson raw > yaml; using pson" on the node and "debug: facts supports formats: > b64_zlib_yaml pson raw yaml; using pson" on the master. At that point, it > hangs for 2 minutes (120 seconds exactly), with no output on either the > node or master. puppetmasterd is using ~25% CPU and ~14 memory. After that > time, I see the below stack trace on the master, followed by a whole slew > of the same for the facts, and "err: Could not retrieve catalog from remote > server: execution expired" on the node. > > Any ideas? I''m going to try adding resources to the master VM, but I find > it hard to believe that this is just caused by compiling a single catalog > on the master... Any advice or insight would be greatly appreciated. > > Thanks, > Jason Antman > > Trace from master: > /usr/lib/ruby/1.8/timeout.rb:64:in `rbuf_fill'' > /usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'' > /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'' > /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'' > /usr/lib/ruby/1.8/net/http.rb:2028:in `read_status_line'' > /usr/lib/ruby/1.8/net/http.rb:2017:in `read_new'' > /usr/lib/ruby/1.8/net/http.rb:1051:in `request'' > /usr/lib/ruby/1.8/net/http.rb:1037:in `request'' > /usr/lib/ruby/1.8/net/http.rb:543:in `start'' > /usr/lib/ruby/1.8/net/http.rb:1035:in `request'' > /usr/lib/ruby/1.8/net/http.rb:857:in `put'' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:98:in `send'' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:98:in `http_request'' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:76:in `http_put'' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:158:in `save'' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:271:in `save'' > /usr/lib/ruby/site_ruby/1.8/puppet/node/facts.rb:20:in `save'' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:26:in > `extract_facts_from_request'' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:31:in > `find'' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:195:in `find'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:109:in `do_find'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `send'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `process'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick/rest.rb:24:in > `service'' > /usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'' > /usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:45:in `listen'' > /usr/lib/ruby/1.8/webrick/server.rb:173:in `call'' > /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'' > /usr/lib/ruby/1.8/webrick/server.rb:162:in `start'' > /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'' > /usr/lib/ruby/1.8/webrick/server.rb:95:in `start'' > /usr/lib/ruby/1.8/webrick/server.rb:92:in `each'' > /usr/lib/ruby/1.8/webrick/server.rb:92:in `start'' > /usr/lib/ruby/1.8/webrick/server.rb:23:in `start'' > /usr/lib/ruby/1.8/webrick/server.rb:82:in `start'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in > `initialize'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `listen'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in > `synchronize'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `listen'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:126:in `listen'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:141:in `start'' > /usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:124:in `start'' > /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:202:in `main'' > /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:146:in > `run_command'' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:416:in `hook'' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:407:in `exit_on_fail'' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'' > /usr/sbin/puppetmasterd:4 > err: /usr/lib/ruby/1.8/timeout.rb:64:in `rbuf_fill'' > /usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'' > /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'' > /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'' > /usr/lib/ruby/1.8/net/http.rb:2028:in `read_status_line'' > /usr/lib/ruby/1.8/net/http.rb:2017:in `read_new'' > /usr/lib/ruby/1.8/net/http.rb:1051:in `request'' > /usr/lib/ruby/1.8/net/http.rb:1037:in `request'' > /usr/lib/ruby/1.8/net/http.rb:543:in `start'' > /usr/lib/ruby/1.8/net/http.rb:1035:in `request'' > /usr/lib/ruby/1.8/net/http.rb:857:in `put'' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:98:in `send'' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:98:in `http_request'' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:76:in `http_put'' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:158:in `save'' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:271:in `save'' > /usr/lib/ruby/site_ruby/1.8/puppet/node/facts.rb:20:in `save'' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:26:in > `extract_facts_from_request'' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:31:in > `find'' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:195:in `find'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:109:in `do_find'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `send'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `process'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick/rest.rb:24:in > `service'' > /usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'' > /usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:45:in `listen'' > /usr/lib/ruby/1.8/webrick/server.rb:173:in `call'' > /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'' > /usr/lib/ruby/1.8/webrick/server.rb:162:in `start'' > /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'' > /usr/lib/ruby/1.8/webrick/server.rb:95:in `start'' > /usr/lib/ruby/1.8/webrick/server.rb:92:in `each'' > /usr/lib/ruby/1.8/webrick/server.rb:92:in `start'' > /usr/lib/ruby/1.8/webrick/server.rb:23:in `start'' > /usr/lib/ruby/1.8/webrick/server.rb:82:in `start'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in > `initialize'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `listen'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in > `synchronize'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `listen'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:126:in `listen'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:141:in `start'' > /usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:124:in `start'' > /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:202:in `main'' > /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:146:in > `run_command'' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:416:in `hook'' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:407:in `exit_on_fail'' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'' > /usr/sbin/puppetmasterd:4 > err: execution expired >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/X-qhrbFle9YJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Daniel Gagnon
2012-Nov-08 10:51 UTC
[Puppet Users] Re: assistance needed - err: Could not retrieve catalog from remote server: execution expired
Hi, I am also experiencing the same issue. Any luck with a resolution ? Personally, I use puppet3 with puppetdb. Thanks, On Friday, September 14, 2012 11:25:55 AM UTC-4, Jason Antman wrote:> > Commented out storeconfigs/thin_storeconfigs as we were only using them to > get facts in a DB and for Dashboard. Also commented out the Inventory > Service stuff. Looks like that did it. Now I just need to figure out the > real fix... > > On Friday, September 14, 2012 10:40:16 AM UTC-4, Jason Antman wrote: >> >> Hi, >> >> I have a puppet infrastructure running 2.6.14 under Passenger with about >> 30 nodes. I''m just spinning up a new Cent 6.3, puppet 2.7.19 master. Right >> now I have the master built and configured using the same configuration as >> my 2.6.14 setup, with a few tweaks for the new version. Puppet on the >> master built itself fine. I have puppetd stopped on the master. I tried >> pointing one node at it, but I''m getting: >> err: Could not retrieve catalog from remote server: execution expired >> >> I disabled passenger and ran through the builtin webrick, and confirmed >> that only one node is talking to the master. No change. No helpful >> information in the master log. I''m running the master with: >> /usr/sbin/puppetmasterd --no-daemonize --debug --logdest console >> --verbose --trace >> >> and some relevant puppet.conf portions on the master: >> storeconfigs = true >> thin_storeconfigs = true >> reports = log,puppet_dashboard >> reportfrom = root >> syslogfacility = user >> # Remote Inventory Service >> facts_terminus = rest >> inventory_server = tv-puppetma1.techtarget.com # fqdn of master >> inventory_port = 8140 >> >> The exact behavior I''m seeing is that when I run puppetd on the node, all >> works well up to seeing "debug: catalog supports formats: marshal pson raw >> yaml; using pson" on the node and "debug: facts supports formats: >> b64_zlib_yaml pson raw yaml; using pson" on the master. At that point, it >> hangs for 2 minutes (120 seconds exactly), with no output on either the >> node or master. puppetmasterd is using ~25% CPU and ~14 memory. After that >> time, I see the below stack trace on the master, followed by a whole slew >> of the same for the facts, and "err: Could not retrieve catalog from remote >> server: execution expired" on the node. >> >> Any ideas? I''m going to try adding resources to the master VM, but I find >> it hard to believe that this is just caused by compiling a single catalog >> on the master... Any advice or insight would be greatly appreciated. >> >> Thanks, >> Jason Antman >> >> Trace from master: >> /usr/lib/ruby/1.8/timeout.rb:64:in `rbuf_fill'' >> /usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'' >> /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'' >> /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'' >> /usr/lib/ruby/1.8/net/http.rb:2028:in `read_status_line'' >> /usr/lib/ruby/1.8/net/http.rb:2017:in `read_new'' >> /usr/lib/ruby/1.8/net/http.rb:1051:in `request'' >> /usr/lib/ruby/1.8/net/http.rb:1037:in `request'' >> /usr/lib/ruby/1.8/net/http.rb:543:in `start'' >> /usr/lib/ruby/1.8/net/http.rb:1035:in `request'' >> /usr/lib/ruby/1.8/net/http.rb:857:in `put'' >> /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:98:in `send'' >> /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:98:in `http_request'' >> /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:76:in `http_put'' >> /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:158:in `save'' >> /usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:271:in `save'' >> /usr/lib/ruby/site_ruby/1.8/puppet/node/facts.rb:20:in `save'' >> /usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:26:in >> `extract_facts_from_request'' >> /usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:31:in >> `find'' >> /usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:195:in `find'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:109:in >> `do_find'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `send'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `process'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick/rest.rb:24:in >> `service'' >> /usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'' >> /usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:45:in `listen'' >> /usr/lib/ruby/1.8/webrick/server.rb:173:in `call'' >> /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'' >> /usr/lib/ruby/1.8/webrick/server.rb:162:in `start'' >> /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'' >> /usr/lib/ruby/1.8/webrick/server.rb:95:in `start'' >> /usr/lib/ruby/1.8/webrick/server.rb:92:in `each'' >> /usr/lib/ruby/1.8/webrick/server.rb:92:in `start'' >> /usr/lib/ruby/1.8/webrick/server.rb:23:in `start'' >> /usr/lib/ruby/1.8/webrick/server.rb:82:in `start'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in >> `initialize'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `listen'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in >> `synchronize'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `listen'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:126:in `listen'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:141:in `start'' >> /usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:124:in `start'' >> /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:202:in `main'' >> /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:146:in >> `run_command'' >> /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'' >> /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:416:in `hook'' >> /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'' >> /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:407:in `exit_on_fail'' >> /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'' >> /usr/sbin/puppetmasterd:4 >> err: /usr/lib/ruby/1.8/timeout.rb:64:in `rbuf_fill'' >> /usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'' >> /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'' >> /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'' >> /usr/lib/ruby/1.8/net/http.rb:2028:in `read_status_line'' >> /usr/lib/ruby/1.8/net/http.rb:2017:in `read_new'' >> /usr/lib/ruby/1.8/net/http.rb:1051:in `request'' >> /usr/lib/ruby/1.8/net/http.rb:1037:in `request'' >> /usr/lib/ruby/1.8/net/http.rb:543:in `start'' >> /usr/lib/ruby/1.8/net/http.rb:1035:in `request'' >> /usr/lib/ruby/1.8/net/http.rb:857:in `put'' >> /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:98:in `send'' >> /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:98:in `http_request'' >> /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:76:in `http_put'' >> /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:158:in `save'' >> /usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:271:in `save'' >> /usr/lib/ruby/site_ruby/1.8/puppet/node/facts.rb:20:in `save'' >> /usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:26:in >> `extract_facts_from_request'' >> /usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:31:in >> `find'' >> /usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:195:in `find'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:109:in >> `do_find'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `send'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `process'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick/rest.rb:24:in >> `service'' >> /usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'' >> /usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:45:in `listen'' >> /usr/lib/ruby/1.8/webrick/server.rb:173:in `call'' >> /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'' >> /usr/lib/ruby/1.8/webrick/server.rb:162:in `start'' >> /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'' >> /usr/lib/ruby/1.8/webrick/server.rb:95:in `start'' >> /usr/lib/ruby/1.8/webrick/server.rb:92:in `each'' >> /usr/lib/ruby/1.8/webrick/server.rb:92:in `start'' >> /usr/lib/ruby/1.8/webrick/server.rb:23:in `start'' >> /usr/lib/ruby/1.8/webrick/server.rb:82:in `start'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in >> `initialize'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `listen'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in >> `synchronize'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `listen'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:126:in `listen'' >> /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:141:in `start'' >> /usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:124:in `start'' >> /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:202:in `main'' >> /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:146:in >> `run_command'' >> /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'' >> /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:416:in `hook'' >> /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'' >> /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:407:in `exit_on_fail'' >> /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'' >> /usr/sbin/puppetmasterd:4 >> err: execution expired >> >-- CONFIDENTIALITÉ: Ce document transmis par courriel est destiné uniquement à la personne ou à l''entité à qui il est adressé et peut contenir des renseignements confidentiels et assujettis au secret professionnel. La confidentialité et le secret professionnel demeurent malgré l''envoi de ce document à la mauvaise personne. Si vous n''êtes pas le destinataire visé ou la personne chargée de remettre ce document à son destinataire, veuillez nous en informer par téléphone et détruire ce courriel immédiatement. Toute distribution, reproduction ou autre utilisation de ce courriel par un destinataire non visé est interdite. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/XKAWYTohhk8J. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.