I just upgraded master and clients to 2.6.2, i''m on a passenger setup, and I''m seeing some odd things with puppet runs. Sometimes runs are relatively fast, and they say they finish, I see this printed out: notice: Finished catalog run in 81.08 seconds but, i''m not dropped back to the command prompt, it hangs here for a long time, and then after some time I see: err: Could not run Puppet configuration client: execution expired What is happening at this stage that is failing? I''m running passenger 2.2.11, libactiverecord-ruby1.8 2.3.5, libactivesupport-ruby1.8 2.3.5 and storedconfigs on mysql. This wasn''t happening when I was running on 0.25.5. passenger-status tells me: ----------- General information ----------- max = 6 count = 6 active = 6 inactive = 0 Waiting on global queue: 13 ----------- Domains ----------- /usr/share/puppet/rack/puppetmasterd: PID: 1988 Sessions: 1 Processed: 70 Uptime: 16m 14s PID: 1934 Sessions: 1 Processed: 297 Uptime: 18m 22s PID: 2951 Sessions: 1 Processed: 134 Uptime: 15m 55s PID: 1801 Sessions: 1 Processed: 228 Uptime: 20m 25s PID: 1788 Sessions: 1 Processed: 165 Uptime: 20m 32s PID: 1818 Sessions: 1 Processed: 139 Uptime: 20m 18s and I am seeing 6 spawned processes which are each eating 32% of the cpu, running quite hard. I''ve run the client and the master in debug verbose mode to try and track down anything interesting, but its not that revealing. This is what I see: on master: ... debug: File[/etc/munin/plugin-conf.d/passenger_stats.conf]: Adding default for ignore notice: Compiled catalog for foo.bar in environment production in 206.49 seconds info: Caching catalog for foo.bar info: Queued catalog for foo.bar in 44.29 seconds info: Could not find filesystem info for file ''modules/apt/modules_dir'' in environment production info: Could not find filesystem info for file ''modules/munin/plugins/modules_dir'' in environment production info: Could not find filesystem info for file ''modules/munin/nodes/modules_dir'' in environment production info: Could not find filesystem info for file ''modules/shorewall/modules_dir'' in environment production info: Could not find filesystem info for file ''modules/virtual/modules_dir'' in environment production info: Could not find filesystem info for file ''modules/virtual/contexts/modules_dir'' in environment production on client: ... debug: Finishing transaction -622176998 debug: Storing state debug: Stored state in 1.68 seconds notice: Finished catalog run in 93.40 seconds debug: Using cached certificate for ca debug: Using cached certificate for foo.bar debug: Using cached certificate_revocation_list for ca debug: Value of ''preferred_serialization_format'' (pson) is invalid for report, using default (b64_zlib_yaml) debug: report supports formats: b64_zlib_yaml marshal raw yaml; using b64_zlib_yaml * long pause here * Any ideas of things I might look into? thanks! micah
On Mon, Nov 1, 2010 at 11:19 PM, Micah Anderson <micah@riseup.net> wrote:> > I just upgraded master and clients to 2.6.2, i''m on a passenger setup, > and I''m seeing some odd things with puppet runs. Sometimes runs are > relatively fast, and they say they finish, I see this printed out: > > notice: Finished catalog run in 81.08 seconds > > but, i''m not dropped back to the command prompt, it hangs here for a > long time, and then after some time I see: > > err: Could not run Puppet configuration client: execution expired > > What is happening at this stage that is failing?I''d love to see more info on this. I was seeing this problem every so often but only when bootstrapping, and wasn''t able to work out what the underlying problem was as it simply took too long to reproduce each time.> > I''m running passenger 2.2.11, libactiverecord-ruby1.8 2.3.5, > libactivesupport-ruby1.8 2.3.5 and storedconfigs on mysql. This wasn''t > happening when I was running on 0.25.5. > > passenger-status tells me: > > ----------- General information ----------- > max = 6 > count = 6 > active = 6 > inactive = 0 > Waiting on global queue: 13 > > ----------- Domains ----------- > /usr/share/puppet/rack/puppetmasterd: > PID: 1988 Sessions: 1 Processed: 70 Uptime: 16m 14s > PID: 1934 Sessions: 1 Processed: 297 Uptime: 18m 22s > PID: 2951 Sessions: 1 Processed: 134 Uptime: 15m 55s > PID: 1801 Sessions: 1 Processed: 228 Uptime: 20m 25s > PID: 1788 Sessions: 1 Processed: 165 Uptime: 20m 32s > PID: 1818 Sessions: 1 Processed: 139 Uptime: 20m 18s > > > and I am seeing 6 spawned processes which are each eating 32% of the > cpu, running quite hard. > > I''ve run the client and the master in debug verbose mode to try and > track down anything interesting, but its not that revealing. This is > what I see: > > on master: > > ... > > debug: File[/etc/munin/plugin-conf.d/passenger_stats.conf]: Adding default for ignore > notice: Compiled catalog for foo.bar in environment production in 206.49 seconds > info: Caching catalog for foo.bar > info: Queued catalog for foo.bar in 44.29 seconds > info: Could not find filesystem info for file ''modules/apt/modules_dir'' in environment production > info: Could not find filesystem info for file ''modules/munin/plugins/modules_dir'' in environment production > info: Could not find filesystem info for file ''modules/munin/nodes/modules_dir'' in environment production > info: Could not find filesystem info for file ''modules/shorewall/modules_dir'' in environment production > info: Could not find filesystem info for file ''modules/virtual/modules_dir'' in environment production > info: Could not find filesystem info for file ''modules/virtual/contexts/modules_dir'' in environment production > > > on client: > > ... > > debug: Finishing transaction -622176998 > debug: Storing state > debug: Stored state in 1.68 seconds > notice: Finished catalog run in 93.40 seconds > debug: Using cached certificate for ca > debug: Using cached certificate for foo.bar > debug: Using cached certificate_revocation_list for ca > debug: Value of ''preferred_serialization_format'' (pson) is invalid for report, using default (b64_zlib_yaml) > debug: report supports formats: b64_zlib_yaml marshal raw yaml; using b64_zlib_yaml > * long pause here * > > Any ideas of things I might look into? > > thanks! > micah >-- Nigel Kersten - Puppet Labs - http://www.puppetlabs.com -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Nigel Kersten <nigel@puppetlabs.com> writes:> On Mon, Nov 1, 2010 at 11:19 PM, Micah Anderson <micah@riseup.net> wrote: >> >> I just upgraded master and clients to 2.6.2, i''m on a passenger setup, >> and I''m seeing some odd things with puppet runs. Sometimes runs are >> relatively fast, and they say they finish, I see this printed out: >> >> notice: Finished catalog run in 81.08 seconds >> >> but, i''m not dropped back to the command prompt, it hangs here for a >> long time, and then after some time I see: >> >> err: Could not run Puppet configuration client: execution expired >> >> What is happening at this stage that is failing? > > I''d love to see more info on this. I was seeing this problem every so > often but only when bootstrapping, and wasn''t able to work out what > the underlying problem was as it simply took too long to reproduce > each time.Looking at our puppetmaster system after the upgrade, our master is now pegged, check out these munin graphs: http://micah.riseup.net/shots/2010.11.01-22.43.1288665832.05NH70xQT5.png http://micah.riseup.net/shots/2010.11.01-22.44.1288665880.GiDdNyIib9.png http://micah.riseup.net/shots/2010.11.01-23.11.1288667460.nJrTAKRSjT.png http://micah.riseup.net/shots/2010.11.01-23.33.1288668801.BfCPa0ggQo.png http://micah.riseup.net/shots/2010.11.01-23.33.1288668838.LBYS836Z6e.png http://micah.riseup.net/shots/2010.11.01-23.34.1288668862.P86SQ9YZcC.png http://micah.riseup.net/shots/2010.11.01-23.34.1288668889.KKwzqhdubP.png http://micah.riseup.net/shots/2010.11.01-23.34.1288668889.KKwzqhdubP.png i''m running the puppetqd with async_storeconfigs=true, in case that is interesting... micah -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Interestingly, I decided to try and switch from passenger to nginx to see if that changed anything, and behold! Everything is working as it should. So, something is strange with 2.6.2 and passenger, because I was running a perfectly functional passenger setup with 0.25.5, and the only thing I changed was puppet which caused things to go sour. micah --
On Mon, Nov 1, 2010 at 4:19 PM, Micah Anderson <micah@riseup.net> wrote:> > I just upgraded master and clients to 2.6.2, i''m on a passenger setup, > and I''m seeing some odd things with puppet runs. Sometimes runs are > relatively fast, and they say they finish, I see this printed out: > > notice: Finished catalog run in 81.08 seconds > > but, i''m not dropped back to the command prompt, it hangs here for a > long time, and then after some time I see:I suggest trying --trace and --debug to see what the puppet agent is doing at this point. It sounds like StoredConfigs may be taking a very long time to write the data into MySQL. -- Jeff McCune http://www.puppetlabs.com/ -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
I would like to follow up on this issue, because it was never really addressed, as the OP switched to nginx. I am experiencing a few of those "finished, yet execution expired" runs, and I can see from the agent''s run (using --trace, as suggested by Jeff) that there''s a timeout (/usr/lib/ruby/1.8/timeout.rb:64:in `rbuf_fill''). Interestingly enough, this event occurs systematically on one type of hosts (I only use role based conf defined by environment), while all other types are completing their runs successfully. Is this the right thread to post in or would it be better to open a new one, yet with the same very effective subject? Many thanks, Angelo -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
I''d suggest waiting for 2.6.6 to be fully released From http://projects.puppetlabs.com/projects/1/wiki/Release_Notes#2.6.5 Faster Passenger support Bug #6257 <http://projects.puppetlabs.com/issues/6257>: Rack POST and PUT request handling is very slow. The speed of the Rack HTTP handler has been dramatically improved. This should prevent timeouts that some users were experiencing when running under Passenger. John On 9 March 2011 03:24, Angelo Corbo <acorbo@chelseatechnology.co.uk> wrote:> I would like to follow up on this issue, because it was never really > addressed, as the OP switched to nginx. > > I am experiencing a few of those "finished, yet execution expired" runs, > and I can see from the agent''s run (using --trace, as suggested by Jeff) > that there''s a timeout (/usr/lib/ruby/1.8/timeout.rb:64:in `rbuf_fill''). > > Interestingly enough, this event occurs systematically on one type of hosts > (I only use role based conf defined by environment), while all other types > are completing their runs successfully. > > Is this the right thread to post in or would it be better to open a new > one, yet with the same very effective subject? > > Many thanks, > > Angelo > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- John Warburton Ph: 0417 299 600 Email: jwarburton@gmail.com -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
…and now it is. :)
On a more serious note, if that doesn''t resolve your problem, please
file a ticket with us so we can work to track it down. We do think
these are very, very important problems, and would like to eliminate
the last of them from our system.
Thanks,
Daniel
On Tue, Mar 8, 2011 at 14:29, John Warburton <jwarburton@gmail.com>
wrote:> I''d suggest waiting for 2.6.6 to be fully released
>
> From http://projects.puppetlabs.com/projects/1/wiki/Release_Notes#2.6.5
>
> Faster Passenger support
>
> Bug #6257: Rack POST and PUT request handling is very slow.
>
> The speed of the Rack HTTP handler has been dramatically improved. This
> should prevent timeouts that some users were experiencing when running
under
> Passenger.
>
> John
>
> On 9 March 2011 03:24, Angelo Corbo <acorbo@chelseatechnology.co.uk>
wrote:
>>
>> I would like to follow up on this issue, because it was never really
>> addressed, as the OP switched to nginx.
>>
>> I am experiencing a few of those "finished, yet execution
expired" runs,
>> and I can see from the agent''s run (using --trace, as
suggested by Jeff)
>> that there''s a timeout (/usr/lib/ruby/1.8/timeout.rb:64:in
`rbuf_fill'').
>>
>> Interestingly enough, this event occurs systematically on one type of
>> hosts (I only use role based conf defined by environment), while all
other
>> types are completing their runs successfully.
>>
>> Is this the right thread to post in or would it be better to open a new
>> one, yet with the same very effective subject?
>>
>> Many thanks,
>>
>> Angelo
>>
>> --
>> You received this message because you are subscribed to the Google
Groups
>> "Puppet Users" group.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> To unsubscribe from this group, send email to
>> puppet-users+unsubscribe@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/puppet-users?hl=en.
>
>
>
> --
> John Warburton
> Ph: 0417 299 600
> Email: jwarburton@gmail.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
--
⎋ Puppet Labs Developer – http://puppetlabs.com
✉ Daniel Pittman <daniel@puppetlabs.com>
✆ Contact me via gtalk, email, or phone: +1 (877) 575-9775
♲ Made with 100 percent post-consumer electrons
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Thank you very much, Daniel. Just to confirm that upgrading from version 2.6.4 to 2.6.6 solved the problem. Angelo -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
In my case is also happening with 2.6.8. making async_storeconfigs=false Seemed to resolve the issue. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Fri, May 27, 2011 at 8:39 PM, Larry Ludwig <larrylud@gmail.com> wrote:> In my case is also happening with 2.6.8. > > making > > async_storeconfigs=false > > Seemed to resolve the issue. >hrm. that''s rather strange. Daniel, if you''re still watching this thread, do you have any comment on whether your fix for: http://projects.puppetlabs.com/issues/5318 could possibly have resolved this issue? For those of you who can somewhat reliably reproduce this, would it be possible to test a master running against 2.6.x HEAD to see if it resolved it?> > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- Nigel Kersten Product, Puppet Labs @nigelkersten -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Actually upon further investigation it turned out to be an odd iptables firewall rule, unrelated to puppet. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.