phundisk
2013-May-21 13:18 UTC
[Puppet Users] Puppet - rbuf_fill timeout error periodically on puppet client runs
Occasionally I am seeing this error below on puppet client runs from
several different clients. The clients and the server in this environment
are rather old and this started to happen more after I used to nagios
module which uses stored configs. Does anyone know where this slowness
might be coming from (client, server, db, etc) or how I can increase this
timeout parameter? Or does this indicate some poor programming in my
puppet code in terms of inefficiency. I would say this happens maybe one
or two times a week out of 20 servers running every 30 minutes.
*Current Setup:*
Puppet Server: puppet-server-2.7.20-1
Ruby: 1.8.7.370
Stored Configs: MySQL 5.5.15
Cron Run Command (no service): /usr/sbin/puppetd --server puppet --onetime
--no-daemonize --verbose --logdest syslog
*Error*
/usr/lib/ruby/1.8/timeout.rb:54:in `rbuf_fill'': execution expired
(Timeout::
Error)
from /usr/lib/ruby/1.8/timeout.rb:56:in `timeout''
from /usr/lib/ruby/1.8/timeout.rb:76:in `timeout''
from /usr/lib/ruby/1.8/net/protocol.rb:132:in `rbuf_fill''
from /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil''
from /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline''
from /usr/lib/ruby/1.8/net/http.rb:2020:in `read_status_line''
from /usr/lib/ruby/1.8/net/http.rb:2009:in `read_new''
from /usr/lib/ruby/1.8/net/http.rb:1050:in `request''
... 24 levels...
from /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in
`run''
from /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:306:in
`exit_on_fail''
from /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in
`run''
from /usr/sbin/puppetd:159
--
_____________________________________________________
This email and any files transmitted with it are confidential and intended
solely for the addressee. If you received this email in error, please do
not disclose the contents to anyone; kindly notify the sender by return
email and delete this email and any attachments from your system.
© 2011 Currensee Inc. is a member of the National Futures Association (NFA)
Member ID 0403251 | Over the counter retail foreign currency (Forex)
trading may involve significant risk of loss. It is not suitable for all
investors and you should make sure you understand the risks involved before
trading and seek independent advice if necessary. Performance, strategies
and charts shown are not necessarily predictive of any particular result
and past performance is no indication of future results. Investor returns
may vary from Trade Leader returns based on slippage, fees, broker spreads,
volatility or other market conditions.
Currensee Inc | 54 Canal St 4th Floor | Boston, MA 02114 | +1.617.624.3824
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
jcbollinger
2013-May-22 13:09 UTC
[Puppet Users] Re: Puppet - rbuf_fill timeout error periodically on puppet client runs
On Tuesday, May 21, 2013 8:18:58 AM UTC-5, phundisk wrote:> > Occasionally I am seeing this error below on puppet client runs from > several different clients. The clients and the server in this environment > are rather old and this started to happen more after I used to nagios > module which uses stored configs. Does anyone know where this slowness > might be coming from (client, server, db, etc) or how I can increase this > timeout parameter? Or does this indicate some poor programming in my > puppet code in terms of inefficiency. I would say this happens maybe one > or two times a week out of 20 servers running every 30 minutes. > > *Current Setup:* > Puppet Server: puppet-server-2.7.20-1 > Ruby: 1.8.7.370 > Stored Configs: MySQL 5.5.15 > Cron Run Command (no service): /usr/sbin/puppetd --server puppet --onetime > --no-daemonize --verbose --logdest syslog > > *Error* > /usr/lib/ruby/1.8/timeout.rb:54:in `rbuf_fill'': execution expired > (Timeout::Error) > from /usr/lib/ruby/1.8/timeout.rb:56:in `timeout'' > from /usr/lib/ruby/1.8/timeout.rb:76:in `timeout'' > from /usr/lib/ruby/1.8/net/protocol.rb:132:in `rbuf_fill'' > from /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'' > from /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'' > from /usr/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'' > from /usr/lib/ruby/1.8/net/http.rb:2009:in `read_new'' > from /usr/lib/ruby/1.8/net/http.rb:1050:in `request'' > ... 24 levels... > from /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in > `run'' > from /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:306:in > `exit_on_fail'' > from /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in > `run'' > from /usr/sbin/puppetd:159 > >It looks like you are running about at the edge of your master''s capacity. Especially since you''re running out of cron, that could be related to an uneven workload, where all of your servers make catalog requests at approximately the same time. If you have not already done so, then you should ensure that your crontab entries spread out requests relatively evenly over time. Are you running the master via its built-in webrick server, or have you set it up in apache/passenger, mongrel, or some other more featureful rack server? The webrick version is single-threaded, serving only one client at a time, and that can cause delays and client-side timeouts when multiple requests arrive close together, even if the server has enough raw power that it should be able to serve them all adequately. Twenty clients is in the range where the single-threaded webrick server may start to have trouble keeping up, though that depends heavily on your manifests and the pattern of requests. I have a webrick server supporting many more clients than that, but the manifest set it''s working with is pretty simple. Nagios resources themselves are not a special problem, but exported resources and stored configs very well could be. Do use thin stored configs if possible, as that will lighten the load somewhat. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
phundisk
2013-May-22 18:39 UTC
[Puppet Users] Re: Puppet - rbuf_fill timeout error periodically on puppet client runs
I am using passenger setup right now. I agree with your thought that it is either the addition of tons of exported resources or the puppet master server. I will run some experiments and see if this can be resolved. On Tuesday, May 21, 2013 9:18:58 AM UTC-4, phundisk wrote:> > Occasionally I am seeing this error below on puppet client runs from > several different clients. The clients and the server in this environment > are rather old and this started to happen more after I used to nagios > module which uses stored configs. Does anyone know where this slowness > might be coming from (client, server, db, etc) or how I can increase this > timeout parameter? Or does this indicate some poor programming in my > puppet code in terms of inefficiency. I would say this happens maybe one > or two times a week out of 20 servers running every 30 minutes. > > *Current Setup:* > Puppet Server: puppet-server-2.7.20-1 > Ruby: 1.8.7.370 > Stored Configs: MySQL 5.5.15 > Cron Run Command (no service): /usr/sbin/puppetd --server puppet --onetime > --no-daemonize --verbose --logdest syslog > > *Error* > /usr/lib/ruby/1.8/timeout.rb:54:in `rbuf_fill'': execution expired > (Timeout::Error) > from /usr/lib/ruby/1.8/timeout.rb:56:in `timeout'' > from /usr/lib/ruby/1.8/timeout.rb:76:in `timeout'' > from /usr/lib/ruby/1.8/net/protocol.rb:132:in `rbuf_fill'' > from /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'' > from /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'' > from /usr/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'' > from /usr/lib/ruby/1.8/net/http.rb:2009:in `read_new'' > from /usr/lib/ruby/1.8/net/http.rb:1050:in `request'' > ... 24 levels... > from /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in > `run'' > from /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:306:in > `exit_on_fail'' > from /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in > `run'' > from /usr/sbin/puppetd:159 > > _____________________________________________________ > This email and any files transmitted with it are confidential and intended > solely for the addressee. If you received this email in error, please do > not disclose the contents to anyone; kindly notify the sender by return > email and delete this email and any attachments from your system. > > © 2011 Currensee Inc. is a member of the National Futures Association > (NFA) Member ID 0403251 | Over the counter retail foreign currency (Forex) > trading may involve significant risk of loss. It is not suitable for all > investors and you should make sure you understand the risks involved before > trading and seek independent advice if necessary. Performance, strategies > and charts shown are not necessarily predictive of any particular result > and past performance is no indication of future results. Investor returns > may vary from Trade Leader returns based on slippage, fees, broker spreads, > volatility or other market conditions. > > Currensee Inc | 54 Canal St 4th Floor | Boston, MA 02114 | +1.617.624.3824 >-- _____________________________________________________ This email and any files transmitted with it are confidential and intended solely for the addressee. If you received this email in error, please do not disclose the contents to anyone; kindly notify the sender by return email and delete this email and any attachments from your system. © 2011 Currensee Inc. is a member of the National Futures Association (NFA) Member ID 0403251 | Over the counter retail foreign currency (Forex) trading may involve significant risk of loss. It is not suitable for all investors and you should make sure you understand the risks involved before trading and seek independent advice if necessary. Performance, strategies and charts shown are not necessarily predictive of any particular result and past performance is no indication of future results. Investor returns may vary from Trade Leader returns based on slippage, fees, broker spreads, volatility or other market conditions. Currensee Inc | 54 Canal St 4th Floor | Boston, MA 02114 | +1.617.624.3824 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
jcbollinger
2013-May-23 12:48 UTC
[Puppet Users] Re: Puppet - rbuf_fill timeout error periodically on puppet client runs
On Wednesday, May 22, 2013 1:39:46 PM UTC-5, phundisk wrote:> > I am using passenger setup right now. I agree with your thought that it > is either the addition of tons of exported resources or the puppet master > server. I will run some experiments and see if this can be resolved. > >Don''t neglect the timing issue -- even with passenger, you could still have problems if your clients all check in at about the same time. You will make the most efficient use of your server''s capacity if client requests are distributed evenly over time. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.