For anyone that has experienced the odd End of file or Cannot describe errors, we''ve found that switching to Mongrel has fixed this problem for us. We''re currently running 5 instance of puppetmaster under mongrel (with the apache proxy in front) and things are going great. For more information on setting up Mongrel, visit: <http://reductivelabs.com/trac/puppet/wiki/UsingMongrel> -- Digant C Kasundra <digant@stanford.edu> Technical Lead, ITS Unix Systems and Applications, Stanford University
On Tue, Oct 02, 2007 at 12:22:06PM -0700, Digant C Kasundra wrote:> For anyone that has experienced the odd End of file or Cannot describe > errors,If those errors have only started cropping up in the last couple of weeks, then they''re probably the bugs I fixed with josb on IRC last week. Just upgrade to 0.23.2-10; it''s got all the fixes in there and saves the hassle of setting up mongrel. If they''re unrelated errors, reporting them might be better than masking them. - Matt -- I have always wished that my computer would be as easy to use as my telephone. My wish has come true. I no longer know how to use my telephone. -- Bjarne Stroustrup
--On Wednesday, October 03, 2007 05:55:43 AM +1000 Matthew Palmer <mpalmer@hezmatt.org> wrote:>> For anyone that has experienced the odd End of file or Cannot describe >> errors, > > If those errors have only started cropping up in the last couple of weeks, > then they''re probably the bugs I fixed with josb on IRC last week. Just > upgrade to 0.23.2-10; it''s got all the fixes in there and saves the hassle > of setting up mongrel. > > If they''re unrelated errors, reporting them might be better than masking > them. > > - MattThe "End of file" and "Could not describe" errors have been reported repeatedly and the root cause was never located. They have existed for many versions of Puppet, perhaps forever. -- Digant C Kasundra <digant@stanford.edu> Technical Lead, ITS Unix Systems and Applications, Stanford University
On Tue, Oct 02, 2007 at 01:57:02PM -0700, Digant C Kasundra wrote:> --On Wednesday, October 03, 2007 05:55:43 AM +1000 Matthew Palmer > <mpalmer@hezmatt.org> wrote: > > >> For anyone that has experienced the odd End of file or Cannot describe > >> errors, > > > > If those errors have only started cropping up in the last couple of weeks, > > then they''re probably the bugs I fixed with josb on IRC last week. Just > > upgrade to 0.23.2-10; it''s got all the fixes in there and saves the hassle > > of setting up mongrel. > > > > If they''re unrelated errors, reporting them might be better than masking > > them. > > The "End of file" and "Could not describe" errors have been reported > repeatedly and the root cause was never located. They have existed for > many versions of Puppet, perhaps forever.And yet I''ve never seen them. How strange. - Matt -- Sure, it''s possible to write C in an object-oriented way. But, in practice, getting an entire team to do that is like telling them to walk along a straight line painted on the floor, with the lights off. -- Tess Snider, slug-chat@slug.org.au
On Oct 2, 2007, at 4:19 PM, Matthew Palmer wrote:>> The "End of file" and "Could not describe" errors have been reported >> repeatedly and the root cause was never located. They have >> existed for >> many versions of Puppet, perhaps forever. > > And yet I''ve never seen them. How strange.I''ve talked at length with Digant about the errors. It''s clearly based on how Puppet is used, so it''s not surprising that one site would see them and others not. -- Man is the only animal that can remain on friendly terms with the victims he intends to eat until he eats them. -- Samuel Butler (1835-1902) --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Luke Kanies wrote:> On Oct 2, 2007, at 4:19 PM, Matthew Palmer wrote: > > >>> The "End of file" and "Could not describe" errors have been reported >>> repeatedly and the root cause was never located. They have >>> existed for >>> many versions of Puppet, perhaps forever. >>> >> And yet I''ve never seen them. How strange. >> > > It''s clearly based on how Puppet is used, so it''s not surprising that > one site would see them and others not. >Luke, Could you please elaborate? What type of usage can raise these errors? I was seeing this often, then i split the fileservering off to a separate process and port which improved it but i am still seeing it occasionally. I had a crack at Mongrel but on RHEL i wasn''t successful, i see David now has the mongrel rpms in his repo so i may try again when i have time. Ben
On Oct 2, 2007, at 7:17 PM, Ben wrote:> Could you please elaborate? What type of usage can raise these > errors?I''m not really sure, actually. I expect it''s generally related to having lots of clients connect to the server at once, probably specifically involving file serving.> I was seeing this often, then i split the fileservering off to a > separate process and port which improved it but i am still seeing it > occasionally. > > I had a crack at Mongrel but on RHEL i wasn''t successful, i see David > now has the mongrel rpms in his repo so i may try again when i have > time.Well, we''ve found that webrick starts throwing resets beyond about 7 concurrent connections, whereas mongrel is good for about 30. -- I had a linguistics professor who said that it''s man''s ability to use language that makes him the dominant species on the planet. That may be. But I think there''s one other thing that separates us from animals. We aren''t afraid of vacuum cleaners. --Jeff Stilson --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
--On Wednesday, October 03, 2007 7:19 AM +1000 Matthew Palmer <mpalmer@hezmatt.org> wrote:>> >> For anyone that has experienced the odd End of file or Cannot describe >> >> errors, >> > >> > If those errors have only started cropping up in the last couple of >> > weeks, then they''re probably the bugs I fixed with josb on IRC last >> > week. Just upgrade to 0.23.2-10; it''s got all the fixes in there and >> > saves the hassle of setting up mongrel. >> > >> > If they''re unrelated errors, reporting them might be better than >> > masking them. >> >> The "End of file" and "Could not describe" errors have been reported >> repeatedly and the root cause was never located. They have existed for >> many versions of Puppet, perhaps forever. > > And yet I''ve never seen them. How strange.I would never recommend masking problems as an alternative to reporting them. Discussion has probably not been as public as I had thought, but Luke and I have talked about it often. <http://www.reductivelabs.com/cgi-bin/puppet.cgi/ticket/506> <http://www.reductivelabs.com/cgi-bin/puppet.cgi/ticket/721> <http://mail.madstop.com/pipermail/puppet-users/2007-July/003923.html> <http://mail.madstop.com/pipermail/puppet-users/2007-April/002375.html>
On Tue, Oct 02, 2007 at 09:46:04PM -0500, Luke Kanies wrote:> Well, we''ve found that webrick starts throwing resets beyond about 7 > concurrent connections, whereas mongrel is good for about 30.For the record: the problem I''m seeing here with Matt''s 0.23.2-10 puppetmasterd using the stock Webrick is that it randomly stops accepting new connections after some time. This in turn causes all our puppetd''s to continuously restart (it doesn''t seem to handle this conditin gracefully) until the puppetmasterd is restarted. The problem seems to center around the while loop in Webrick''s GenericServer.start which calls accept_client(). Running lsof/strace on the puppetmasterd shows that the MasterPort listen socket stops appearing in the select() readfds mask used by the Ruby interpreter. Also at this point accept_client() is no longer called so maybe the thread is stuck somehow. I haven''t determined yet what the thread managing the listen socket is doing at the time when the listen socket fd disappears from the select mask. Increasing Webrick''s MaxClients (which determines size of the @tokens SizedQeue) from within puppetmasterd doesn''t appear to fix the problem but seems to make it occur less frequently. But maybe I''m imaging that. -- Jos Backus jos at catnook.com
Jos, This sounds very similar to the symptoms of not having the mysql gem installed while using storeconfigs (apparently the default ruby mysql connector has issues). If this is the issue, then you''re freezing when puppetmaster calls freshness and tries to connect to the database seemingly after the connection has timed out. After this puppetmaster never accepts another connection and all the clients die from timeouts. Thanks Brian On 10/9/07, Jos Backus <jos@catnook.com> wrote:> On Tue, Oct 02, 2007 at 09:46:04PM -0500, Luke Kanies wrote: > > Well, we''ve found that webrick starts throwing resets beyond about 7 > > concurrent connections, whereas mongrel is good for about 30. > > For the record: the problem I''m seeing here with Matt''s 0.23.2-10 > puppetmasterd using the stock Webrick is that it randomly stops accepting new > connections after some time. This in turn causes all our puppetd''s to > continuously restart (it doesn''t seem to handle this conditin gracefully) > until the puppetmasterd is restarted. > > The problem seems to center around the while loop in Webrick''s > GenericServer.start which calls accept_client(). Running lsof/strace on the > puppetmasterd shows that the MasterPort listen socket stops appearing in the > select() readfds mask used by the Ruby interpreter. Also at this point > accept_client() is no longer called so maybe the thread is stuck somehow. I > haven''t determined yet what the thread managing the listen socket is doing at > the time when the listen socket fd disappears from the select mask. > > Increasing Webrick''s MaxClients (which determines size of the @tokens > SizedQeue) from within puppetmasterd doesn''t appear to fix the problem but > seems to make it occur less frequently. But maybe I''m imaging that. > > -- > Jos Backus > jos at catnook.com > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users >
Hi Brian, On Tue, Oct 09, 2007 at 05:02:33PM -0700, Brian Finney wrote:> Jos, > > This sounds very similar to the symptoms of not having the mysql gem > installed while using storeconfigs (apparently the default ruby mysql > connector has issues). > > If this is the issue, then you''re freezing when puppetmaster calls > freshness and tries to connect to the database seemingly after the > connection has timed out. After this puppetmaster never accepts > another connection and all the clients die from timeouts.Thanks, but this setup doesn''t use MySQL or storeconfigs at all, so I doubt the above scenario applies. Jos> Thanks > Brian > > On 10/9/07, Jos Backus <jos@catnook.com> wrote: > > On Tue, Oct 02, 2007 at 09:46:04PM -0500, Luke Kanies wrote: > > > Well, we''ve found that webrick starts throwing resets beyond about 7 > > > concurrent connections, whereas mongrel is good for about 30. > > > > For the record: the problem I''m seeing here with Matt''s 0.23.2-10 > > puppetmasterd using the stock Webrick is that it randomly stops accepting new > > connections after some time. This in turn causes all our puppetd''s to > > continuously restart (it doesn''t seem to handle this conditin gracefully) > > until the puppetmasterd is restarted. > > > > The problem seems to center around the while loop in Webrick''s > > GenericServer.start which calls accept_client(). Running lsof/strace on the > > puppetmasterd shows that the MasterPort listen socket stops appearing in the > > select() readfds mask used by the Ruby interpreter. Also at this point > > accept_client() is no longer called so maybe the thread is stuck somehow. I > > haven''t determined yet what the thread managing the listen socket is doing at > > the time when the listen socket fd disappears from the select mask. > > > > Increasing Webrick''s MaxClients (which determines size of the @tokens > > SizedQeue) from within puppetmasterd doesn''t appear to fix the problem but > > seems to make it occur less frequently. But maybe I''m imaging that. > > > > -- > > Jos Backus > > jos at catnook.com > > _______________________________________________ > > Puppet-users mailing list > > Puppet-users@madstop.com > > https://mail.madstop.com/mailman/listinfo/puppet-users > >-- Jos Backus jos at catnook.com