While running puppetd on a linux client I get the following, the actual spot that the segfault happens varies, but it always happens. This is on ubuntu with the stock apt ruby1.8.4 package. Puppet also installed from apt. Puppet version 0.18.4 It seems very likely that this isnt a bug in puppet per se, but it is tickling a yaml bug (regardless of where it fails, it is always yaml). But I figure someone else may have seen this. Thanks, Andy Delcambre # puppetd -d debug: puppetd: Setting logdir to ''/var/log/puppet'' debug: puppetd: Setting vardir to ''/var/lib/puppet'' debug: puppetd: Setting rundir to ''/var/run'' debug: puppetconfig/puppet/file=/var/lib/puppet/state/state.yaml: Autorequiring file /var/lib/puppet/state debug: puppetconfig/puppet/file=/var/lib/puppet/state/state.yaml: subscribes to /var/lib/puppet/state debug: puppetconfig/puppet/file=/etc/puppet/ssl: Autorequiring file /etc/puppet debug: puppetconfig/puppet/file=/etc/puppet/ssl: subscribes to /etc/puppet debug: puppetconfig/puppet/file=/etc/puppet/namespaceauth.conf: Autorequiring file /etc/puppet debug: puppetconfig/puppet/file=/etc/puppet/namespaceauth.conf: subscribes to /etc/puppet debug: puppetconfig/certificates/file=/etc/puppet/ssl/certs: Autorequiring file /etc/puppet debug: puppetconfig/certificates/file=/etc/puppet/ssl/certs: Autorequiring file /etc/puppet/ssl debug: puppetconfig/certificates/file=/etc/puppet/ssl/certs: subscribes to /etc/puppet debug: puppetconfig/certificates/file=/etc/puppet/ssl/certs: subscribes to /etc/puppet/ssl debug: puppetconfig/certificates/file=/etc/puppet/ssl/private: Autorequiring file /etc/puppet debug: puppetconfig/certificates/file=/etc/puppet/ssl/private: Autorequiring file /etc/puppet/ssl debug: puppetconfig/certificates/file=/etc/puppet/ssl/private: subscribes to /etc/puppet debug: puppetconfig/certificates/file=/etc/puppet/ssl/private: subscribes to /etc/puppet/ssl debug: puppetconfig/certificates/file=/etc/puppet/ssl/private/password: Autorequiring file /etc/puppet debug: puppetconfig/certificates/file=/etc/puppet/ssl/private/password: Autorequiring file /etc/puppet/ssl debug: puppetconfig/certificates/file=/etc/puppet/ssl/private/password: Autorequiring file /etc/puppet/ssl/private debug: puppetconfig/certificates/file=/etc/puppet/ssl/private/password: subscribes to /etc/puppet debug: puppetconfig/certificates/file=/etc/puppet/ssl/private/password: subscribes to /etc/puppet/ssl debug: puppetconfig/certificates/file=/etc/puppet/ssl/private/password: subscribes to /etc/puppet/ssl/private debug: puppetconfig/certificates/file=/etc/puppet/ssl/private_keys: Autorequiring file /etc/puppet debug: puppetconfig/certificates/file=/etc/puppet/ssl/private_keys: Autorequiring file /etc/puppet/ssl debug: puppetconfig/certificates/file=/etc/puppet/ssl/private_keys: subscribes to /etc/puppet debug: puppetconfig/certificates/file=/etc/puppet/ssl/private_keys: subscribes to /etc/puppet/ssl debug: puppetconfig/certificates/file=/etc/puppet/ssl/public_keys: Autorequiring file /etc/puppet debug: puppetconfig/certificates/file=/etc/puppet/ssl/public_keys: Autorequiring file /etc/puppet/ssl debug: puppetconfig/certificates/file=/etc/puppet/ssl/public_keys: subscribes to /etc/puppet debug: puppetconfig/certificates/file=/etc/puppet/ssl/public_keys: subscribes to /etc/puppet/ssl debug: puppetconfig/puppet/file=/var/lib/puppet/state/state.yaml: File does not exist debug: puppetconfig/puppet/file=/var/lib/puppet/state/state.yaml: Changing mode debug: puppetconfig/puppet/file=/var/lib/puppet/state/state.yaml: 1 change(s) debug: puppetconfig/puppet/file=/var/lib/puppet/state/state.yaml/mode: File does not exist; cannot set mode debug: puppetconfig/puppet/file=/var/lib/puppet/plugins: File does not exist debug: puppetconfig/certificates/file=/etc/puppet/ssl/private/password: File does not exist debug: puppetconfig/certificates/file=/etc/puppet/ssl/private/password: Changing mode debug: puppetconfig/certificates/file=/etc/puppet/ssl/private/password: 1 change(s) debug: puppetconfig/certificates/file=/etc/puppet/ssl/private/password/mode: File does not exist; cannot set mode debug: Finishing transaction -608763208 with 2 changes notice: Starting Puppet client version 0.18.4 notice: Stale lockfile /var/lib/puppet/state/puppetdlock left by process 22425; removing debug: getting config debug: Calling puppetmaster.getconfig debug: Retrieved configuration in 3.96 seconds debug: puppetconfig/puppetd/file=/var/log/puppet/puppetd.log: Autorequiring file /var/log/puppet debug: puppetconfig/puppetd/file=/var/log/puppet/puppetd.log: subscribes to /var/log/puppet debug: puppetconfig/puppetd/file=/var/log/puppet/http.log: Autorequiring file /var/log/puppet debug: puppetconfig/puppetd/file=/var/log/puppet/http.log: subscribes to /var/log/puppet debug: puppetconfig/puppetd/file=/var/lib/puppet/state/puppetdlock: Autorequiring file /var/lib/puppet debug: puppetconfig/puppetd/file=/var/lib/puppet/state/puppetdlock: subscribes to /var/lib/puppet debug: puppetconfig/puppetd/file=/var/log/puppet/puppetd.log: File does not exist debug: puppetconfig/puppetd/file=/var/log/puppet/puppetd.log: Changing owner,mode debug: puppetconfig/puppetd/file=/var/log/puppet/puppetd.log: 2 change(s) debug: puppetconfig/puppetd/file=/var/log/puppet/puppetd.log/owner: File does not exist; cannot set owner debug: puppetconfig/puppetd/file=/var/log/puppet/puppetd.log/mode: File does not exist; cannot set mode debug: puppetconfig/puppetd/file=/var/log/puppet/http.log: File does not exist debug: puppetconfig/puppetd/file=/var/log/puppet/http.log: Changing owner,mode debug: puppetconfig/puppetd/file=/var/log/puppet/http.log: 2 change(s) debug: puppetconfig/puppetd/file=/var/log/puppet/http.log/owner: File does not exist; cannot set owner debug: puppetconfig/puppetd/file=/var/log/puppet/http.log/mode: File does not exist; cannot set mode debug: puppetconfig/puppetd/file=/etc/puppet/localconfig: File does not exist debug: puppetconfig/puppetd/file=/etc/puppet/localconfig: Changing owner,mode debug: puppetconfig/puppetd/file=/etc/puppet/localconfig: 2 change(s) debug: puppetconfig/puppetd/file=/etc/puppet/localconfig/owner: File does not exist; cannot set owner debug: puppetconfig/puppetd/file=/etc/puppet/localconfig/mode: File does not exist; cannot set mode debug: Finishing transaction -608323498 with 6 changes info: Caching configuration at /etc/puppet/localconfig.yaml /usr/lib/ruby/1.8/yaml.rb:133: [BUG] Segmentation fault ruby 1.8.4 (2005-12-24) [i486-linux] Aborted (core dumped)
On Mon, Jan 15, 2007 at 11:21:18AM -0800, Andy Delcambre wrote:> While running puppetd on a linux client I get the following, the > actual spot that the segfault happens varies, but it always happens. > > This is on ubuntu with the stock apt ruby1.8.4 package. Puppet also > installed from apt. Puppet version 0.18.4 > > It seems very likely that this isnt a bug in puppet per se, but it is > tickling a yaml bug (regardless of where it fails, it is always yaml). > But I figure someone else may have seen this.Congratulations, you''ve found the elusive Ruby Segfault bug. It turns out that the Ruby interpreter isn''t quite as robust as one might like, and has an occasional hissy-fit (as you''ve seen below) when run with large programs (like puppet). What makes the problem particularly tricky to hunt down is that it is so subtle in it''s effects that it appears to be almost non-deterministic. To give you a personal example, I had a Xen VM that was configured (by hand, long long ago) a certain way, and it was determined that we needed two of these VMs, setup identically. So I just duplicated the disk image, changed the hostname and IP address, and booted them both up. When it came time to Puppetise these two machines, one of them reliably chucks a segfault (not YAML-related, in my case, but it is always at around the same spot) and the other one doesn''t. To make sure it wasn''t the hostname/IP address causing the problem, we went and reinstalled the failing machine from scratch (but with all the same config and packages) and now Puppet runs fine on it. There have been anecdotal reports that installing the rdoc package fixes the problem for some people, but I don''t think that''s a root cause so much as just loading extra modules changes the exact locations in the process address space of some critical data structure, which masks the problem. The issue is also general enough that reports have come from users of various Linux distributions, with both distributor-built and self-built Ruby, so it''s not an Ubuntu or Debian problem. It''s quite widespread. Some people, again, have reported that building their own Ruby masks the problem, but others have tried the same thing to no effect. From all this, hopefully you''ve gotten some idea of the difficulty level of tracking this thing down -- and it''s why you''re still getting the problem after we''ve known about it for so long. If you''ve got good C programming skills and have the time to hunt this down, I''ve got some thoughts on ways you could go about trying to find the root cause of the bug, but let me warn you that the Ruby source code is... interesting, and not for the faint-hearted. - Matt -- MySQL seems to be the Windows of the database world. Broken, underspecced, and mainly only popular due to inertia and people who don''t really know what they''re doing. -- Peter Corlett, in the Monastery
On Jan 16, 2007, at 6:21 AM, Andy Delcambre wrote:> While running puppetd on a linux client I get the following, the > actual spot that the segfault happens varies, but it always happens. > > This is on ubuntu with the stock apt ruby1.8.4 package. Puppet also > installed from apt. Puppet version 0.18.4I know the latest release of Ruby on Debian has consistently inconsistent segfaults. That is, it almost always fails, but never in the same place. I think the release is the r5 release of 1.8.4. As Matt said, we''ve not been able to track this down at all; the only known workaround is to pin ruby to a previous version. See the Installation[1] page for details on how to do that. 1 - http://reductivelabs.com/projects/puppet/documentation/installing/ installation.html -- Levy''s Law: The truth is always more interesting than your preconception of what it might be. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Tue, Jan 16, 2007 at 09:15:52AM +1100, Luke Kanies wrote:> On Jan 16, 2007, at 6:21 AM, Andy Delcambre wrote: > > > While running puppetd on a linux client I get the following, the > > actual spot that the segfault happens varies, but it always happens. > > > > This is on ubuntu with the stock apt ruby1.8.4 package. Puppet also > > installed from apt. Puppet version 0.18.4 > > I know the latest release of Ruby on Debian has consistently > inconsistent segfaults. That is, it almost always fails, but never > in the same place. I think the release is the r5 release of 1.8.4. > > As Matt said, we''ve not been able to track this down at all; the only > known workaround is to pin ruby to a previous version. See the > Installation[1] page for details on how to do that. > > 1 - http://reductivelabs.com/projects/puppet/documentation/installing/ > installation.htmlThis was fixed a while ago, in 1.8.5. Christian
On Jan 16, 2007, at 9:27 AM, Christian G. Warden wrote:> > This was fixed a while ago, in 1.8.5.Really? I didn''t know. Have the other problems in 1.8.5 been fixed? It was having a lot of problems with rdoc::usage, although I think I''ve successfully worked around them. Are people successfully using 1.8.5 right now? Has Debian/testing switched to that as the ruby release? -- Morgan''s Second Law: To a first approximation all appointments are canceled. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Tue, Jan 16, 2007 at 09:50:54AM +1100, Luke Kanies wrote:> On Jan 16, 2007, at 9:27 AM, Christian G. Warden wrote: > > > > This was fixed a while ago, in 1.8.5. > > Really? I didn''t know. > > Have the other problems in 1.8.5 been fixed? It was having a lot of > problems with rdoc::usage, although I think I''ve successfully worked > around them. > > Are people successfully using 1.8.5 right now? Has Debian/testing > switched to that as the ruby release?I''ve been using it for months without problems. I only use ruby for puppet, and I''m still using 0.18.4, so I don''t know if there are any other problems. 1.8.5-4 is the current Etch (and Sid) version. Christian
On Mon, Jan 15, 2007 at 02:27:33PM -0800, Christian G. Warden wrote:> On Tue, Jan 16, 2007 at 09:15:52AM +1100, Luke Kanies wrote: > > On Jan 16, 2007, at 6:21 AM, Andy Delcambre wrote: > > > > > While running puppetd on a linux client I get the following, the > > > actual spot that the segfault happens varies, but it always happens. > > > > > > This is on ubuntu with the stock apt ruby1.8.4 package. Puppet also > > > installed from apt. Puppet version 0.18.4 > > > > I know the latest release of Ruby on Debian has consistently > > inconsistent segfaults. That is, it almost always fails, but never > > in the same place. I think the release is the r5 release of 1.8.4. > > > > As Matt said, we''ve not been able to track this down at all; the only > > known workaround is to pin ruby to a previous version. See the > > Installation[1] page for details on how to do that. > > > > 1 - http://reductivelabs.com/projects/puppet/documentation/installing/ > > installation.html > > This was fixed a while ago, in 1.8.5.Could you pin down "This" a bit? If work has been done to improve Ruby''s robustness in the face of possible segfaults, that''s fantastic. I''d be careful of claiming that 1.8.5 has solved all possible segfaults, though, in case the first person who manages to segfault 1.8.5 has just subscribed to the list. <grin> - Matt -- A polar bear is a rectangular bear after a coordinate transform.
On Tue, Jan 16, 2007 at 10:17:37AM +1100, Matthew Palmer wrote:> On Mon, Jan 15, 2007 at 02:27:33PM -0800, Christian G. Warden wrote: > > On Tue, Jan 16, 2007 at 09:15:52AM +1100, Luke Kanies wrote: > > > On Jan 16, 2007, at 6:21 AM, Andy Delcambre wrote: > > > > > > > While running puppetd on a linux client I get the following, the > > > > actual spot that the segfault happens varies, but it always happens. > > > > > > > > This is on ubuntu with the stock apt ruby1.8.4 package. Puppet also > > > > installed from apt. Puppet version 0.18.4 > > > > > > I know the latest release of Ruby on Debian has consistently > > > inconsistent segfaults. That is, it almost always fails, but never > > > in the same place. I think the release is the r5 release of 1.8.4. > > > > > > As Matt said, we''ve not been able to track this down at all; the only > > > known workaround is to pin ruby to a previous version. See the > > > Installation[1] page for details on how to do that. > > > > > > 1 - http://reductivelabs.com/projects/puppet/documentation/installing/ > > > installation.html > > > > This was fixed a while ago, in 1.8.5. > > Could you pin down "This" a bit? If work has been done to improve Ruby''s > robustness in the face of possible segfaults, that''s fantastic. I''d be > careful of claiming that 1.8.5 has solved all possible segfaults, though, in > case the first person who manages to segfault 1.8.5 has just subscribed to > the list. <grin>I''m afraid I don''t know anything about ruby internals. I''m just happy to report that I haven''t gotten segfaults running puppet for a long time. YMMV. Christian