Hello...
I recently ran across a ( IMHO ) very bad bug in puppet.
systems:
puppetmaster:
FreeBSD 6.x
puppet 0.24.4 # from ports
using servertype=mongrel
apache-2.2.8 # from ports
puppet clients
RHEL 5 64bit
puppet 0.24.4
With a file serving config like:
# get correct config for host/OS/version
# or local CustomFact
file { "sshd_config":
path => "/etc/ssh/sshd_config",
tag => sshd,
source => [
"puppet:///dist/sshd_config.$hostname",
"puppet:///dist/sshd_config.$CustomFact",
"puppet:///dist/sshd_config.$operatingsystem.$operatingsystemrelease",
"puppet:///dist/sshd_config.$operatingsystem",
"puppet:///dist/sshd_config"
],
}
I get errors like:
$DATE //Node[puppetclient]/rhel51/ssh/File[sshd_config]/source (err):
Could not describe /dist/sshd_config.puppetclient: Could not connect to
puppet.pricegrabber.com on port 8140
$DATE //Node[puppetclient]/rhel51/ssh/File[sshd_config]/source (notice):
replacing from source puppet:///dist/sshd_config.RedHat.5 with contents
{md5}ea7589d8687cae6f93eaf5ac251e6892
$DATE + next
run //Node[puppetclient]/rhel51/ssh/File[sshd_config]/source (notice):
replacing from source puppet:///dist/sshd_config.puppetclient with
contents {md5}75e7daa0be990642a257c5a2ac65c974
In the past puppet served up the file dist/sshd_config.puppetclient to
the host called puppetclient. This run puppetmaster was busy during the
request for dist/sshd_config.puppetclient (which exists) , but was
available for the next request for dist/sshd_config.RedHat.5 ( which
also exists)
two hours later the puppetmaster is less busy, so there are no timouts
and the correct dist/sshd_config.puppetclient file is re-installed.
Does my explanation make sense?
looking back through my reports , I saw this at least twice in the past
24 hours. And only on RHEL5 with puppet 0.24.4 which are the majority
of my puppet clients.
In the above case, there is no real harm as sshd needs to be restarted
for the changes to be applied, but someone else with a service
dependancy to restart on a config file change could get bit hard by this
bug.
Has anyone else seen this behavior?
This seems to be a client side issue. yes?
--
Christopher McCrory
"The guy that keeps the servers running"
To the optimist, the glass is half full.
To the pessimist, the glass is half empty.
To the engineer, the glass is twice as big as it needs to be.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---
On May 30, 2008, at 5:18 PM, Christopher McCrory wrote:> > Hello... > > I recently ran across a ( IMHO ) very bad bug in puppet.[...]> > In the above case, there is no real harm as sshd needs to be restarted > for the changes to be applied, but someone else with a service > dependancy to restart on a config file change could get bit hard by > this > bug. > > > Has anyone else seen this behavior? > > This seems to be a client side issue. yes?You''re saying that if the client can''t contact the server, it empties the file? I''ve not seen that, but I''d agree it''s a bad bug. -- "They called me mad, and I called them mad, and damn them, they outvoted me." -- Nathaniel Lee, on being consigned to a mental institution, circa 17th c. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Christopher McCrory
2008-May-30 23:19 UTC
[Puppet Users] Re: Arg! , bad bug in file serving
Hello... On Fri, 2008-05-30 at 17:28 -0500, Luke Kanies wrote:> On May 30, 2008, at 5:18 PM, Christopher McCrory wrote: > > > > > Hello... > > > > I recently ran across a ( IMHO ) very bad bug in puppet. > [...] > > > > In the above case, there is no real harm as sshd needs to be restarted > > for the changes to be applied, but someone else with a service > > dependancy to restart on a config file change could get bit hard by > > this > > bug. > > > > > > Has anyone else seen this behavior? > > > > This seems to be a client side issue. yes? > > > You''re saying that if the client can''t contact the server, it empties > the file? >It doesn''t empty the file, it uses the next file in the source [ .. ] sequence. 99% of the time when the server is too busy, all files are unavailable. the 1% is a killer though.> I''ve not seen that, but I''d agree it''s a bad bug. >-- Christopher McCrory "The guy that keeps the servers running" To the optimist, the glass is half full. To the pessimist, the glass is half empty. To the engineer, the glass is twice as big as it needs to be. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On May 30, 2008, at 6:19 PM, Christopher McCrory wrote:> It doesn''t empty the file, it uses the next file in the source [ .. ] > sequence. 99% of the time when the server is too busy, all files are > unavailable. the 1% is a killer though.Ah, I see. Hrmm, that''s a strange one. So you''re saying if there''s an exception, nothing should happen, and only if there''s a normal "no such file" should we do something? That might be tricky to do, but I guess file it and I''ll see what I can see. -- You can''t build a reputation on what you are going to do. -- Henry Ford --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Christopher McCrory
2008-May-30 23:33 UTC
[Puppet Users] Re: Arg! , bad bug in file serving
Hello... On Fri, 2008-05-30 at 18:23 -0500, Luke Kanies wrote:> On May 30, 2008, at 6:19 PM, Christopher McCrory wrote: > > > It doesn''t empty the file, it uses the next file in the source [ .. ] > > sequence. 99% of the time when the server is too busy, all files are > > unavailable. the 1% is a killer though. > > > Ah, I see. Hrmm, that''s a strange one. > > So you''re saying if there''s an exception, nothing should happen, and > only if there''s a normal "no such file" should we do something? >yea, I would think something like: pseudo code: for each possible source in sourcearray ; do get source if any failure error entire for loop else use source if success fi done NOTE: I am not a programmer :)> That might be tricky to do, but I guess file it and I''ll see what I > can see. >-- Christopher McCrory "The guy that keeps the servers running" To the optimist, the glass is half full. To the pessimist, the glass is half empty. To the engineer, the glass is twice as big as it needs to be. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Fri, May 30, 2008 at 03:18:01PM -0700, Christopher McCrory wrote:> Has anyone else seen this behavior?Hm. Over here I have a class, foo, assigned to a host (using external nodes). The foo class sets a variable, $syslog_ng_conf_suffix, and includes another class, syslog-ng, which uses the variable to select a file: class syslog-ng { file { "/etc/syslog-ng/syslog-ng.conf": ensure => file, mode => 644, owner => "root", group => "root", source => [ "puppet:///cc/ops/config/syslog-ng/syslog-ng.conf.$syslog_ng_conf_suffix", "puppet:///cc/ops/config/syslog-ng/syslog-ng.conf" ], notify => Service[''syslog-ng''], require => Package["syslog-ng"]; } ... } (Other classes include the syslog-ng class as well but they don''t set the variable, so the last entry is used in those cases.) What I''m seeing is that every once in a while the default file is used even in the case where the variable is set and points to an existing file. After a while the correct file as selected by the variable''s contents is put back again. This does seem to be related to puppetd having connection issues as a result of an unresponsive puppetmaster (need to set up more backends). -- Jos Backus jos at catnook.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On May 30, 2008, at 3:18 PM, Christopher McCrory wrote:> I recently ran across a ( IMHO ) very bad bug in puppet. > > > file { "sshd_config": > path => "/etc/ssh/sshd_config", > tag => sshd, > source => [ > "puppet:///dist/sshd_config.$hostname", > "puppet:///dist/sshd_config.$CustomFact", > "puppet:///dist/sshd_config.$operatingsystem.$operatingsystemrelease", > "puppet:///dist/sshd_config.$operatingsystem", > "puppet:///dist/sshd_config" > ], > }IMO, this is working as advertised. Multiple sources are meant to be a list of identical file sources, so that if any of them fail it has more than one place to find the file. I''m not sure why you''re using it this way, there are other ways of ensuring the correct file is always retrieved, such as using a case statement and/or selectors. -Blake --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On May 31, 2008, at 11:17 PM, Blake Barnett wrote:> IMO, this is working as advertised. Multiple sources are meant to be > a list of identical file sources, so that if any of them fail it has > more than one place to find the file. I''m not sure why you''re using > it this way, there are other ways of ensuring the correct file is > always retrieved, such as using a case statement and/or selectors.Generally, people use multiple sources as a way to provide host specificity -- i.e., look for a host-specific file, and if not found, look for a generic file. I consider this a bug, really. -- I wanna hang a map of the world in my house. Then I''m gonna put pins into all the locations that I''ve traveled to. But first, I''m gonna have to travel to the top two corners of the map so it won''t fall down. -- Mitch Hedberg --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On May 30, 2008, at 4:33 PM, Christopher McCrory wrote:> yea, I would think something like: > > pseudo code: > > for each possible source in sourcearray ; do > get source > if any failure > error entire for loop > else > use source if success > fi > doneThis should work. Currently the ''source'' parameter catches exceptions and just logs them, but it looks like the server returns '''' on missing files, so it should be safe to just raise exceptions. Anyone feel like making and testing this patch?> > NOTE: I am not a programmer :)*Today* you''re not. Tomorrow....who knows? :) -- To my embarrassment I was born in bed with a lady. --Wilson Mizner --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---