Derek
2011-Jul-06 20:33 UTC
[Puppet Users] Issue with puppet file serving api not parsing yaml content correctly
I am working on building a facter tag based node classifier similar to https://github.com/jordansissel/puppet-examples/tree/master/nodeless-puppet/. However, I have run into an issue where I cannot use puppet''s require file ability to push the yaml file containing the facts file to the client because it would require two runs of puppet to pickup changes. Consequently, I have written into the facter ruby script the ability to connect to puppet''s restful api and get the yaml file from the private store. This works fine in irb, ruby, and facter if called directly. However, when run inside of a puppet run it seems to fail on parsing the http response correctly into yaml. As a result, it does not get saved to disk and loaded as a fact for the puppet run. There is probably a simpler way to do this. Essentially we want to have tags on a server and use that to selectively include or remove modules from a server by facter tags rather than by a server''s name. Some Version Information: - os = CentOS release 5.2 (Final) - ruby = ruby 1.8.6 (2008-08-11 patchlevel 287) [x86_64-linux] - facter = 1.6.0 (updated because my script loads multiple facts and the older version we were running requires the filename to match the fact name. This was not working because I did not want to split my ruby load script into multiple files to match each of the fact names.) - puppet = 0.25.4 Yaml file it is trying to grab from a private store: --- role: - base - db env: - dev The yaml file downloads correctly via a puppet run without my script. I can also wget the file and use net/https via ruby to get the file. All methods return the correct file with matching md5sums. Under my module called "truth" I have the following: - files -> private -> domain.inter -> hostname -> truth_tags.yml ex: --- role: - base env: - dev - lib -> facter -> load_truth_tags.rb problem area: def apitruthtag(calltype) # set some client side variables to build on later sslbasedir = ''/etc/puppet/ssl'' sslprivdir = sslbasedir + ''/private_keys'' sslpubdir = sslbasedir + ''/certs'' sslcafile = sslpubdir + ''/ca.pem'' # this sets if we want metadata or content from puppet datatype = calltype # We want yaml back from puppet header = {''Accept'' => ''yaml''} # Setup some connection variables to our puppet server and what we want from it proto = ''https'' server = ''puppet.domain.inter'' port = ''8140'' path = ''/production/file_'' + datatype + ''/truth_private/ truth_tags.yml'' # Build the full uri to request from our puppet server. Then parse it for port and things uri = URI.parse(proto + ''://'' + server + '':'' + port + path) # Setup the http module and set it for getting data http = Net::HTTP.new(uri.host, uri.port) request = Net::HTTP::Get.new(uri.request_uri, header) http.use_ssl = true if uri.scheme == ''https'' # Enable ssl verification to ensure we are talking to the correct people http.verify_mode = OpenSSL::SSL::VERIFY_PEER # Cert Auth: # Set certificate paths # puppet certificate authority file if File.readable?(sslcafile) then # Puppet ca file http.ca_file = sslcafile puts "readable? " + sslprivdir + ''/'' + hostname + ''.pem'' if $debug if File.readable?(sslprivdir + ''/'' + hostname + ''.pem'') then # client private key http.key = OpenSSL::PKey::RSA.new(File.read(sslprivdir + ''/'' + hostname + ''.pem'')) puts "readable? " + sslpubdir + ''/'' + hostname + ''.pem'' if $debug if File.readable?(sslpubdir + ''/'' + hostname + ''.pem'') then # client public key http.cert = OpenSSL::X509::Certificate.new(File.read(sslpubdir + ''/'' + hostname + ''.pem'')) # Make the request response = http.request(request) else raise "No readable client pubic key in #{sslpubdir}/ #{hostname}.pem" end # End public key check else raise "No readable client private key in #{sslprivdir}/ #{hostname}.pem" end # End private key check else raise "No readable ca cert in #{sslcafile}" end # End ca file check # Check to make sure we got some data back if response != nil # Check to see if we have a good server response before saving the variable puts "check code " + response.code if $debug if ((response.code < "300") and (response.code >= "200")) return response.body else raise "server did not return an acceptable reponse code" end # end server response code check else raise "No response from #{server}" end # end nil response check end # end apitruthtag servermd5 = YAML.load(apitruthtag("metadata")).ivars["checksum"] # When executed from a puppet run I tells me that ivars is undefined. - lib -> puppet -> parser -> functions -> truth_tags.rb - manifests -> init.pp ex: class truth inherits truth::init_bootstrap { if truth_tag(''role'', ''base'') and !truth_tag(''role'', ''nobase'') { notice("${::hostname}: Including role, base modules...") notice(" ${::hostname}: role, base: including network") include network } } - manifests -> init_bootstrap.pp just makes sure the /etc/truth_tags.yml file exists on disk Process (if the facter yaml load was working): 1. puppet client downloads "lib -> facter -> load_truth_tags.rb". 2. facter runs with the external ruby fact script. - this fails on puppet runs but functions correctly directly in ruby, irb, and facter itself using "facter -d" - If working load_truth_tags.rb would do this: 1. See if a cached /etc/truth_tags.yml file exists 2. if it does exist then it md5 hashes the file. Next, it uses net/ https to connect to the resful api to get the puppetmaster''s hash to see if the local file has changed. If they differ it pulls down the changed file from puppetmaster and writes it to disk. 3. if the file does not exist on disk it pulls it down from puppetmaster using net/https and checks both the server and client hashes to see if it was modified in transit and that it downloaded the correct content. 3. Next, manifests -> init.pp loads the truth_tags function from lib -> puppet -> parser -> functions -> truth_tags.rb for the puppet run.- the manifest init.pp file is basically a set of rules that says if role = db then include these other functions. The goal being we don''t want to classify servers by name but rather by their functions. This would allow us to be more flexible and not have to worry about double including things in the nodes.pp file. Sorry I have not included all of the "load_truth_tags.rb" script as it is long (317 lines). I can if requested. Issue I can see which differs from a puppet run vs stand alone ruby run: irb, ruby, or facter: yaml parsed http response = #<YAML::Object:0x2ada01f7cf00> puppet run: yaml parsed http response = #<Puppet::FileServing::Metadata: 0x2ac7987b9c08> with error: undefined method `ivars'' for #<Puppet::FileServing::Metadata: 0x2ac7987152c0> #<NoMethodError: undefined method `ivars'' for #<Puppet::FileServing::Metadata:0x2ac7987152c0>> and traceback: /var/puppet/lib/facter/load_truth_tags.rb:253:in `calcservertruthmd5'' /var/puppet/lib/facter/load_truth_tags.rb:268 /opt/ruby-1.8.6-p287/lib/ruby/1.8/timeout.rb:62:in `timeout'' /var/puppet/lib/facter/load_truth_tags.rb:37 /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/facter/util/loader.rb: 73:in `load'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/facter/util/loader.rb: 73:in `load_file'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/facter/util/loader.rb: 38:in `load_all'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/facter/util/loader.rb: 33:in `each'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/facter/util/loader.rb: 33:in `load_all'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/facter/util/loader.rb: 30:in `each'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/facter/util/loader.rb: 30:in `load_all'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/facter/util/collection.rb: 94:in `load_all'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/facter.rb:218:in `loadfacts'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/configurer/ fact_handler.rb:61:in `reload_facter'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/configurer/ fact_handler.rb:18:in `find_facts'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/configurer/ fact_handler.rb:29:in `facts_for_uploading'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/configurer.rb: 100:in `retrieve_catalog'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/configurer.rb: 162:in `run'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/agent.rb:53:in `run'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb: 21:in `lock'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/agent.rb:53:in `run'' /opt/ruby-1.8.6-p287/lib/ruby/1.8/sync.rb:229:in `synchronize'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/agent.rb:53:in `run'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/agent.rb:134:in `with_client'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/agent.rb:51:in `run'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/application/ puppetd.rb:103:in `onetime'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/application.rb: 226:in `send'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/application.rb: 226:in `run_command'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/application.rb: 217:in `run'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/application.rb: 306:in `exit_on_fail'' /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/application.rb: 217:in `run'' /usr/sbin/puppetd:159 It appears that yaml is not parsing correctly when run inside a puppet run. As a result, the response.body.ivars["checksum"] variable is not being set. Any ideas? -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Nigel Kersten
2011-Jul-09 21:38 UTC
Re: [Puppet Users] Issue with puppet file serving api not parsing yaml content correctly
On Wed, Jul 6, 2011 at 1:33 PM, Derek <dtamsen@gmail.com> wrote:> I am working on building a facter tag based node classifier similar to > > https://github.com/jordansissel/puppet-examples/tree/master/nodeless-puppet/ > . > However, I have run into an issue where I cannot use puppet''s require > file ability to push the yaml file containing the facts file to the > client because it would require two runs of puppet to pickup changes. > Consequently, I have written into the facter ruby script the ability > to connect to puppet''s restful api and get the yaml file from the > private store. This works fine in irb, ruby, and facter if called > directly. However, when run inside of a puppet run it seems to fail on > parsing the http response correctly into yaml. As a result, it does > not get saved to disk and loaded as a fact for the puppet run. > > There is probably a simpler way to do this. Essentially we want to > have tags on a server and use that to selectively include or remove > modules from a server by facter tags rather than by a server''s name. > > Some Version Information: > - os = CentOS release 5.2 (Final) > - ruby = ruby 1.8.6 (2008-08-11 patchlevel 287) [x86_64-linux] > - facter = 1.6.0 (updated because my script loads multiple facts and > the older version we were running requires the filename to match the > fact name. This was not working because I did not want to split my > ruby load script into multiple files to match each of the fact names.) > - puppet = 0.25.4 > > Yaml file it is trying to grab from a private store: > --- > role: > - base > - db > env: > - dev > > The yaml file downloads correctly via a puppet run without my script. > I can also wget the file and use net/https via ruby to get the file. > All methods return the correct file with matching md5sums. > > Under my module called "truth" I have the following: > - files -> private -> domain.inter -> hostname -> truth_tags.yml > ex: > --- > role: > - base > env: > - dev > > - lib -> facter -> load_truth_tags.rb > problem area: > def apitruthtag(calltype) > > # set some client side variables to build on later > sslbasedir = ''/etc/puppet/ssl'' > sslprivdir = sslbasedir + ''/private_keys'' > sslpubdir = sslbasedir + ''/certs'' > sslcafile = sslpubdir + ''/ca.pem'' > > # this sets if we want metadata or content from puppet > datatype = calltype > > # We want yaml back from puppet > header = {''Accept'' => ''yaml''} > > # Setup some connection variables to our puppet server and what we > want from it > proto = ''https'' > server = ''puppet.domain.inter'' > port = ''8140'' > path = ''/production/file_'' + datatype + ''/truth_private/ > truth_tags.yml'' > > # Build the full uri to request from our puppet server. Then parse > it for port and things > uri = URI.parse(proto + ''://'' + server + '':'' + port + path) > > # Setup the http module and set it for getting data > http = Net::HTTP.new(uri.host, uri.port) > request = Net::HTTP::Get.new(uri.request_uri, header) > > http.use_ssl = true if uri.scheme == ''https'' > > # Enable ssl verification to ensure we are talking to the correct > people > http.verify_mode = OpenSSL::SSL::VERIFY_PEER > > # Cert Auth: > # Set certificate paths > # puppet certificate authority file > > if File.readable?(sslcafile) then > # Puppet ca file > http.ca_file = sslcafile > puts "readable? " + sslprivdir + ''/'' + hostname + ''.pem'' if $debug > if File.readable?(sslprivdir + ''/'' + hostname + ''.pem'') then > # client private key > http.key = OpenSSL::PKey::RSA.new(File.read(sslprivdir + ''/'' + > hostname + ''.pem'')) > puts "readable? " + sslpubdir + ''/'' + hostname + ''.pem'' if > $debug > if File.readable?(sslpubdir + ''/'' + hostname + ''.pem'') then > # client public key > http.cert = OpenSSL::X509::Certificate.new(File.read(sslpubdir > + ''/'' + hostname + ''.pem'')) > > # Make the request > response = http.request(request) > else > raise "No readable client pubic key in #{sslpubdir}/ > #{hostname}.pem" > end # End public key check > else > raise "No readable client private key in #{sslprivdir}/ > #{hostname}.pem" > end # End private key check > else > raise "No readable ca cert in #{sslcafile}" > end # End ca file check > > # Check to make sure we got some data back > if response != nil > # Check to see if we have a good server response before saving the > variable > puts "check code " + response.code if $debug > if ((response.code < "300") and (response.code >= "200")) > return response.body > else > raise "server did not return an acceptable reponse code" > end # end server response code check > else > raise "No response from #{server}" > end # end nil response check > > end # end apitruthtag > > servermd5 = YAML.load(apitruthtag("metadata")).ivars["checksum"] # > When executed from a puppet run I tells me that ivars is undefined. > > - lib -> puppet -> parser -> functions -> truth_tags.rb > - manifests -> init.pp > ex: > class truth inherits truth::init_bootstrap { > if truth_tag(''role'', ''base'') and !truth_tag(''role'', ''nobase'') { > notice("${::hostname}: Including role, base modules...") > > notice(" ${::hostname}: role, base: including network") > include network > } > } > - manifests -> init_bootstrap.pp > just makes sure the /etc/truth_tags.yml file exists on disk > > Process (if the facter yaml load was working): > 1. puppet client downloads "lib -> facter -> load_truth_tags.rb". > 2. facter runs with the external ruby fact script. > - this fails on puppet runs but functions correctly directly in > ruby, irb, and facter itself using "facter -d" > - If working load_truth_tags.rb would do this: > 1. See if a cached /etc/truth_tags.yml file exists > 2. if it does exist then it md5 hashes the file. Next, it uses net/ > https to connect to the resful api to get the puppetmaster''s hash to > see if the local file has changed. If they differ it pulls down the > changed file from puppetmaster and writes it to disk. > 3. if the file does not exist on disk it pulls it down from > puppetmaster using net/https and checks both the server and client > hashes to see if it was modified in transit and that it downloaded the > correct content. > 3. Next, manifests -> init.pp loads the truth_tags function from lib - > > puppet -> parser -> functions -> truth_tags.rb for the puppet run. > - the manifest init.pp file is basically a set of rules that says if > role = db then include these other functions. The goal being we don''t > want to classify servers by name but rather by their functions. This > would allow us to be more flexible and not have to worry about double > including things in the nodes.pp file. > > > > Sorry I have not included all of the "load_truth_tags.rb" script as it > is long (317 lines). I can if requested. > > Issue I can see which differs from a puppet run vs stand alone ruby > run: > > irb, ruby, or facter: > yaml parsed http response = #<YAML::Object:0x2ada01f7cf00> > > puppet run: > yaml parsed http response = #<Puppet::FileServing::Metadata: > 0x2ac7987b9c08> > with error: > > undefined method `ivars'' for #<Puppet::FileServing::Metadata: > 0x2ac7987152c0> > #<NoMethodError: undefined method `ivars'' for > #<Puppet::FileServing::Metadata:0x2ac7987152c0>> > > and traceback: > > /var/puppet/lib/facter/load_truth_tags.rb:253:in `calcservertruthmd5'' > /var/puppet/lib/facter/load_truth_tags.rb:268 > /opt/ruby-1.8.6-p287/lib/ruby/1.8/timeout.rb:62:in `timeout'' > /var/puppet/lib/facter/load_truth_tags.rb:37 > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/facter/util/loader.rb: > 73:in `load'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/facter/util/loader.rb: > 73:in `load_file'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/facter/util/loader.rb: > 38:in `load_all'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/facter/util/loader.rb: > 33:in `each'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/facter/util/loader.rb: > 33:in `load_all'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/facter/util/loader.rb: > 30:in `each'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/facter/util/loader.rb: > 30:in `load_all'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/facter/util/collection.rb: > 94:in `load_all'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/facter.rb:218:in > `loadfacts'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/configurer/ > fact_handler.rb:61:in `reload_facter'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/configurer/ > fact_handler.rb:18:in `find_facts'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/configurer/ > fact_handler.rb:29:in `facts_for_uploading'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/configurer.rb: > 100:in `retrieve_catalog'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/configurer.rb: > 162:in `run'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/agent.rb:53:in > `run'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb: > 21:in `lock'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/agent.rb:53:in > `run'' > /opt/ruby-1.8.6-p287/lib/ruby/1.8/sync.rb:229:in `synchronize'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/agent.rb:53:in > `run'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/agent.rb:134:in > `with_client'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/agent.rb:51:in > `run'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/application/ > puppetd.rb:103:in `onetime'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/application.rb: > 226:in `send'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/application.rb: > 226:in `run_command'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/application.rb: > 217:in `run'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/application.rb: > 306:in `exit_on_fail'' > /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/application.rb: > 217:in `run'' > /usr/sbin/puppetd:159 > > > > > > > > > It appears that yaml is not parsing correctly when run inside a puppet > run. As a result, the response.body.ivars["checksum"] variable is not > being set. Any ideas? > >So this kind of rare, but you''ve almost provided too much info here :) and I''m finding it difficult to work out what your actual problem is. To back up a little... "There is probably a simpler way to do this. Essentially we want to have tags on a server and use that to selectively include or remove modules from a server by facter tags rather than by a server''s name." If you''re looking everything up by hostname anyway, why not just have an external node classifier that does all this logic server-side? -- Nigel Kersten Product Manager, Puppet Labs Twitter: @nigelkersten *Join us for **PuppetConf *<http://www.bit.ly/puppetconfsig> September 22nd and 23rd in Portland, Oregon, USA. * * -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Ken Barber
2011-Jul-09 22:03 UTC
Re: [Puppet Users] Issue with puppet file serving api not parsing yaml content correctly
So ...> servermd5 = YAML.load(apitruthtag("metadata")).ivars["checksum"] # > When executed from a puppet run I tells me that ivars is undefined.What does the output of apitruthtag("metadata") show you between each run in facter, irb and puppet? Can you output each to a file and analyze the difference?> irb, ruby, or facter: > yaml parsed http response = #<YAML::Object:0x2ada01f7cf00> > > puppet run: > yaml parsed http response = #<Puppet::FileServing::Metadata: > 0x2ac7987b9c08> > with error: > > undefined method `ivars'' for #<Puppet::FileServing::Metadata: > 0x2ac7987152c0> > #<NoMethodError: undefined method `ivars'' for > #<Puppet::FileServing::Metadata:0x2ac7987152c0>>So when you run it with all the Puppet libraries the YAML object is being serialized back into a Ruby object Puppet::FileServing::Metadata ... I''m really curious what the raw YAML output looks like. This should only happen I think if the YAML output has meta information that matches that class type for example: --- !ruby/object:Puppet::FileServing::Metadata {} I get the feeling the YAML you are getting back isn''t what you were expecting and looking at the contents of the raw output might give a better clue as to why. ken. -- "Join us for PuppetConf, September 22nd and 23rd in Portland, OR: http://bit.ly/puppetconfsig" -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Derek
2011-Jul-11 18:36 UTC
[Puppet Users] Re: Issue with puppet file serving api not parsing yaml content correctly
Thanks for everyone''s help. I figured out what the issue was over the weekend. It appears that when the ruby script is run in a puppet run, puppet loads the yaml dictionaries specific to puppet. However, when I was testing the script in ruby I was not loading the yaml dictionaries for puppet. This was causing my script to get a generic loaded yaml of #<YAML::Object:0x2ada01f7cf00> instead of the correctly loaded yaml #<Puppet::FileServing::Metadata:0x2ac7987b9c08>. When the dictionaries are loaded for puppet it no longer has an "ivars" hash with content or metadata. I just needed to do "YAML.load(apitruthtag("content")).content" instead of "YAML.load(apitruthtag("content")).ivars["content"]. However, because I am still not loading the dictionary outside of the puppet run my script will not function correctly if just executed in ruby. Does anybody know how to properly load the puppet yaml dictionary outside of a puppet run? FYI: The raw yaml data from puppet is: --- !ruby/object:Puppet::FileServing::Content content: | --- role: - base env: - dev expiration: 2011-07-11 18:57:08.413941 +00:00 links: :manage path: /etc/puppet/modules/truth/files/private/domain.inter/server01/ truth_tags.yml stat_method: :lstat -- Derek On Jul 9, 3:03 pm, Ken Barber <k...@puppetlabs.com> wrote:> So ... > > > servermd5 = YAML.load(apitruthtag("metadata")).ivars["checksum"] # > > When executed from a puppet run I tells me that ivars is undefined. > > What does the output of apitruthtag("metadata") show you between each > run in facter, irb and puppet? Can you output each to a file and > analyze the difference? > > > irb, ruby, or facter: > > yaml parsed http response = #<YAML::Object:0x2ada01f7cf00> > > > puppet run: > > yaml parsed http response = #<Puppet::FileServing::Metadata: > > 0x2ac7987b9c08> > > with error: > > > undefined method `ivars'' for #<Puppet::FileServing::Metadata: > > 0x2ac7987152c0> > > #<NoMethodError: undefined method `ivars'' for > > #<Puppet::FileServing::Metadata:0x2ac7987152c0>> > > So when you run it with all the Puppet libraries the YAML object is > being serialized back into a Ruby object Puppet::FileServing::Metadata > ... I''m really curious what the raw YAML output looks like. This > should only happen I think if the YAML output has meta information > that matches that class type for example: > > --- > !ruby/object:Puppet::FileServing::Metadata {} > > I get the feeling the YAML you are getting back isn''t what you were > expecting and looking at the contents of the raw output might give a > better clue as to why. > > ken. > > -- > "Join us for PuppetConf, September 22nd and 23rd in Portland, OR:http://bit.ly/puppetconfsig"-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Nigel Kersten
2011-Jul-11 21:56 UTC
Re: [Puppet Users] Re: Issue with puppet file serving api not parsing yaml content correctly
On Mon, Jul 11, 2011 at 11:36 AM, Derek <dtamsen@gmail.com> wrote:> Thanks for everyone''s help. I figured out what the issue was over the > weekend. It appears that when the ruby script is run in a puppet run, > puppet loads the yaml dictionaries specific to puppet. However, when I > was testing the script in ruby I was not loading the yaml dictionaries > for puppet. This was causing my script to get a generic loaded yaml of > #<YAML::Object:0x2ada01f7cf00> instead of the correctly loaded yaml > #<Puppet::FileServing::Metadata:0x2ac7987b9c08>. When the dictionaries > are loaded for puppet it no longer has an "ivars" hash with content or > metadata. I just needed to do > "YAML.load(apitruthtag("content")).content" instead of > "YAML.load(apitruthtag("content")).ivars["content"]. > > However, because I am still not loading the dictionary outside of the > puppet run my script will not function correctly if just executed in > ruby. Does anybody know how to properly load the puppet yaml > dictionary outside of a puppet run? >You should be able to load it as require ''puppet/util/zaml'' and use ZAML instead of YAML. That''s the version of YAML we''re vendoring to fix a whole list of bugs we ran into. There''s more complication going on in: lib/puppet/util/monkey_patches.rb showing how we monkey patch the various yaml methods on objects to actually talk ZAML instead.> > FYI: > The raw yaml data from puppet is: > > --- !ruby/object:Puppet::FileServing::Content > content: | > --- > role: > - base > env: > - dev > > expiration: 2011-07-11 18:57:08.413941 +00:00 > links: :manage > path: /etc/puppet/modules/truth/files/private/domain.inter/server01/ > truth_tags.yml > stat_method: :lstat > > -- > Derek > > > On Jul 9, 3:03 pm, Ken Barber <k...@puppetlabs.com> wrote: > > So ... > > > > > servermd5 = YAML.load(apitruthtag("metadata")).ivars["checksum"] # > > > When executed from a puppet run I tells me that ivars is undefined. > > > > What does the output of apitruthtag("metadata") show you between each > > run in facter, irb and puppet? Can you output each to a file and > > analyze the difference? > > > > > irb, ruby, or facter: > > > yaml parsed http response = #<YAML::Object:0x2ada01f7cf00> > > > > > puppet run: > > > yaml parsed http response = #<Puppet::FileServing::Metadata: > > > 0x2ac7987b9c08> > > > with error: > > > > > undefined method `ivars'' for #<Puppet::FileServing::Metadata: > > > 0x2ac7987152c0> > > > #<NoMethodError: undefined method `ivars'' for > > > #<Puppet::FileServing::Metadata:0x2ac7987152c0>> > > > > So when you run it with all the Puppet libraries the YAML object is > > being serialized back into a Ruby object Puppet::FileServing::Metadata > > ... I''m really curious what the raw YAML output looks like. This > > should only happen I think if the YAML output has meta information > > that matches that class type for example: > > > > --- > > !ruby/object:Puppet::FileServing::Metadata {} > > > > I get the feeling the YAML you are getting back isn''t what you were > > expecting and looking at the contents of the raw output might give a > > better clue as to why. > > > > ken. > > > > -- > > "Join us for PuppetConf, September 22nd and 23rd in Portland, OR: > http://bit.ly/puppetconfsig" > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- Nigel Kersten Product Manager, Puppet Labs Twitter: @nigelkersten *Join us for **PuppetConf *<http://www.bit.ly/puppetconfsig> September 22nd and 23rd in Portland, Oregon, USA. * * -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Derek Tamsen
2011-Jul-11 23:17 UTC
[Puppet Users] Re: Issue with puppet file serving api not parsing yaml content correctly
Thanks. Unfortunately, it seems I will need to wait until we upgrade puppet as version 0.25.4 does not seem to have ''puppet/util/zaml''. -- Derek On Jul 11, 2:56 pm, Nigel Kersten <ni...@puppetlabs.com> wrote:> On Mon, Jul 11, 2011 at 11:36 AM, Derek <dtam...@gmail.com> wrote: > > Thanks for everyone''s help. I figured out what the issue was over the > > weekend. It appears that when the ruby script is run in a puppet run, > > puppet loads the yaml dictionaries specific to puppet. However, when I > > was testing the script in ruby I was not loading the yaml dictionaries > > for puppet. This was causing my script to get a generic loaded yaml of > > #<YAML::Object:0x2ada01f7cf00> instead of the correctly loaded yaml > > #<Puppet::FileServing::Metadata:0x2ac7987b9c08>. When the dictionaries > > are loaded for puppet it no longer has an "ivars" hash with content or > > metadata. I just needed to do > > "YAML.load(apitruthtag("content")).content" instead of > > "YAML.load(apitruthtag("content")).ivars["content"]. > > > However, because I am still not loading the dictionary outside of the > > puppet run my script will not function correctly if just executed in > > ruby. Does anybody know how to properly load the puppet yaml > > dictionary outside of a puppet run? > > You should be able to load it as > > require ''puppet/util/zaml'' > > and use ZAML instead of YAML. That''s the version of YAML we''re vendoring to > fix a whole list of bugs we ran into. > > There''s more complication going on in: > > lib/puppet/util/monkey_patches.rb > > showing how we monkey patch the various yaml methods on objects to actually > talk ZAML instead. > > > > > > > > > > > > > FYI: > > The raw yaml data from puppet is: > > > --- !ruby/object:Puppet::FileServing::Content > > content: | > > --- > > role: > > - base > > env: > > - dev > > > expiration: 2011-07-11 18:57:08.413941 +00:00 > > links: :manage > > path: /etc/puppet/modules/truth/files/private/domain.inter/server01/ > > truth_tags.yml > > stat_method: :lstat > > > -- > > Derek > > > On Jul 9, 3:03 pm, Ken Barber <k...@puppetlabs.com> wrote: > > > So ... > > > > > servermd5 = YAML.load(apitruthtag("metadata")).ivars["checksum"] # > > > > When executed from a puppet run I tells me that ivars is undefined. > > > > What does the output of apitruthtag("metadata") show you between each > > > run in facter, irb and puppet? Can you output each to a file and > > > analyze the difference? > > > > > irb, ruby, or facter: > > > > yaml parsed http response = #<YAML::Object:0x2ada01f7cf00> > > > > > puppet run: > > > > yaml parsed http response = #<Puppet::FileServing::Metadata: > > > > 0x2ac7987b9c08> > > > > with error: > > > > > undefined method `ivars'' for #<Puppet::FileServing::Metadata: > > > > 0x2ac7987152c0> > > > > #<NoMethodError: undefined method `ivars'' for > > > > #<Puppet::FileServing::Metadata:0x2ac7987152c0>> > > > > So when you run it with all the Puppet libraries the YAML object is > > > being serialized back into a Ruby object Puppet::FileServing::Metadata > > > ... I''m really curious what the raw YAML output looks like. This > > > should only happen I think if the YAML output has meta information > > > that matches that class type for example: > > > > --- > > > !ruby/object:Puppet::FileServing::Metadata {} > > > > I get the feeling the YAML you are getting back isn''t what you were > > > expecting and looking at the contents of the raw output might give a > > > better clue as to why. > > > > ken. > > > > -- > > > "Join us for PuppetConf, September 22nd and 23rd in Portland, OR: > >http://bit.ly/puppetconfsig" > > > -- > > You received this message because you are subscribed to the Google Groups > > "Puppet Users" group. > > To post to this group, send email to puppet-users@googlegroups.com. > > To unsubscribe from this group, send email to > > puppet-users+unsubscribe@googlegroups.com. > > For more options, visit this group at > >http://groups.google.com/group/puppet-users?hl=en. > > -- > Nigel Kersten > Product Manager, Puppet Labs > Twitter: @nigelkersten > > *Join us for **PuppetConf *<http://www.bit.ly/puppetconfsig> > September 22nd and 23rd in Portland, Oregon, USA. > * > *-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Nigel Kersten
2011-Jul-11 23:51 UTC
Re: [Puppet Users] Re: Issue with puppet file serving api not parsing yaml content correctly
On Mon, Jul 11, 2011 at 4:17 PM, Derek Tamsen <dtamsen@gmail.com> wrote:> Thanks. Unfortunately, it seems I will need to wait until we upgrade > puppet as version 0.25.4 does not seem to have ''puppet/util/zaml''. >Then I''m more confused :) does require ''puppet'' require ''yaml'' just do the right thing? If not, I''d post to the puppet-dev list and see if the more developer-focused crowd can give you a better answer for 0.25.4.> > -- > Derek > > On Jul 11, 2:56 pm, Nigel Kersten <ni...@puppetlabs.com> wrote: > > On Mon, Jul 11, 2011 at 11:36 AM, Derek <dtam...@gmail.com> wrote: > > > Thanks for everyone''s help. I figured out what the issue was over the > > > weekend. It appears that when the ruby script is run in a puppet run, > > > puppet loads the yaml dictionaries specific to puppet. However, when I > > > was testing the script in ruby I was not loading the yaml dictionaries > > > for puppet. This was causing my script to get a generic loaded yaml of > > > #<YAML::Object:0x2ada01f7cf00> instead of the correctly loaded yaml > > > #<Puppet::FileServing::Metadata:0x2ac7987b9c08>. When the dictionaries > > > are loaded for puppet it no longer has an "ivars" hash with content or > > > metadata. I just needed to do > > > "YAML.load(apitruthtag("content")).content" instead of > > > "YAML.load(apitruthtag("content")).ivars["content"]. > > > > > However, because I am still not loading the dictionary outside of the > > > puppet run my script will not function correctly if just executed in > > > ruby. Does anybody know how to properly load the puppet yaml > > > dictionary outside of a puppet run? > > > > You should be able to load it as > > > > require ''puppet/util/zaml'' > > > > and use ZAML instead of YAML. That''s the version of YAML we''re vendoring > to > > fix a whole list of bugs we ran into. > > > > There''s more complication going on in: > > > > lib/puppet/util/monkey_patches.rb > > > > showing how we monkey patch the various yaml methods on objects to > actually > > talk ZAML instead. > > > > > > > > > > > > > > > > > > > > > > > > > FYI: > > > The raw yaml data from puppet is: > > > > > --- !ruby/object:Puppet::FileServing::Content > > > content: | > > > --- > > > role: > > > - base > > > env: > > > - dev > > > > > expiration: 2011-07-11 18:57:08.413941 +00:00 > > > links: :manage > > > path: /etc/puppet/modules/truth/files/private/domain.inter/server01/ > > > truth_tags.yml > > > stat_method: :lstat > > > > > -- > > > Derek > > > > > On Jul 9, 3:03 pm, Ken Barber <k...@puppetlabs.com> wrote: > > > > So ... > > > > > > > servermd5 = YAML.load(apitruthtag("metadata")).ivars["checksum"] # > > > > > When executed from a puppet run I tells me that ivars is undefined. > > > > > > What does the output of apitruthtag("metadata") show you between each > > > > run in facter, irb and puppet? Can you output each to a file and > > > > analyze the difference? > > > > > > > irb, ruby, or facter: > > > > > yaml parsed http response = #<YAML::Object:0x2ada01f7cf00> > > > > > > > puppet run: > > > > > yaml parsed http response = #<Puppet::FileServing::Metadata: > > > > > 0x2ac7987b9c08> > > > > > with error: > > > > > > > undefined method `ivars'' for #<Puppet::FileServing::Metadata: > > > > > 0x2ac7987152c0> > > > > > #<NoMethodError: undefined method `ivars'' for > > > > > #<Puppet::FileServing::Metadata:0x2ac7987152c0>> > > > > > > So when you run it with all the Puppet libraries the YAML object is > > > > being serialized back into a Ruby object > Puppet::FileServing::Metadata > > > > ... I''m really curious what the raw YAML output looks like. This > > > > should only happen I think if the YAML output has meta information > > > > that matches that class type for example: > > > > > > --- > > > > !ruby/object:Puppet::FileServing::Metadata {} > > > > > > I get the feeling the YAML you are getting back isn''t what you were > > > > expecting and looking at the contents of the raw output might give a > > > > better clue as to why. > > > > > > ken. > > > > > > -- > > > > "Join us for PuppetConf, September 22nd and 23rd in Portland, OR: > > >http://bit.ly/puppetconfsig" > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "Puppet Users" group. > > > To post to this group, send email to puppet-users@googlegroups.com. > > > To unsubscribe from this group, send email to > > > puppet-users+unsubscribe@googlegroups.com. > > > For more options, visit this group at > > >http://groups.google.com/group/puppet-users?hl=en. > > > > -- > > Nigel Kersten > > Product Manager, Puppet Labs > > Twitter: @nigelkersten > > > > *Join us for **PuppetConf *<http://www.bit.ly/puppetconfsig> > > September 22nd and 23rd in Portland, Oregon, USA. > > * > > * > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- Nigel Kersten Product Manager, Puppet Labs Twitter: @nigelkersten *Join us for **PuppetConf *<http://www.bit.ly/puppetconfsig> September 22nd and 23rd in Portland, Oregon, USA. * * -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Derek Tamsen
2011-Jul-12 16:54 UTC
[Puppet Users] Re: Issue with puppet file serving api not parsing yaml content correctly
Unfortunately, it does not work. That was the first thing I had tried to try and reproduce the problem in ruby outside a puppet run. Also, here are the ruby util modules for puppet in 0.25.4: /opt/ruby-1.8.6-p287/lib/ruby/site_ruby/1.8/puppet/util autoload autoload.rb backups.rb cacher.rb checksums.rb classgen.rb config_store.rb constant_inflector.rb diff.rb docs.rb errors.rb execution.rb feature.rb file_locking.rb fileparsing.rb filetype.rb graph.rb inifile.rb instance_loader.rb ldap ldap.rb loadedfile.rb logging.rb log_paths.rb log.rb metaid.rb methodhelper.rb metric.rb monkey_patches.rb nagios_maker.rb package.rb pidlock.rb posix.rb provider_features.rb pson.rb queue queue.rb rails rdoc rdoc.rb reference.rb resource_template.rb selinux.rb settings settings.rb storage.rb subclass_loader.rb suidmanager.rb tagging.rb user_attr.rb warnings.rb Also, I did not see anything in the monkey_patches.rb file pertaining to a yaml like patch only rdoc.: module RDoc def self.caller(skip=nil) in_gem_wrapper = false Kernel.caller.reject { |call| in_gem_wrapper ||= call =~ /#{Regexp.escape $0}:\d+:in `load''/ } end end I think at this point we will probably upgrade puppet anyways so I will just add "require puppet/util/yaml" once we have upgraded. -- Derek On Jul 11, 4:51 pm, Nigel Kersten <ni...@puppetlabs.com> wrote:> On Mon, Jul 11, 2011 at 4:17 PM, Derek Tamsen <dtam...@gmail.com> wrote: > > Thanks. Unfortunately, it seems I will need to wait until we upgrade > > puppet as version 0.25.4 does not seem to have ''puppet/util/zaml''. > > Then I''m more confused :) > > does > > require ''puppet'' > require ''yaml'' > > just do the right thing? If not, I''d post to the puppet-dev list and see if > the more developer-focused crowd can give you a better answer for 0.25.4. > > > > > > > > > > > > > -- > > Derek > > > On Jul 11, 2:56 pm, Nigel Kersten <ni...@puppetlabs.com> wrote: > > > On Mon, Jul 11, 2011 at 11:36 AM, Derek <dtam...@gmail.com> wrote: > > > > Thanks for everyone''s help. I figured out what the issue was over the > > > > weekend. It appears that when the ruby script is run in a puppet run, > > > > puppet loads the yaml dictionaries specific to puppet. However, when I > > > > was testing the script in ruby I was not loading the yaml dictionaries > > > > for puppet. This was causing my script to get a generic loaded yaml of > > > > #<YAML::Object:0x2ada01f7cf00> instead of the correctly loaded yaml > > > > #<Puppet::FileServing::Metadata:0x2ac7987b9c08>. When the dictionaries > > > > are loaded for puppet it no longer has an "ivars" hash with content or > > > > metadata. I just needed to do > > > > "YAML.load(apitruthtag("content")).content" instead of > > > > "YAML.load(apitruthtag("content")).ivars["content"]. > > > > > However, because I am still not loading the dictionary outside of the > > > > puppet run my script will not function correctly if just executed in > > > > ruby. Does anybody know how to properly load the puppet yaml > > > > dictionary outside of a puppet run? > > > > You should be able to load it as > > > > require ''puppet/util/zaml'' > > > > and use ZAML instead of YAML. That''s the version of YAML we''re vendoring > > to > > > fix a whole list of bugs we ran into. > > > > There''s more complication going on in: > > > > lib/puppet/util/monkey_patches.rb > > > > showing how we monkey patch the various yaml methods on objects to > > actually > > > talk ZAML instead. > > > > > FYI: > > > > The raw yaml data from puppet is: > > > > > --- !ruby/object:Puppet::FileServing::Content > > > > content: | > > > > --- > > > > role: > > > > - base > > > > env: > > > > - dev > > > > > expiration: 2011-07-11 18:57:08.413941 +00:00 > > > > links: :manage > > > > path: /etc/puppet/modules/truth/files/private/domain.inter/server01/ > > > > truth_tags.yml > > > > stat_method: :lstat > > > > > -- > > > > Derek > > > > > On Jul 9, 3:03 pm, Ken Barber <k...@puppetlabs.com> wrote: > > > > > So ... > > > > > > > servermd5 = YAML.load(apitruthtag("metadata")).ivars["checksum"] # > > > > > > When executed from a puppet run I tells me that ivars is undefined. > > > > > > What does the output of apitruthtag("metadata") show you between each > > > > > run in facter, irb and puppet? Can you output each to a file and > > > > > analyze the difference? > > > > > > > irb, ruby, or facter: > > > > > > yaml parsed http response = #<YAML::Object:0x2ada01f7cf00> > > > > > > > puppet run: > > > > > > yaml parsed http response = #<Puppet::FileServing::Metadata: > > > > > > 0x2ac7987b9c08> > > > > > > with error: > > > > > > > undefined method `ivars'' for #<Puppet::FileServing::Metadata: > > > > > > 0x2ac7987152c0> > > > > > > #<NoMethodError: undefined method `ivars'' for > > > > > > #<Puppet::FileServing::Metadata:0x2ac7987152c0>> > > > > > > So when you run it with all the Puppet libraries the YAML object is > > > > > being serialized back into a Ruby object > > Puppet::FileServing::Metadata > > > > > ... I''m really curious what the raw YAML output looks like. This > > > > > should only happen I think if the YAML output has meta information > > > > > that matches that class type for example: > > > > > > --- > > > > > !ruby/object:Puppet::FileServing::Metadata {} > > > > > > I get the feeling the YAML you are getting back isn''t what you were > > > > > expecting and looking at the contents of the raw output might give a > > > > > better clue as to why. > > > > > > ken. > > > > > > -- > > > > > "Join us for PuppetConf, September 22nd and 23rd in Portland, OR: > > > >http://bit.ly/puppetconfsig" > > > > > -- > > > > You received this message because you are subscribed to the Google > > Groups > > > > "Puppet Users" group. > > > > To post to this group, send email to puppet-users@googlegroups.com. > > > > To unsubscribe from this group, send email to > > > > puppet-users+unsubscribe@googlegroups.com. > > > > For more options, visit this group at > > > >http://groups.google.com/group/puppet-users?hl=en. > > > > -- > > > Nigel Kersten > > > Product Manager, Puppet Labs > > > Twitter: @nigelkersten > > > > *Join us for **PuppetConf *<http://www.bit.ly/puppetconfsig> > > > September 22nd and 23rd in Portland, Oregon, USA. > > > * > > > * > > > -- > > You received this message because you are subscribed to the Google Groups > > "Puppet Users" group. > > To post to this group, send email to puppet-users@googlegroups.com. > > To unsubscribe from this group, send email to > > puppet-users+unsubscribe@googlegroups.com. > > For more options, visit this group at > >http://groups.google.com/group/puppet-users?hl=en. > > -- > Nigel Kersten > Product Manager, Puppet Labs > Twitter: @nigelkersten > > *Join us for **PuppetConf *<http://www.bit.ly/puppetconfsig> > September 22nd and 23rd in Portland, Oregon, USA. > * > *-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.