Puppet users - Dec 2013 - puppet 3.4.0 upgrade, foreman 1.1-stable, problems that are not provisioning related

I am having problems with our puppet/foreman infrastructure that seem to 
coincide with an upgrade to 3.4.0-1, but that are persisting after a 
downgrade back to 3.3.2-1. There are no obvious errors I can find and I 
don't seem to be having any problems with puppet talking to foreman, so 
overall things look completely normal other than I have puppet classes when 
foreman responds with node information and an empty catalog after the 
puppet master compiles. I am pretty stumped by this and would appreciate 
any thoughts others may have on troubleshooting this problem.

First, here's some more important information about our system. We had been 
running puppet 3.3.2-1 and foreman 1.1-stable for quite some time before 
things bumped up to 3.4.0 over the weekend.
- centos 6.4 for puppet master and nodes
- puppet and puppet-server 3.3.2-1 (I downgraded both puppet and 
puppet-server back to this version since we'd been running successfully 
with it for some time now)
- puppetdb and puppetdb-terminus 1.5.0-1
- foreman 1.1-stable (we've also been running this for some time, we
don't
do any provisioning with it but we use it as an ENC for storing data and 
classifying nodes into roles)
- puppet and foreman running behind nginx/passenger

If the upgrade to 3.4.0 was the sole source of my problem I would expect 
things to have returned back to normal after the downgrade. Since the 
catalog is still empty though, there's got to be something I'm missing. 

Here are a few things I've tried while digging into this.

1) puppet master --compile <my node>
The node is found, but there are no classes identified. Therein lies the 
problem.
...
"version": 1387830624,
    "classes": [
      "settings",
      "default"
    ],
    "environment": "production"
...

2) Firing up irb, I manually checked what foreman returns for <my node>. I
won't post the whole output including parameters, but the important part is 
that the keyset for the classes key in the yaml document matches what I 
would expect for this particular monolithic test box.
...
{"play"=>nil, "sudo"=>nil, "mongo"=>nil,
"rsyslog"=>nil,
"mysql::server"=>nil, "jmx"=>nil,
"nginx::loadbalancer"=>nil, "ssh"=>nil,
"play::install"=>nil, "mongo::seed"=>nil,
"users"=>nil,
"mysql::server::master"=>nil, "logstash"=>nil,
"common"=>nil,
"puppet"=>nil, "zabbix"=>nil,
"motd"=>nil, "logrotate"=>nil,
"java"=>nil,
"yum"=>nil, "ntp"=>nil}

3) I noticed that facter also updated to version 1.7.4 during this time, 
but at least from what I've tried so far this doesn't seem to be a
problem
of any kind.

4) Yes, I did try turning it off and on again. :)

So it seems that somewhere between talking to the ENC and actually 
compiling the catalog that my puppet classes disappear into the ether that 
I have not yet identified. I would appreciate any suggestions people have 
on further troubleshooting, or why the upgrade might have gotten things 
into an odd state that rolling back the package somehow doesn't fix.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/8ab0dbef-8ffc-4b57-b26e-e30991c41085%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.