Puppet users - Apr 2014 - Serious performance issues with Puppet 3.5.x when using Directory Environments

Dear Puppet Users community, I would like to report an issue I experience 
since the first release of Puppet 3.5, and possibly get some input from 
people who face the same problem (if any).


I run a Puppet environment in which all 48 nodes run CentOS 6.5. I have 
been using config file-based environments so far without any trouble. When 
Puppet 3.5.0 was officially released I decided to give give it a try, along 
with the new Directory Environments feature. I disabled my old-style 
environment config while making sure my directory tree was correctly set, 
as described in the "Directory Environments" documentation
page<http://docs.puppetlabs.com/puppet/latest/reference/environments.html>
.

The problem shows up immediately after I restart httpd: Passenger starts 
spawning more and more processes which all consume 25% of my machine CPU, 
reaching a load average of 10+ in less than a minute as soon as the first 
node requests a catalog. This makes the master unusable, it takes between 
200 and 300sec to compile any catalog.


What I have tried:

   1. Switching back from Passenger to WEBrick: same performance issue
   2. Disabling the puppet service on all nodes, and triggering *one*single
manual Puppet run on a random node: same
   performance issue
   3. Disabling my external node classifier (Foreman): same performance 
   issue
   4. Upgrading to Puppet 3.5.1: same performance issue
   5. Switching back to config-based environments and re-enabling all nodes: no 
   issue, my master behaves normally and the load-average remains under 0.25 
   constantly

As you might have noticed, this is definitely not a sizing issue. My master 
has 2GB of RAM and 2 allocated vCPU, but the Puppet master rack process 
starts consuming all resources as soon as I re-enable Directory 
Environments.

When I want to use Directory Environments, the only thing I do is 
commenting the [development] and [production] sections in my puppet.conf 
file, and adding this to the [main] section: environmentpath = 
$confdir/environments

Let me describe my configuration:

*-----------------puppet.conf--------------*
[main]
    logdir = /var/log/puppet.
    rundir = /var/run/puppet
    ssldir = $vardir/ssl
    privatekeydir = $ssldir/private_keys { group = service }
    hostprivkey = $privatekeydir/$certname.pem { mode = 640 }
    autosign       = $confdir/autosign.conf { mode = 664 }

[master]
    autosign       = $confdir/autosign.conf { mode = 664 }
    reports        = foreman
    external_nodes = /etc/puppet/node.rb
    node_terminus  = exec
    ca             = true
    ssldir         = /var/lib/puppet/ssl
    storeconfigs = true
    storeconfigs_backend = puppetdb

[development]
    modulepath     = 
/etc/puppet/environments/development/modules:/usr/share/puppet/modules
    manifest       = /etc/puppet/environments/development/manifests/site.pp
    config_version [production]
    modulepath     = 
/etc/puppet/environments/production/modules:/usr/share/puppet/modules
    manifest       = /etc/puppet/environments/production/manifests/site.pp
    config_version 
*--------------------------------------------------*

*--------------directory tree--------------*/etc/puppet/environments/
|-- development
|   |-- manifests
|   |   `-- site.pp
|   `-- modules
|       |-- activemq
|       |-- apache
|       |-- apt
|       |-- concat
|       |-- datacat
|       |-- environment
|       |-- epel
|       |-- erlang
|       |-- firewall
|       |-- java
|       |-- java_ks
|       |-- mcollective
|       |-- postfix
|       |-- profiles
|       |-- rabbitmq
|       |-- repos
|       |-- stdlib
|       `-- tomcat
`-- production
    |-- manifests
    |   `-- site.pp
    `-- modules
        |-- activemq
        |-- apache
        |-- apt
        |-- concat
        |-- datacat
        |-- environment
        |-- epel
        |-- erlang
        |-- firewall
        |-- inifile
        |-- java
        |-- java_ks
        |-- mcollective
        |-- mysql
        |-- ntp
        |-- postfix
        |-- postgresql
        |-- profiles
        |-- puppetdb
        |-- rabbitmq
        |-- repos
        |-- stdlib
        |-- tomcat
        `-- vsftpd
*--------------------------------------------------*



Can anybody reproduce this it on any platform? Thank you in advance for 
your help!

Toni

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/8036c0fa-57ee-401b-b9d7-03c43dda0fd2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.