Puppet users - Sep 2010 - Puppet performance on Ubuntu Lucid with mod_passanger

Hi,

i''m actually using puppet+mongrel for 630 servers. Performances are
correct, but we need to reboot the puppetmaster every 3 hours because
of memory problems :-(

I decided to migrate to a configuration puppet+apache2+passanger on a
Ubuntu 10.04 system.

During my tests, everything was working correctly but when i moved in
production with the 630 servers, performances was very bad: most of
the clients could not retreived their configurations (with an error
message "err: Configuration retrieval timed out") and the puppetmaster
has a high load (load=~20 with 10 puppetmaster launched, load=4 with 1
puppetmaster launched).

I check everywhere what could be wrong on my system without finding
something :-(
Maybe one here may have a clue ?

Thanks !!


My configuration:
OS: ubuntu 10.04
puppetmaster                        0.25.4-2ubuntu6
libapache2-mod-passenger            2.2.7debian-1

My installation procedure (maybe could help) :
   $ apt-get install puppetmaster
   $ update-rc.d -f puppetmaster remove
   $ perl -i -pe''s@^START=yes@START=no@''
/etc/default/puppetmaster
   $ mkdir /etc/puppet/run
   $ chown puppet:root /etc/puppet/run
   $ apt-get install apache2 libapache2-mod-passenger rails librack-
ruby
   $ a2enmod ssl
   $ a2enmod headers
   => configure Apache and passanger
   $ cat > /etc/apache2/mods-available/passenger.conf << EOF
<IfModule mod_passenger.c>
        PassengerRoot /usr
        PassengerRuby /usr/bin/ruby
        PassengerHighPerformance on
        PassengerMaxPoolSize 10
        PassengerPoolIdleTime 300
        PassengerUseGlobalQueue on
        PassengerStatThrottleRate 120
        RackAutoDetect Off
        RackBaseURI /
        RailsAutoDetect Off
</IfModule>
EOF
   $ cat > /etc/apache2/sites-available/puppetmasterd << EOF
# you probably want to tune these settings
PassengerHighPerformance on
PassengerMaxPoolSize 1
PassengerPoolIdleTime 1500
# PassengerMaxRequests 1000
PassengerStatThrottleRate 120
RackAutoDetect Off
RailsAutoDetect Off

Listen 8140
<VirtualHost *:8140>
        SSLEngine on
        SSLProtocol -ALL +SSLv3 +TLSv1
        SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
        SSLCertificateFile /var/lib/puppet/ssl/certs/
frcrbvesrv0029.ymagis.net.pem
        SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/
frcrbvesrv0029.ymagis.net.pem
        SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
        SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
        # CRL checking should be enabled; if you have problems with
Apache complaining about the CRL,
        SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
        SSLVerifyClient optional
        SSLVerifyDepth 1
        SSLOptions +StdEnvVars

        ErrorLog /var/log/apache2/puppetmaster-error.log
        CustomLog /var/log/apache2/puppetmaster-access.log combined
        CustomLog /var/log/apache2/puppetmaster-ssl_request.log "%t %h
%{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

        DocumentRoot /etc/puppet/rack/public
        RackBaseURI /
        <Directory /etc/puppet/rack/>
                Options None
                AllowOverride None
                Order allow,deny
                allow from all
        </Directory>
</VirtualHost>
EOF
   $ a2ensite puppetmasterd
   mkdir -p /etc/puppet/rack/public
   # ex: http://www.puppetlabs.com/downloads/puppet/puppet-0.25.0beta1.tar.gz
   # see the file ext/rack/README
   cat > /etc/puppet/rack/config.ru << EOF
# a config.ru, for use with every rack-compatible webserver.
# SSL needs to be handled outside this, though.

# if puppet is not in your RUBYLIB:
# $:.push(''/opt/puppet/lib'')

$0 = "puppetmasterd"
require ''puppet''

# if you want debugging:
# ARGV << "--debug"

ARGV << "--rack"
require ''puppet/application/puppetmasterd''
# we''re usually running inside a Rack::Builder.new {} block,
# therefore we need to call run *here*.
run Puppet::Application[:puppetmasterd].run
EOF
   chown puppet:root /etc/puppet/rack/config.ru

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Sep 24, 2010, at 5:03 AM, jtournier wrote:
> Hi,
> 
> i''m actually using puppet+mongrel for 630 servers. Performances
are
> correct, but we need to reboot the puppetmaster every 3 hours because
> of memory problems :-(
> 
> I decided to migrate to a configuration puppet+apache2+passanger on a
> Ubuntu 10.04 system.
> 
> During my tests, everything was working correctly but when i moved in
> production with the 630 servers, performances was very bad: most of
> the clients could not retreived their configurations (with an error
> message "err: Configuration retrieval timed out") and the
puppetmaster
> has a high load (load=~20 with 10 puppetmaster launched, load=4 with 1
> puppetmaster launched).
> 
> I check everywhere what could be wrong on my system without finding
> something :-(
> Maybe one here may have a clue ?
Any chance you ran out of RAM?  This happened to me and the swapping killed my
performance.  Webrick only runs one instance and you''re now running 10
instances which will use up much more RAM.

Also, are you using storedconfigs with mysql-lite?  That can cause strange
things too.
> Thanks !!
> 
> 
> My configuration:
> OS: ubuntu 10.04
> puppetmaster                        0.25.4-2ubuntu6
> libapache2-mod-passenger            2.2.7debian-1
> 
> My installation procedure (maybe could help) :
>   $ apt-get install puppetmaster
>   $ update-rc.d -f puppetmaster remove
>   $ perl -i -pe''s@^START=yes@START=no@''
/etc/default/puppetmaster
>   $ mkdir /etc/puppet/run
>   $ chown puppet:root /etc/puppet/run
>   $ apt-get install apache2 libapache2-mod-passenger rails librack-
> ruby
>   $ a2enmod ssl
>   $ a2enmod headers
>   => configure Apache and passanger
>   $ cat > /etc/apache2/mods-available/passenger.conf << EOF
> <IfModule mod_passenger.c>
>        PassengerRoot /usr
>        PassengerRuby /usr/bin/ruby
>        PassengerHighPerformance on
>        PassengerMaxPoolSize 10
>        PassengerPoolIdleTime 300
>        PassengerUseGlobalQueue on
>        PassengerStatThrottleRate 120
>        RackAutoDetect Off
>        RackBaseURI /
>        RailsAutoDetect Off
> </IfModule>
> EOF
>   $ cat > /etc/apache2/sites-available/puppetmasterd << EOF
> # you probably want to tune these settings
> PassengerHighPerformance on
> PassengerMaxPoolSize 1
> PassengerPoolIdleTime 1500
> # PassengerMaxRequests 1000
> PassengerStatThrottleRate 120
> RackAutoDetect Off
> RailsAutoDetect Off
> 
> Listen 8140
> <VirtualHost *:8140>
>        SSLEngine on
>        SSLProtocol -ALL +SSLv3 +TLSv1
>        SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
>        SSLCertificateFile /var/lib/puppet/ssl/certs/
> frcrbvesrv0029.ymagis.net.pem
>        SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/
> frcrbvesrv0029.ymagis.net.pem
>        SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
>        SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
>        # CRL checking should be enabled; if you have problems with
> Apache complaining about the CRL,
>        SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
>        SSLVerifyClient optional
>        SSLVerifyDepth 1
>        SSLOptions +StdEnvVars
> 
>        ErrorLog /var/log/apache2/puppetmaster-error.log
>        CustomLog /var/log/apache2/puppetmaster-access.log combined
>        CustomLog /var/log/apache2/puppetmaster-ssl_request.log "%t %h
> %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
> 
>        DocumentRoot /etc/puppet/rack/public
>        RackBaseURI /
>        <Directory /etc/puppet/rack/>
>                Options None
>                AllowOverride None
>                Order allow,deny
>                allow from all
>        </Directory>
> </VirtualHost>
> EOF
>   $ a2ensite puppetmasterd
>   mkdir -p /etc/puppet/rack/public
>   # ex:
http://www.puppetlabs.com/downloads/puppet/puppet-0.25.0beta1.tar.gz
>   # see the file ext/rack/README
>   cat > /etc/puppet/rack/config.ru << EOF
> # a config.ru, for use with every rack-compatible webserver.
> # SSL needs to be handled outside this, though.
> 
> # if puppet is not in your RUBYLIB:
> # $:.push(''/opt/puppet/lib'')
> 
> $0 = "puppetmasterd"
> require ''puppet''
> 
> # if you want debugging:
> # ARGV << "--debug"
> 
> ARGV << "--rack"
> require ''puppet/application/puppetmasterd''
> # we''re usually running inside a Rack::Builder.new {} block,
> # therefore we need to call run *here*.
> run Puppet::Application[:puppetmasterd].run
> EOF
>   chown puppet:root /etc/puppet/rack/config.ru
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
> 
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Fri, Sep 24, 2010 at 6:44 PM, Patrick <kc7zzv@gmail.com> wrote:
> Any chance you ran out of RAM?  This happened to me and the swapping killed
> my performance.  Webrick only runs one instance and you''re now
running 10
> instances which will use up much more RAM.
>
No, not a RAM related problem:
$ free -m
                  total       used       free     shared    buffers
cached
Mem:          2013        332       1680          0         67        142
-/+ buffers/cache:        123       1889
Swap:         1303          0       1303


> Also, are you using storedconfigs with mysql-lite?  That can cause strange
> things too.
>
I''m not using storedconfigs.

puppetmasterd can compile the catalogs always in a time < 0.5s. Everything
is looks ok on this side.
But the puppetd client need between 5 and 20 minutes to finish. Maybe the
problem is related to Apache2, but i can''t see where :-(

-- 
Jérôme




> > Thanks !!
> >
> >
> > My configuration:
> > OS: ubuntu 10.04
> > puppetmaster                        0.25.4-2ubuntu6
> > libapache2-mod-passenger            2.2.7debian-1
> >
> > My installation procedure (maybe could help) :
> >   $ apt-get install puppetmaster
> >   $ update-rc.d -f puppetmaster remove
> >   $ perl -i -pe''s@^START=yes@START=no@''
/etc/default/puppetmaster
> >   $ mkdir /etc/puppet/run
> >   $ chown puppet:root /etc/puppet/run
> >   $ apt-get install apache2 libapache2-mod-passenger rails librack-
> > ruby
> >   $ a2enmod ssl
> >   $ a2enmod headers
> >   => configure Apache and passanger
> >   $ cat > /etc/apache2/mods-available/passenger.conf << EOF
> > <IfModule mod_passenger.c>
> >        PassengerRoot /usr
> >        PassengerRuby /usr/bin/ruby
> >        PassengerHighPerformance on
> >        PassengerMaxPoolSize 10
> >        PassengerPoolIdleTime 300
> >        PassengerUseGlobalQueue on
> >        PassengerStatThrottleRate 120
> >        RackAutoDetect Off
> >        RackBaseURI /
> >        RailsAutoDetect Off
> > </IfModule>
> > EOF
> >   $ cat > /etc/apache2/sites-available/puppetmasterd << EOF
> > # you probably want to tune these settings
> > PassengerHighPerformance on
> > PassengerMaxPoolSize 1
> > PassengerPoolIdleTime 1500
> > # PassengerMaxRequests 1000
> > PassengerStatThrottleRate 120
> > RackAutoDetect Off
> > RailsAutoDetect Off
> >
> > Listen 8140
> > <VirtualHost *:8140>
> >        SSLEngine on
> >        SSLProtocol -ALL +SSLv3 +TLSv1
> >        SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
> >        SSLCertificateFile /var/lib/puppet/ssl/certs/
> > frcrbvesrv0029.ymagis.net.pem
> >        SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/
> > frcrbvesrv0029.ymagis.net.pem
> >        SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
> >        SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
> >        # CRL checking should be enabled; if you have problems with
> > Apache complaining about the CRL,
> >        SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
> >        SSLVerifyClient optional
> >        SSLVerifyDepth 1
> >        SSLOptions +StdEnvVars
> >
> >        ErrorLog /var/log/apache2/puppetmaster-error.log
> >        CustomLog /var/log/apache2/puppetmaster-access.log combined
> >        CustomLog /var/log/apache2/puppetmaster-ssl_request.log
"%t %h
> > %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
> >
> >        DocumentRoot /etc/puppet/rack/public
> >        RackBaseURI /
> >        <Directory /etc/puppet/rack/>
> >                Options None
> >                AllowOverride None
> >                Order allow,deny
> >                allow from all
> >        </Directory>
> > </VirtualHost>
> > EOF
> >   $ a2ensite puppetmasterd
> >   mkdir -p /etc/puppet/rack/public
> >   # ex:
> http://www.puppetlabs.com/downloads/puppet/puppet-0.25.0beta1.tar.gz
> >   # see the file ext/rack/README
> >   cat > /etc/puppet/rack/config.ru << EOF
> > # a config.ru, for use with every rack-compatible webserver.
> > # SSL needs to be handled outside this, though.
> >
> > # if puppet is not in your RUBYLIB:
> > # $:.push(''/opt/puppet/lib'')
> >
> > $0 = "puppetmasterd"
> > require ''puppet''
> >
> > # if you want debugging:
> > # ARGV << "--debug"
> >
> > ARGV << "--rack"
> > require ''puppet/application/puppetmasterd''
> > # we''re usually running inside a Rack::Builder.new {} block,
> > # therefore we need to call run *here*.
> > run Puppet::Application[:puppetmasterd].run
> > EOF
> >   chown puppet:root /etc/puppet/rack/config.ru
> >
> > --
> > You received this message because you are subscribed to the Google
Groups
> "Puppet Users" group.
> > To post to this group, send email to puppet-users@googlegroups.com.
> > To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> > For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>

-- 
Jérôme

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.