thr3ads.net - Puppet users - [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5 [Apr 2010]

If this information is useful, please help other people find it:
Share via:

Arnaud Gomes-do-Vale

2010-Apr-27 13:05 UTC

[Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5

Hi folks,

When I enabled stored configs on my puppet master, it began keeping lots
of open connections to the MySQL server (as far as I understand it
opened one connection for each client run and never closed them).

I set up the server to close idle connections after 5 minutes. Now the
database server does close the connections but the puppet master keeps
them in CLOSE_WAIT state and eventually runs out of file descriptors.

Here is a sample of the output of lsof on the puppet master:

.../...
puppetmas 16418 puppet  617u  IPv4            2683389               TCP
puppetmaster.ircam.fr:57097->mysql2.ircam.fr:mysql (CLOSE_WAIT)
puppetmas 16418 puppet  618u  IPv4            2683494               TCP
puppetmaster.ircam.fr:57099->mysql2.ircam.fr:mysql (CLOSE_WAIT)
puppetmas 16418 puppet  619u  IPv4            2683506               TCP
puppetmaster.ircam.fr:57100->mysql2.ircam.fr:mysql (CLOSE_WAIT)
puppetmas 16418 puppet  620u  IPv4            2684180               TCP
puppetmaster.ircam.fr:57108->mysql2.ircam.fr:mysql (CLOSE_WAIT)
puppetmas 16418 puppet  621u  IPv4            2683726               TCP
puppetmaster.ircam.fr:57102->mysql2.ircam.fr:mysql (CLOSE_WAIT)
.../...

The matching sockets don''t appear on the MySQL server.

The puppet master runs CentOS 5 with (pieces of) EPEL; here are the
installed versions of the relevant packages:

puppet-0.25.4-1.el5
puppet-server-0.25.4-1.el5
rubygem-activeresource-2.1.1-1.el5
rubygem-activesupport-2.1.1-2.el5
rubygem-activerecord-2.1.1-2.el5
rubygems-1.3.1-1.el5
rubygem-rails-2.1.1-2.el5
ruby-1.8.5-5.el5_4.8
ruby-mysql-2.7.3-1.el5

And the puppetmaster config :

[puppetmasterd]
    node_terminus = ldap
    ldapserver = nsldap.ircam.fr
    ldapbase = ou=Hosts,dc=ircam,dc=fr
    modulepath = /etc/puppet/modules:/usr/share/puppet:/var/lib/puppet/modules

    storeconfigs = true
    dbserver = mysql2.ircam.fr
    dbadapter = mysql
    dbname = XXXXXXXXXXXXXXXX
    dbuser = XXXXXXXXXXXXXXXX
    dbpassword = XXXXXXXXXXXXXXXXXXXXXXXXXXX

    reports = store,log,rrdgraph
    rrdgraph = true

-- 
Arnaud

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Brice Figureau

2010-Apr-27 14:30 UTC

head link

Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5

Hi,

On Tue, 2010-04-27 at 15:05 +0200, Arnaud Gomes-do-Vale
wrote:> Hi folks,
> 
> When I enabled stored configs on my puppet master, it began keeping lots
> of open connections to the MySQL server (as far as I understand it
> opened one connection for each client run and never closed them).
> 
> I set up the server to close idle connections after 5 minutes. Now the
> database server does close the connections but the puppet master keeps
> them in CLOSE_WAIT state and eventually runs out of file descriptors.
> 
> Here is a sample of the output of lsof on the puppet master:
> 
> .../...
> puppetmas 16418 puppet  617u  IPv4            2683389               TCP
puppetmaster.ircam.fr:57097->mysql2.ircam.fr:mysql (CLOSE_WAIT)
> puppetmas 16418 puppet  618u  IPv4            2683494               TCP
puppetmaster.ircam.fr:57099->mysql2.ircam.fr:mysql (CLOSE_WAIT)
> puppetmas 16418 puppet  619u  IPv4            2683506               TCP
puppetmaster.ircam.fr:57100->mysql2.ircam.fr:mysql (CLOSE_WAIT)
> puppetmas 16418 puppet  620u  IPv4            2684180               TCP
puppetmaster.ircam.fr:57108->mysql2.ircam.fr:mysql (CLOSE_WAIT)
> puppetmas 16418 puppet  621u  IPv4            2683726               TCP
puppetmaster.ircam.fr:57102->mysql2.ircam.fr:mysql (CLOSE_WAIT)
> .../...
> 
> The matching sockets don''t appear on the MySQL server.
> 
> The puppet master runs CentOS 5 with (pieces of) EPEL; here are the
> installed versions of the relevant packages:
> 
> puppet-0.25.4-1.el5
> puppet-server-0.25.4-1.el5
> rubygem-activeresource-2.1.1-1.el5
> rubygem-activesupport-2.1.1-2.el5
> rubygem-activerecord-2.1.1-2.el5
Can you try with a newer rails/active record?
This one is a little bit old, and I''m not sure there aren''t
some bugs.
I vaguely remember that puppet required rails 2.2, but I might be wrong.
-- 
Brice Figureau
Follow the latest Puppet Community evolutions on www.planetpuppet.org!

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Scott Smith

2010-Apr-27 15:06 UTC

head link

Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5

You might also try the latest ruby-mysql gem.

Brice Figureau <brice-puppet@daysofwonder.com> wrote:
>Hi,
>
>On Tue, 2010-04-27 at 15:05 +0200, Arnaud Gomes-do-Vale wrote:
>> Hi folks,
>> 
>> When I enabled stored configs on my puppet master, it began keeping
lots
>> of open connections to the MySQL server (as far as I understand it
>> opened one connection for each client run and never closed them).
>> 
>> I set up the server to close idle connections after 5 minutes. Now the
>> database server does close the connections but the puppet master keeps
>> them in CLOSE_WAIT state and eventually runs out of file descriptors.
>> 
>> Here is a sample of the output of lsof on the puppet master:
>> 
>> .../...
>> puppetmas 16418 puppet  617u  IPv4            2683389               TCP
puppetmaster.ircam.fr:57097->mysql2.ircam.fr:mysql (CLOSE_WAIT)
>> puppetmas 16418 puppet  618u  IPv4            2683494               TCP
puppetmaster.ircam.fr:57099->mysql2.ircam.fr:mysql (CLOSE_WAIT)
>> puppetmas 16418 puppet  619u  IPv4            2683506               TCP
puppetmaster.ircam.fr:57100->mysql2.ircam.fr:mysql (CLOSE_WAIT)
>> puppetmas 16418 puppet  620u  IPv4            2684180               TCP
puppetmaster.ircam.fr:57108->mysql2.ircam.fr:mysql (CLOSE_WAIT)
>> puppetmas 16418 puppet  621u  IPv4            2683726               TCP
puppetmaster.ircam.fr:57102->mysql2.ircam.fr:mysql (CLOSE_WAIT)
>> .../...
>> 
>> The matching sockets don''t appear on the MySQL server.
>> 
>> The puppet master runs CentOS 5 with (pieces of) EPEL; here are the
>> installed versions of the relevant packages:
>> 
>> puppet-0.25.4-1.el5
>> puppet-server-0.25.4-1.el5
>> rubygem-activeresource-2.1.1-1.el5
>> rubygem-activesupport-2.1.1-2.el5
>> rubygem-activerecord-2.1.1-2.el5
>
>Can you try with a newer rails/active record?
>This one is a little bit old, and I''m not sure there
aren''t some bugs.
>I vaguely remember that puppet required rails 2.2, but I might be wrong.
>-- 
>Brice Figureau
>Follow the latest Puppet Community evolutions on www.planetpuppet.org!
>
>-- 
>You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
>To post to this group, send email to puppet-users@googlegroups.com.
>To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
>For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Arnaud Gomes-do-Vale

2010-Apr-27 15:46 UTC

head link

Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5

Brice Figureau <brice-puppet@daysofwonder.com> writes:
> Can you try with a newer rails/active record?
> This one is a little bit old, and I''m not sure there
aren''t some bugs.
> I vaguely remember that puppet required rails 2.2, but I might be wrong.
Will try. I was hoping I could get away with using whatever is in EPEL;
has anybody tested this kind of setup?

-- 
Arnaud

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Christopher

2010-Apr-27 18:01 UTC

head link

Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5

Hello...


On Tue, 2010-04-27 at 15:05 +0200, Arnaud Gomes-do-Vale
wrote:> Hi folks,
> 
> When I enabled stored configs on my puppet master, it began keeping lots
> of open connections to the MySQL server (as far as I understand it
> opened one connection for each client run and never closed them).
> 
> I set up the server to close idle connections after 5 minutes. Now the
> database server does close the connections but the puppet master keeps
> them in CLOSE_WAIT state and eventually runs out of file descriptors.
> 

I ran into this same thing.  Update (rebuild) to the following rpms
versions from fedora 11:

rubygem-activerecord.noarch             1:2.3.2-4.fc11
rubygem-activesupport.noarch            1:2.3.2-2.fc11
rubygem-rack.noarch                     1.0.0-1.fc11

For various reasons, these versions cannot be pushed into EPEL.




> Here is a sample of the output of lsof on the puppet master:
> 
> .../...
> puppetmas 16418 puppet  617u  IPv4            2683389               TCP
puppetmaster.ircam.fr:57097->mysql2.ircam.fr:mysql (CLOSE_WAIT)
> puppetmas 16418 puppet  618u  IPv4            2683494               TCP
puppetmaster.ircam.fr:57099->mysql2.ircam.fr:mysql (CLOSE_WAIT)
> puppetmas 16418 puppet  619u  IPv4            2683506               TCP
puppetmaster.ircam.fr:57100->mysql2.ircam.fr:mysql (CLOSE_WAIT)
> puppetmas 16418 puppet  620u  IPv4            2684180               TCP
puppetmaster.ircam.fr:57108->mysql2.ircam.fr:mysql (CLOSE_WAIT)
> puppetmas 16418 puppet  621u  IPv4            2683726               TCP
puppetmaster.ircam.fr:57102->mysql2.ircam.fr:mysql (CLOSE_WAIT)
> .../...
> 
> The matching sockets don''t appear on the MySQL server.
> 
> The puppet master runs CentOS 5 with (pieces of) EPEL; here are the
> installed versions of the relevant packages:
> 
> puppet-0.25.4-1.el5
> puppet-server-0.25.4-1.el5
> rubygem-activeresource-2.1.1-1.el5
> rubygem-activesupport-2.1.1-2.el5
> rubygem-activerecord-2.1.1-2.el5
> rubygems-1.3.1-1.el5
> rubygem-rails-2.1.1-2.el5
> ruby-1.8.5-5.el5_4.8
> ruby-mysql-2.7.3-1.el5
> 
> And the puppetmaster config :
> 
> [puppetmasterd]
>     node_terminus = ldap
>     ldapserver = nsldap.ircam.fr
>     ldapbase = ou=Hosts,dc=ircam,dc=fr
>     modulepath =
/etc/puppet/modules:/usr/share/puppet:/var/lib/puppet/modules
> 
>     storeconfigs = true
>     dbserver = mysql2.ircam.fr
>     dbadapter = mysql
>     dbname = XXXXXXXXXXXXXXXX
>     dbuser = XXXXXXXXXXXXXXXX
>     dbpassword = XXXXXXXXXXXXXXXXXXXXXXXXXXX
> 
>     reports = store,log,rrdgraph
>     rrdgraph = true
> 
> -- 
> Arnaud
> 

-- 
Christopher McCrory
 "The guy that keeps the servers running"
 
chrismcc@pricegrabber.com
 http://www.pricegrabber.com
 
Let''s face it, there''s no Hollow Earth, no robots, and
no ''mute rays.'' And even if there were, waxed paper is
no defense.  I tried it.  Only tinfoil works.


-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Todd Zullinger

2010-Apr-28 13:21 UTC

head link

Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5

Arnaud Gomes-do-Vale wrote:> Brice Figureau <brice-puppet@daysofwonder.com> writes:
>
>> Can you try with a newer rails/active record?  This one is a little
>> bit old, and I''m not sure there aren''t some bugs.  I
vaguely
>> remember that puppet required rails 2.2, but I might be wrong.
Well, puppet 0.25 doesn''t refuse to run with rails 2.1 and 0.25 got
some patches to allow better support for 2.1, so it''s not completely
"unsupported" from a user perspective.  The wiki does list >= 2.2.2
for 0.25 and <= 2.2.2 for 0.24 though.
> Will try. I was hoping I could get away with using whatever is in
> EPEL; has anybody tested this kind of setup?
It''s definitely unfortunate that what is in EPEL suffers from this
bug.  I would love to see either puppet or rails there patched to work
around this.  I filed https://bugzilla.redhat.com/572722 for this
issue but haven''t heard anything back from the rails packagers yet.

There was a bug in the Puppet Labs redmine about these leaks, but I
think it was closed.  I need to find that bug again and reopen it.

-- 
Todd        OpenPGP -> KeyID: 0xBEAF0CE3 | URL: www.pobox.com/~tmz/pgp
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A neurosis is a secret you don''t know you''re keeping.
    -- Kenneth Tynan

Arnaud Gomes-do-Vale

2010-Apr-29 15:58 UTC

head link

Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5

Christopher <chrismcc@pricegrabber.com> writes:
> I ran into this same thing.  Update (rebuild) to the following rpms
> versions from fedora 11:
>
> rubygem-activerecord.noarch             1:2.3.2-4.fc11
> rubygem-activesupport.noarch            1:2.3.2-2.fc11
> rubygem-rack.noarch                     1.0.0-1.fc11
Thanks, this fixed the issue.

-- 
Arnaud

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Jason Koppe

2010-Jun-19 01:48 UTC

head link

Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5

I''m having the same issue on CentOS5.2 with the following packages:

*yum*
puppet-0.25.5-1.el5
puppet-server-0.25.5-1.el5
rubygems-1.3.1-1.el5
ruby-shadow-1.4.1-7.el5
ruby-devel-1.8.6.111-1
ruby-1.8.6.111-1
ruby-ri-1.8.6.111-1
ruby-docs-1.8.5-5.el5_3.7
ruby-augeas-0.3.0-1.el5
ruby-irb-1.8.6.111-1
ruby-rdoc-1.8.6.111-1
ruby-mode-1.8.5-5.el5_3.7
libselinux-ruby-1.33.4-5.5.el5
ruby-libs-1.8.6.111-1

*gem*
activerecord (2.3.8, 2.3.4)
activeresource (2.3.8)
activesupport (2.3.8, 2.3.4)
mysql (2.8.1)
rack (1.1.0)
rails (2.3.8)

Suggestions?


On Thu, Apr 29, 2010 at 10:58 AM, Arnaud Gomes-do-Vale <
Arnaud.Gomes@ircam.fr> wrote:
> Christopher <chrismcc@pricegrabber.com> writes:
>
> > I ran into this same thing.  Update (rebuild) to the following rpms
> > versions from fedora 11:
> >
> > rubygem-activerecord.noarch             1:2.3.2-4.fc11
> > rubygem-activesupport.noarch            1:2.3.2-2.fc11
> > rubygem-rack.noarch                     1.0.0-1.fc11
>
> Thanks, this fixed the issue.
>
> --
> Arnaud
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>

-- 
Jason Koppe
Jason.Robert.Koppe@gmail.com
Cell (210) 445-8242

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Reasonably Related Threads

Search for more possibly parallel threads

Puppet users - Apr 2010 - Puppetmaster with stored configs leaks file descriptors on CentOS 5

[Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5

Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5

Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5

Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5

Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5

Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5

Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5

Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5

Reasonably Related Threads