Arnaud Gomes-do-Vale
2010-Apr-27 13:05 UTC
[Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5
Hi folks, When I enabled stored configs on my puppet master, it began keeping lots of open connections to the MySQL server (as far as I understand it opened one connection for each client run and never closed them). I set up the server to close idle connections after 5 minutes. Now the database server does close the connections but the puppet master keeps them in CLOSE_WAIT state and eventually runs out of file descriptors. Here is a sample of the output of lsof on the puppet master: .../... puppetmas 16418 puppet 617u IPv4 2683389 TCP puppetmaster.ircam.fr:57097->mysql2.ircam.fr:mysql (CLOSE_WAIT) puppetmas 16418 puppet 618u IPv4 2683494 TCP puppetmaster.ircam.fr:57099->mysql2.ircam.fr:mysql (CLOSE_WAIT) puppetmas 16418 puppet 619u IPv4 2683506 TCP puppetmaster.ircam.fr:57100->mysql2.ircam.fr:mysql (CLOSE_WAIT) puppetmas 16418 puppet 620u IPv4 2684180 TCP puppetmaster.ircam.fr:57108->mysql2.ircam.fr:mysql (CLOSE_WAIT) puppetmas 16418 puppet 621u IPv4 2683726 TCP puppetmaster.ircam.fr:57102->mysql2.ircam.fr:mysql (CLOSE_WAIT) .../... The matching sockets don''t appear on the MySQL server. The puppet master runs CentOS 5 with (pieces of) EPEL; here are the installed versions of the relevant packages: puppet-0.25.4-1.el5 puppet-server-0.25.4-1.el5 rubygem-activeresource-2.1.1-1.el5 rubygem-activesupport-2.1.1-2.el5 rubygem-activerecord-2.1.1-2.el5 rubygems-1.3.1-1.el5 rubygem-rails-2.1.1-2.el5 ruby-1.8.5-5.el5_4.8 ruby-mysql-2.7.3-1.el5 And the puppetmaster configĀ : [puppetmasterd] node_terminus = ldap ldapserver = nsldap.ircam.fr ldapbase = ou=Hosts,dc=ircam,dc=fr modulepath = /etc/puppet/modules:/usr/share/puppet:/var/lib/puppet/modules storeconfigs = true dbserver = mysql2.ircam.fr dbadapter = mysql dbname = XXXXXXXXXXXXXXXX dbuser = XXXXXXXXXXXXXXXX dbpassword = XXXXXXXXXXXXXXXXXXXXXXXXXXX reports = store,log,rrdgraph rrdgraph = true -- Arnaud -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Brice Figureau
2010-Apr-27 14:30 UTC
Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5
Hi, On Tue, 2010-04-27 at 15:05 +0200, Arnaud Gomes-do-Vale wrote:> Hi folks, > > When I enabled stored configs on my puppet master, it began keeping lots > of open connections to the MySQL server (as far as I understand it > opened one connection for each client run and never closed them). > > I set up the server to close idle connections after 5 minutes. Now the > database server does close the connections but the puppet master keeps > them in CLOSE_WAIT state and eventually runs out of file descriptors. > > Here is a sample of the output of lsof on the puppet master: > > .../... > puppetmas 16418 puppet 617u IPv4 2683389 TCP puppetmaster.ircam.fr:57097->mysql2.ircam.fr:mysql (CLOSE_WAIT) > puppetmas 16418 puppet 618u IPv4 2683494 TCP puppetmaster.ircam.fr:57099->mysql2.ircam.fr:mysql (CLOSE_WAIT) > puppetmas 16418 puppet 619u IPv4 2683506 TCP puppetmaster.ircam.fr:57100->mysql2.ircam.fr:mysql (CLOSE_WAIT) > puppetmas 16418 puppet 620u IPv4 2684180 TCP puppetmaster.ircam.fr:57108->mysql2.ircam.fr:mysql (CLOSE_WAIT) > puppetmas 16418 puppet 621u IPv4 2683726 TCP puppetmaster.ircam.fr:57102->mysql2.ircam.fr:mysql (CLOSE_WAIT) > .../... > > The matching sockets don''t appear on the MySQL server. > > The puppet master runs CentOS 5 with (pieces of) EPEL; here are the > installed versions of the relevant packages: > > puppet-0.25.4-1.el5 > puppet-server-0.25.4-1.el5 > rubygem-activeresource-2.1.1-1.el5 > rubygem-activesupport-2.1.1-2.el5 > rubygem-activerecord-2.1.1-2.el5Can you try with a newer rails/active record? This one is a little bit old, and I''m not sure there aren''t some bugs. I vaguely remember that puppet required rails 2.2, but I might be wrong. -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Scott Smith
2010-Apr-27 15:06 UTC
Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5
You might also try the latest ruby-mysql gem. Brice Figureau <brice-puppet@daysofwonder.com> wrote:>Hi, > >On Tue, 2010-04-27 at 15:05 +0200, Arnaud Gomes-do-Vale wrote: >> Hi folks, >> >> When I enabled stored configs on my puppet master, it began keeping lots >> of open connections to the MySQL server (as far as I understand it >> opened one connection for each client run and never closed them). >> >> I set up the server to close idle connections after 5 minutes. Now the >> database server does close the connections but the puppet master keeps >> them in CLOSE_WAIT state and eventually runs out of file descriptors. >> >> Here is a sample of the output of lsof on the puppet master: >> >> .../... >> puppetmas 16418 puppet 617u IPv4 2683389 TCP puppetmaster.ircam.fr:57097->mysql2.ircam.fr:mysql (CLOSE_WAIT) >> puppetmas 16418 puppet 618u IPv4 2683494 TCP puppetmaster.ircam.fr:57099->mysql2.ircam.fr:mysql (CLOSE_WAIT) >> puppetmas 16418 puppet 619u IPv4 2683506 TCP puppetmaster.ircam.fr:57100->mysql2.ircam.fr:mysql (CLOSE_WAIT) >> puppetmas 16418 puppet 620u IPv4 2684180 TCP puppetmaster.ircam.fr:57108->mysql2.ircam.fr:mysql (CLOSE_WAIT) >> puppetmas 16418 puppet 621u IPv4 2683726 TCP puppetmaster.ircam.fr:57102->mysql2.ircam.fr:mysql (CLOSE_WAIT) >> .../... >> >> The matching sockets don''t appear on the MySQL server. >> >> The puppet master runs CentOS 5 with (pieces of) EPEL; here are the >> installed versions of the relevant packages: >> >> puppet-0.25.4-1.el5 >> puppet-server-0.25.4-1.el5 >> rubygem-activeresource-2.1.1-1.el5 >> rubygem-activesupport-2.1.1-2.el5 >> rubygem-activerecord-2.1.1-2.el5 > >Can you try with a newer rails/active record? >This one is a little bit old, and I''m not sure there aren''t some bugs. >I vaguely remember that puppet required rails 2.2, but I might be wrong. >-- >Brice Figureau >Follow the latest Puppet Community evolutions on www.planetpuppet.org! > >-- >You received this message because you are subscribed to the Google Groups "Puppet Users" group. >To post to this group, send email to puppet-users@googlegroups.com. >To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. >For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Arnaud Gomes-do-Vale
2010-Apr-27 15:46 UTC
Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5
Brice Figureau <brice-puppet@daysofwonder.com> writes:> Can you try with a newer rails/active record? > This one is a little bit old, and I''m not sure there aren''t some bugs. > I vaguely remember that puppet required rails 2.2, but I might be wrong.Will try. I was hoping I could get away with using whatever is in EPEL; has anybody tested this kind of setup? -- Arnaud -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Christopher
2010-Apr-27 18:01 UTC
Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5
Hello... On Tue, 2010-04-27 at 15:05 +0200, Arnaud Gomes-do-Vale wrote:> Hi folks, > > When I enabled stored configs on my puppet master, it began keeping lots > of open connections to the MySQL server (as far as I understand it > opened one connection for each client run and never closed them). > > I set up the server to close idle connections after 5 minutes. Now the > database server does close the connections but the puppet master keeps > them in CLOSE_WAIT state and eventually runs out of file descriptors. >I ran into this same thing. Update (rebuild) to the following rpms versions from fedora 11: rubygem-activerecord.noarch 1:2.3.2-4.fc11 rubygem-activesupport.noarch 1:2.3.2-2.fc11 rubygem-rack.noarch 1.0.0-1.fc11 For various reasons, these versions cannot be pushed into EPEL.> Here is a sample of the output of lsof on the puppet master: > > .../... > puppetmas 16418 puppet 617u IPv4 2683389 TCP puppetmaster.ircam.fr:57097->mysql2.ircam.fr:mysql (CLOSE_WAIT) > puppetmas 16418 puppet 618u IPv4 2683494 TCP puppetmaster.ircam.fr:57099->mysql2.ircam.fr:mysql (CLOSE_WAIT) > puppetmas 16418 puppet 619u IPv4 2683506 TCP puppetmaster.ircam.fr:57100->mysql2.ircam.fr:mysql (CLOSE_WAIT) > puppetmas 16418 puppet 620u IPv4 2684180 TCP puppetmaster.ircam.fr:57108->mysql2.ircam.fr:mysql (CLOSE_WAIT) > puppetmas 16418 puppet 621u IPv4 2683726 TCP puppetmaster.ircam.fr:57102->mysql2.ircam.fr:mysql (CLOSE_WAIT) > .../... > > The matching sockets don''t appear on the MySQL server. > > The puppet master runs CentOS 5 with (pieces of) EPEL; here are the > installed versions of the relevant packages: > > puppet-0.25.4-1.el5 > puppet-server-0.25.4-1.el5 > rubygem-activeresource-2.1.1-1.el5 > rubygem-activesupport-2.1.1-2.el5 > rubygem-activerecord-2.1.1-2.el5 > rubygems-1.3.1-1.el5 > rubygem-rails-2.1.1-2.el5 > ruby-1.8.5-5.el5_4.8 > ruby-mysql-2.7.3-1.el5 > > And the puppetmaster config : > > [puppetmasterd] > node_terminus = ldap > ldapserver = nsldap.ircam.fr > ldapbase = ou=Hosts,dc=ircam,dc=fr > modulepath = /etc/puppet/modules:/usr/share/puppet:/var/lib/puppet/modules > > storeconfigs = true > dbserver = mysql2.ircam.fr > dbadapter = mysql > dbname = XXXXXXXXXXXXXXXX > dbuser = XXXXXXXXXXXXXXXX > dbpassword = XXXXXXXXXXXXXXXXXXXXXXXXXXX > > reports = store,log,rrdgraph > rrdgraph = true > > -- > Arnaud >-- Christopher McCrory "The guy that keeps the servers running" chrismcc@pricegrabber.com http://www.pricegrabber.com Let''s face it, there''s no Hollow Earth, no robots, and no ''mute rays.'' And even if there were, waxed paper is no defense. I tried it. Only tinfoil works. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Todd Zullinger
2010-Apr-28 13:21 UTC
Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5
Arnaud Gomes-do-Vale wrote:> Brice Figureau <brice-puppet@daysofwonder.com> writes: > >> Can you try with a newer rails/active record? This one is a little >> bit old, and I''m not sure there aren''t some bugs. I vaguely >> remember that puppet required rails 2.2, but I might be wrong.Well, puppet 0.25 doesn''t refuse to run with rails 2.1 and 0.25 got some patches to allow better support for 2.1, so it''s not completely "unsupported" from a user perspective. The wiki does list >= 2.2.2 for 0.25 and <= 2.2.2 for 0.24 though.> Will try. I was hoping I could get away with using whatever is in > EPEL; has anybody tested this kind of setup?It''s definitely unfortunate that what is in EPEL suffers from this bug. I would love to see either puppet or rails there patched to work around this. I filed https://bugzilla.redhat.com/572722 for this issue but haven''t heard anything back from the rails packagers yet. There was a bug in the Puppet Labs redmine about these leaks, but I think it was closed. I need to find that bug again and reopen it. -- Todd OpenPGP -> KeyID: 0xBEAF0CE3 | URL: www.pobox.com/~tmz/pgp ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A neurosis is a secret you don''t know you''re keeping. -- Kenneth Tynan
Arnaud Gomes-do-Vale
2010-Apr-29 15:58 UTC
Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5
Christopher <chrismcc@pricegrabber.com> writes:> I ran into this same thing. Update (rebuild) to the following rpms > versions from fedora 11: > > rubygem-activerecord.noarch 1:2.3.2-4.fc11 > rubygem-activesupport.noarch 1:2.3.2-2.fc11 > rubygem-rack.noarch 1.0.0-1.fc11Thanks, this fixed the issue. -- Arnaud -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Jason Koppe
2010-Jun-19 01:48 UTC
Re: [Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5
I''m having the same issue on CentOS5.2 with the following packages: *yum* puppet-0.25.5-1.el5 puppet-server-0.25.5-1.el5 rubygems-1.3.1-1.el5 ruby-shadow-1.4.1-7.el5 ruby-devel-1.8.6.111-1 ruby-1.8.6.111-1 ruby-ri-1.8.6.111-1 ruby-docs-1.8.5-5.el5_3.7 ruby-augeas-0.3.0-1.el5 ruby-irb-1.8.6.111-1 ruby-rdoc-1.8.6.111-1 ruby-mode-1.8.5-5.el5_3.7 libselinux-ruby-1.33.4-5.5.el5 ruby-libs-1.8.6.111-1 *gem* activerecord (2.3.8, 2.3.4) activeresource (2.3.8) activesupport (2.3.8, 2.3.4) mysql (2.8.1) rack (1.1.0) rails (2.3.8) Suggestions? On Thu, Apr 29, 2010 at 10:58 AM, Arnaud Gomes-do-Vale < Arnaud.Gomes@ircam.fr> wrote:> Christopher <chrismcc@pricegrabber.com> writes: > > > I ran into this same thing. Update (rebuild) to the following rpms > > versions from fedora 11: > > > > rubygem-activerecord.noarch 1:2.3.2-4.fc11 > > rubygem-activesupport.noarch 1:2.3.2-2.fc11 > > rubygem-rack.noarch 1.0.0-1.fc11 > > Thanks, this fixed the issue. > > -- > Arnaud > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- Jason Koppe Jason.Robert.Koppe@gmail.com Cell (210) 445-8242 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.