Puppet users - May 2011 - Puppetmaster leaving files open with "too many files open" error

Every few days I have to restart the puppetmaster process due to this
error

"Could not read YAML data for node .... Too many open files".

I have 9 clients being controlled by Puppet and each one of those will
generate the same error once this happens.  Running ''lsof''
during this
time shows me 100''s of entries like this

"puppetmas 10443           puppet   14u     unix
0xffff810014c7e180            17057251 socket".

The only way to remedy this is by restarting the puppetmaster
process.  Once I restart the puppet master process it seems that a new
lsof entry is created and stays present for every client check.

All clients and puppetmaster are running 2.6.8 on CentOS 5.6 x64.  Any
ideas?  If more output and debug info is needed please let me know.
Also if switching to a mysql data store could get rid of this then
that would work as I was planning on it eventually.

Thanks
- Trey

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I''m pretty sure there is an open bug on this, I was looking around on
this
the other day. It''s related to using stored configs and Puppet/ruby not
closing the connections to the database servers.

--
Nathan Clemons
http://www.livemocha.com
The worlds largest online language learning community



On Fri, May 20, 2011 at 10:01 PM, treydock <treydock@gmail.com> wrote:
> Every few days I have to restart the puppetmaster process due to this
> error
>
> "Could not read YAML data for node .... Too many open files".
>
> I have 9 clients being controlled by Puppet and each one of those will
> generate the same error once this happens.  Running
''lsof'' during this
> time shows me 100''s of entries like this
>
> "puppetmas 10443           puppet   14u     unix
> 0xffff810014c7e180            17057251 socket".
>
> The only way to remedy this is by restarting the puppetmaster
> process.  Once I restart the puppet master process it seems that a new
> lsof entry is created and stays present for every client check.
>
> All clients and puppetmaster are running 2.6.8 on CentOS 5.6 x64.  Any
> ideas?  If more output and debug info is needed please let me know.
> Also if switching to a mysql data store could get rid of this then
> that would work as I was planning on it eventually.
>
> Thanks
> - Trey
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

What is the current issue number?  The only one I could find that
mentions the same problem is this , http://projects.puppetlabs.com/issues/7203.

Would any changes to my current puppet.conf (see below) make this
problem no longer an issue?  I''m not suing storeconfig, and not really
sure if I need to be doing anything as far as saving the inventory
information.  My setup is very basic, but right now can''t run for more
than 2 or 3 days without having to restart the puppetmaster service.

[master]
        certname=puppetmaster
        reports = http, store

        # Configure Inventory Service
        facts_terminus = inventory_active_record
        dbadapter = mysql
        dbname = puppet_inv
        dbuser = puppet_inv
        dbpassword = cabb0ac9ed925867f93587ccfb147ebf15dec8a3
        dbserver = localhost
        dbsocket = /var/lib/mysql/mysql.sock


Thanks
- Trey

On May 21, 6:08 am, Nathan Clemons <nat...@livemocha.com>
wrote:
> I''m pretty sure there is an open bug on this, I was looking around
on this
> the other day. It''s related to using stored configs and
Puppet/ruby not
> closing the connections to the database servers.
>
> --
> Nathan Clemonshttp://www.livemocha.com
> The worlds largest online language learning community
>
>
>
>
>
>
>
> On Fri, May 20, 2011 at 10:01 PM, treydock <treyd...@gmail.com>
wrote:
> > Every few days I have to restart the puppetmaster process due to this
> > error
>
> > "Could not read YAML data for node .... Too many open
files".
>
> > I have 9 clients being controlled by Puppet and each one of those will
> > generate the same error once this happens.  Running
''lsof'' during this
> > time shows me 100''s of entries like this
>
> > "puppetmas 10443           puppet   14u     unix
> > 0xffff810014c7e180            17057251 socket".
>
> > The only way to remedy this is by restarting the puppetmaster
> > process.  Once I restart the puppet master process it seems that a new
> > lsof entry is created and stays present for every client check.
>
> > All clients and puppetmaster are running 2.6.8 on CentOS 5.6 x64.  Any
> > ideas?  If more output and debug info is needed please let me know.
> > Also if switching to a mysql data store could get rid of this then
> > that would work as I was planning on it eventually.
>
> > Thanks
> > - Trey
>
> > --
> > You received this message because you are subscribed to the Google
Groups
> > "Puppet Users" group.
> > To post to this group, send email to puppet-users@googlegroups.com.
> > To unsubscribe from this group, send email to
> > puppet-users+unsubscribe@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/puppet-users?hl=en.
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

treydock wrote:
> What is the current issue number?  The only one I could find that
> mentions the same problem is this ,
http://projects.puppetlabs.com/issues/7203.
https://bugzilla.redhat.com/show_bug.cgi?id=572722 and
http://projects.puppetlabs.com/issues/3693 are both relevant here.

I''d love to see a fix in either the rails activerecord component or in
puppet.  Seeing that older puppet versions worked with the rails stack
in EPEL, it seems like it should be possible to make that work again
with some changes to puppet.

-- 
Todd        OpenPGP -> KeyID: 0xBEAF0CE3 | URL: www.pobox.com/~tmz/pgp
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It is strangely absurd to suppose that a million human beings
collected together are not under the same moral laws which bind them
separately.
    -- Thomas Jefferson

http://projects.puppetlabs.com/issues/3238 is the issue I was thinking of,
but 3693 is probably a duplicate of that.

These bugs have been open a really long time. Outside of lowering the MySQL
timeout value, are there any workarounds for this problem? I can''t
really
understand how Zynga can be managing as many hosts via Puppet as they do
without being affected by a bug like this, unless they''re not using
stored
configs (which would surprise me).

--
Nathan Clemons
http://www.livemocha.com
The worlds largest online language learning community



On Tue, May 24, 2011 at 2:42 PM, Todd Zullinger <tmz@pobox.com> wrote:
> treydock wrote:
> > What is the current issue number?  The only one I could find that
> > mentions the same problem is this ,
> http://projects.puppetlabs.com/issues/7203.
>
> https://bugzilla.redhat.com/show_bug.cgi?id=572722 and
> http://projects.puppetlabs.com/issues/3693 are both relevant here.
>
> I''d love to see a fix in either the rails activerecord component
or in
> puppet.  Seeing that older puppet versions worked with the rails stack
> in EPEL, it seems like it should be possible to make that work again
> with some changes to puppet.
>
> --
> Todd        OpenPGP -> KeyID: 0xBEAF0CE3 | URL: www.pobox.com/~tmz/pgp
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> It is strangely absurd to suppose that a million human beings
> collected together are not under the same moral laws which bind them
> separately.
>    -- Thomas Jefferson
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Nathan Clemons wrote:
> http://projects.puppetlabs.com/issues/3238 is the issue I was
> thinking of, but 3693 is probably a duplicate of that.
>
> These bugs have been open a really long time. Outside of lowering
> the MySQL timeout value, are there any workarounds for this problem?
> I can''t really understand how Zynga can be managing as many hosts
> via Puppet as they do without being affected by a bug like this,
> unless they''re not using stored configs (which would surprise me).
Sadly, I don''t know of any decent workarounds. I ended up rolling a
rails-2.3.5 for EL-5 in the production instances I wanted to use
storedconfigs.  I would love to see a workaround or a patch for either
the activerecord or puppet packages.  I know that either way this can
be fixed will get into EPEL shortly, as it''s a very annoying bug that
was introduced with the update from 0.24 to 0.25.

-- 
Todd        OpenPGP -> KeyID: 0xBEAF0CE3 | URL: www.pobox.com/~tmz/pgp
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I figure that if God actually does exist, He''s big enough to
understand an honest difference of opinion.
    -- Isaac Asimov

I don''t know how helpful this would be but for now I think I found a
"workaround" of sorts for my small environment.

I commented out the following in my puppet.conf...

        facts_terminus = inventory_active_record
        dbadapter = mysql
        dbname = puppet_inv
        dbuser = puppet_inv
        dbpassword = cabb0ac9ed925867f93587ccfb147ebf15dec8a3
        dbserver = localhost
        dbsocket = /var/lib/mysql/mysql.sock

I got rid of all that and after 2 days of puppetmaster running I only
see a few entries when doing "lsof | grep puppetmas" rather than 100s.

I''m also not doing stored configs but I think those lines above were
still the problem as it was puppet interfacing with MySQL.

What potential benefit would be lost from not using MySQL for
"facts_terminus" ?

- Trey

On May 24, 7:17 pm, Todd Zullinger <t...@pobox.com>
wrote:
> Nathan Clemons wrote:
> >http://projects.puppetlabs.com/issues/3238is the issue I was
> > thinking of, but 3693 is probably a duplicate of that.
>
> > These bugs have been open a really long time. Outside of lowering
> > the MySQL timeout value, are there any workarounds for this problem?
> > I can''t really understand how Zynga can be managing as many
hosts
> > via Puppet as they do without being affected by a bug like this,
> > unless they''re not using stored configs (which would surprise
me).
>
> Sadly, I don''t know of any decent workarounds. I ended up rolling
a
> rails-2.3.5 for EL-5 in the production instances I wanted to use
> storedconfigs.  I would love to see a workaround or a patch for either
> the activerecord or puppet packages.  I know that either way this can
> be fixed will get into EPEL shortly, as it''s a very annoying bug
that
> was introduced with the update from 0.24 to 0.25.
>
> --
> Todd        OpenPGP -> KeyID: 0xBEAF0CE3 | URL:www.pobox.com/~tmz/pgp
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> I figure that if God actually does exist, He''s big enough to
> understand an honest difference of opinion.
>     -- Isaac Asimov
>
>  application_pgp-signature_part
> < 1KViewDownload
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I''ve noticed the same thing with our puppetmaster server. 

We''re still in testing so dramatic changes to puppet.conf are not a 
problem. But commenting lines out doesn''t fix the problem, it just
turns on
the feature that was being broken by too many files being open.

Trey: We''ve been testing on 12 nodes and get this error after about two
days as well. Wondering if this could still be an issue with 100s of nodes. 

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/_cfuA5bPu0QJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.