Puppet users - Jun 2012 - HTTP 400 errors when using storeconfigs

As mentioned in my previous thread at
http://groups.google.com/group/puppet-users/browse_thread/thread/d491889c89c1a0e0/2ba280696d1ecf96?lnk=gst&q=matthew+burgess#2ba280696d1ecf96,
I''m seeing intermittent errors on my nodes of the form:

Wed May 30 18:02:24 +0100 2012 Puppet (err): undefined method `>''
for
["2.7.14", "2.7.14"]:Array on node master.domain.com
Wed May 30 18:02:24 +0100 2012 Puppet (err): undefined method `>''
for
["2.7.14", "2.7.14"]:Array on node master.domain.com
Wed May 30 18:02:24 +0100 2012 Puppet (err): undefined method `>''
for
["2.7.14", "2.7.14"]:Array on node master.domain.com
Wed May 30 18:02:24 +0100 2012 Puppet (err): undefined method `>''
for
["2.7.14", "2.7.14"]:Array on node master.domain.com

Upon further investigation I''ve managed to, I think, finger the blame
on storeconfigs.

I''ve set up a new puppet master, completely segregated from the rest
of the puppet infrastructure, so there is a total of
1 puppet master and 1 puppet client in this environment.  An ENC is
configured that currently states that this node should
not have any manifests applied.

With storeconfigs set to true, and pointing to a MySQL database on the
same host, I can reliably trigger the issue using
puppet-load.rb.

As soon as I set storeconfigs to false, and restart httpd (puppet is
being served behind mod_passenger), a puppet-load
run with 100 concurrent clients works with 100% success.

Now, the storeconfigs instructions at
http://projects.puppetlabs.com/projects/1/wiki/Using_Stored_Configuration
mentions the use of a message broker, which was going to be by next
plan of attack, but I guess at the moment I''m just kind of interested
to see if anyone else has hit problems similar to mine and what kind
of scaling they''ve achieved using plain storeconfigs without a message
broker.

Thanks,

Matt.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Jun 13, 2012 at 2:53 PM, Matthew Burgess
<matthew.2.burgess@googlemail.com> wrote:
> As soon as I set storeconfigs to false, and restart httpd (puppet is
> being served behind mod_passenger), a puppet-load
> run with 100 concurrent clients works with 100% success.
Sorry, this should read 20 concurrent clients.  100 concurrent clients
happens to work as well, but
that''s way above the level of scaling we require at present.

In addition, the tests were run after performing a ''puppetd
--test'' to
store the initial config
in the DB.  With no manifests being applied, I''m not expecting any
further writes to the DB during
the test runs.  Is that a reasonable assumption, and therefore is it
also reasonable to assume
that I should be focusing my efforts on mysql read optimisations/tuning?

Regards,

Matt.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I ran into this issue when Active(Record|Model|Support) had a version
higher than 3.0.11. I''m running RHEL6 x64 with Ruby 1.8.7 from RPM and
MySQL gem 2.8.1 built from source, if that rings true for you. Which
versions of the installed gems do you have? And if you run the puppet
master as
puppet master --no-daemonize -vd
does it give any pertinent output for us to look at?

As for data reads/writes, every time a node checks in, if storedconfigs
is on, it will update the facts and some other ancillary information in
the database. I''ve got about 400 nodes in my setup and it''s
still not
very demanding on MySQL even with quite a lot of data going in and out,
so the stock config values should suffice while you spend time tracking
down all the other issues with getting the environment stood up.

Good luck!

-Jared

On 06/13/2012 10:28 AM, Matthew Burgess wrote:
> On Wed, Jun 13, 2012 at 2:53 PM, Matthew Burgess
> <matthew.2.burgess@googlemail.com> wrote:
> 
>> As soon as I set storeconfigs to false, and restart httpd (puppet is
>> being served behind mod_passenger), a puppet-load
>> run with 100 concurrent clients works with 100% success.
> 
> Sorry, this should read 20 concurrent clients.  100 concurrent clients
> happens to work as well, but
> that''s way above the level of scaling we require at present.
> 
> In addition, the tests were run after performing a ''puppetd
--test'' to
> store the initial config
> in the DB.  With no manifests being applied, I''m not expecting any
> further writes to the DB during
> the test runs.  Is that a reasonable assumption, and therefore is it
> also reasonable to assume
> that I should be focusing my efforts on mysql read optimisations/tuning?
> 
> Regards,
> 
> Matt.
> 
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Jun 13, 2012 at 3:58 PM, Jared Ballou <jballou@jballou.com>
wrote:
> I ran into this issue when Active(Record|Model|Support) had a version
> higher than 3.0.11. I''m running RHEL6 x64 with Ruby 1.8.7 from RPM
and
> MySQL gem 2.8.1 built from source, if that rings true for you. Which
> versions of the installed gems do you have? And if you run the puppet
> master as
> puppet master --no-daemonize -vd
> does it give any pertinent output for us to look at?
Thanks for the quick response, Jared.  It all sounds far too familiar :-)

I''m running a RHEL 5.8 x86_64 server here.  Ruby is 1.8.7-299 from
aegisco''s repository and rubygems-1.8.5 from the same location.

I did try with activerecord-3.2.6 but quickly ran into
http://projects.puppetlabs.com/issues/9290 so downgraded to 3.0.15.
Based on your message I''ve tried both 3.0.11 and 3.0.10 and still have
the same results.

All gems are installed via ''gem install --local --verbose <gem
name>''

$ gem list

*** LOCAL GEMS ***

activemodel (3.0.10)
activerecord (3.0.10)
activesupport (3.0.10)
addressable (2.2.8)
arel (2.0.10)
builder (2.1.2)
cookiejar (0.3.0)
daemon_controller (1.0.0)
em-http-request (1.0.2)
em-socksify (0.2.0)
eventmachine (1.0.0.beta.4)
fastthread (1.0.7)
http_parser.rb (0.5.3)
i18n (0.5.0)
json (1.7.3)
multi_json (1.3.6)
mysql (2.8.1)
passenger (3.0.12)
rack (1.4.1)
rake (0.9.2.2)
tzinfo (0.3.,33)

In terms of output from the puppet master, I get nothing of note with
those options.  If I try with:

# puppet master --no-daemonize --debug --trace --verbose --logdest
/tmp/puppetmaster.log

I get the results in the attached log.
> As for data reads/writes, every time a node checks in, if storedconfigs
> is on, it will update the facts and some other ancillary information in
> the database. I''ve got about 400 nodes in my setup and
it''s still not
> very demanding on MySQL even with quite a lot of data going in and out,
> so the stock config values should suffice while you spend time tracking
> down all the other issues with getting the environment stood up.
That''s all useful info, thanks!

Regards,

Matt.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

>On Wed, Jun 13, 2012 at 3:58 PM, Jared Ballou 
><jballou@jballou.com> wrote:
>> I ran into this issue when Active(Record|Model|Support) had a version
>> higher than 3.0.11. I''m running RHEL6 x64 with Ruby 1.8.7 
>from RPM and
>> MySQL gem 2.8.1 built from source, if that rings true for you. Which
>> versions of the installed gems do you have? And if you run the puppet
>> master as
>> puppet master --no-daemonize -vd
>> does it give any pertinent output for us to look at?
>
>Thanks for the quick response, Jared.  It all sounds far too 
>familiar :-)
>
>I''m running a RHEL 5.8 x86_64 server here.  Ruby is 1.8.7-299 from
>aegisco''s repository and rubygems-1.8.5 from the same location.
>
>I did try with activerecord-3.2.6 but quickly ran into
>http://projects.puppetlabs.com/issues/9290 so downgraded to 3.0.15.
>Based on your message I''ve tried both 3.0.11 and 3.0.10 and still
have
>the same results.
>
Are you perhaps running into a max db connection issue (this should be recorded
in logs).

We set something like wait_timeout=60 in our mysql my.cnf to kill
"idle" connections that don''t seem to get properly reused.


This email communication and any files transmitted with it may contain
confidential and or proprietary information and is provided for the use of the
intended recipient only. Any review, retransmission or dissemination of this
information by anyone other than the intended recipient is prohibited. If you
receive this email in error, please contact the sender and delete this
communication and any copies immediately. Thank you.

http://www.encana.com

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Jun 13, 2012 at 4:54 PM, Kinzel, David <David.Kinzel@encana.com>
wrote:
> Are you perhaps running into a max db connection issue (this should be
recorded in logs).
>
> We set something like wait_timeout=60 in our mysql my.cnf to kill
"idle" connections that don''t seem to get properly reused.
I don''t think so.  I can reproduce the failure by restarting the
puppet master, then running puppet-load.rb with ''--concurrency
2''
*twice*.  The first time will always succeed; both clients complete
successfully.  The 2nd and subsequent invocations will always fail
with the 400 error.

If anyone here is able to share their production apache vhost configs
including any passenger directives, I''d be grateful, just so I can
compare with mine that were cobbled together via various online
sources.

Regards,

Matt.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

So, continuing this saga, I decided yesterday to try and make use of
puppetdb as it''s supposed to have performance improvements over and
above storeconfigs.

My first change was to move from a MySQL to PostgreSQL database behind
storeconfigs.  That still showed the issues I''m seeing.  With
PostgreSQL in place, I then configured puppetdb, but I''m still seeing
these issues with concurrent users.

At this point, is it worth raising an issue at puppetlabs.com?  I have
the failing environment on a VM now, so if someone wants a copy/or
login details to it, I''m happy to provide them if it helps figure out
what''s going on.

Thanks,

Matt.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Matt,

I think it would be worth opening a ticket for this ( 
http://projects.puppetlabs.com/projects/puppet/issues/new ).  The fact that 
you are seeing it with both storeconfigs/mysql and puppetdb/postgres leads 
me to believe that the issue is more likely in puppet core than in either 
of the catalog storage backends.

It would also be useful if you could run the puppet master with the 
"--trace" command-line option (and possibly with --debug as well) and 
capture that log file.

It sounds like you are running the master via apache/passenger, correct?

A manifest file that could reproduce your setup would be ideal to attach to 
the ticket, but the VM will be very useful if it''s not easy to put
together
such a manifest.  How large is the VM image?  It might be too big to attach 
directly to the ticket, but if you mention it in the ticket we can figure 
out the best way to get the file from you.

Thanks!
Chris

On Thursday, June 14, 2012 4:29:17 AM UTC-7, Matthew Burgess
wrote:
>
> So, continuing this saga, I decided yesterday to try and make use of 
> puppetdb as it''s supposed to have performance improvements over
and
> above storeconfigs. 
>
> My first change was to move from a MySQL to PostgreSQL database behind 
> storeconfigs.  That still showed the issues I''m seeing.  With 
> PostgreSQL in place, I then configured puppetdb, but I''m still
seeing
> these issues with concurrent users. 
>
> At this point, is it worth raising an issue at puppetlabs.com?  I have 
> the failing environment on a VM now, so if someone wants a copy/or 
> login details to it, I''m happy to provide them if it helps figure
out
> what''s going on. 
>
> Thanks, 
>
> Matt. 
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/NdraCH3guVAJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thu, Jun 14, 2012 at 1:31 PM, Chris Price <chris@puppetlabs.com>
wrote:
> Hi Matt,
>
> I think it would be worth opening a ticket for this
> ( http://projects.puppetlabs.com/projects/puppet/issues/new ).  The fact
> that you are seeing it with both storeconfigs/mysql and puppetdb/postgres
> leads me to believe that the issue is more likely in puppet core than in
> either of the catalog storage backends.
Thanks Chris.  I''ll open a ticket when I can step away from the
firefighting for a moment :-)

In the meantime, just to clarify, I''m seeing this in
storeconfigs/mysql, storeconfigs/postgres
and puppetdb/postgres.  As previously mentioned though, I only see
this with storeconfigs=true
and never with storeconfigs=false.  So maybe it is in puppet core
somewhere, but only in an
area that storeconfigs touches?
> It would also be useful if you could run the puppet master with the
> "--trace" command-line option (and possibly with --debug as well)
and
> capture that log file.
OK, will add that log to the ticket.
> It sounds like you are running the master via apache/passenger, correct?
Yes, but I think it''s reproducible outside of apache/passenger too,
though it''s a bit hard to
tickle there due to Webrick crapping out with a different issue when
relatively lightly loaded.
> A manifest file that could reproduce your setup would be ideal to attach to
> the ticket, but the VM will be very useful if it''s not easy to put
together
> such a manifest.
That''s the thing, this is being triggered on a puppetmaster with *no*
manifests; determined
after originally thinking that one of my manifests was to blame, I got
rid of the lot and could
still trigger it!
> How large is the VM image?  It might be too big to attach
> directly to the ticket, but if you mention it in the ticket we can figure
> out the best way to get the file from you.
It''ll be in the order of 1GB I think.  As there''s no sensitive
details
in that environment though
I think I''ll attach puppet.conf, auth.conf and the httpd configs too
so that someone can try
to reproduce.  Hopefully that''s enough, otherwise I''ll have
another
look at getting the
VM uploaded or otherwise made available somehow.

Regards,

Matt.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thu, Jun 14, 2012 at 1:31 PM, Chris Price <chris@puppetlabs.com>
wrote:
> Hi Matt,
>
> I think it would be worth opening a ticket for this
> ( http://projects.puppetlabs.com/projects/puppet/issues/new ).
Hi Chris,

This is now #15086. Hopefully there''s enough info on the ticket to
reproduce.

The VM is 1.5GB in size, so may be a little awkward to make available
to anyone working on the ticket, but I''m happy to give anything a try!

Regards,

Matt.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.