Puppet users - Jun 2009 - Puppetmasterd looses contact with the mysql database (using storeconfigs)

Hi All

I''ve just bootstrapped my new, shining puppet setup (replacing an 2,5  
year old, messy one), with (almost) all the bells and whistles: A  
Passenger-based setup with storeconfigs enabled.

For a start I''ve only got three node definitions; two specific ones  
and node default. The default one doesn''t do anything yet, and only  
one of the specific nodes has actual classes enabled.

Now, when I startup mysql and puppetmasterd, the first one or two runs  
of puppetd on the client is fine. Puppetmasterd stores the configs,  
but suddenly (and this is with only one host contacting the  
puppetmaster) the log says:

Could not store configs: Mysql::Error: MySQL server has gone away:  
SELECT * FROM `hosts`   WHERE (`hosts`.`name` =
''web01.mydomain.com'')
LIMIT 1

At first I thought it was MySQL drowning, but I''ve found that a  
restart of puppetmasterd (with /etc/init.d/apache2 restart, as it run  
through Passenger) fixes the problem - but only temporarily. Next time  
a host contacts the puppetmaster, I get the error stated above. :- 
( And it fails until I restart puppetmaster again.

I have the following optimizations in my passenger configuration (cut  
from Nigel Kerstens mails about his Passengerbased setup):

PassengerLogLevel 1
PassengerMaxRequests 3000
PassengerMaxPoolSize 15
PassengerStatThrottleRate 600

I''ve also tried to optimize the MySQL configuration, but as a restart  
of Apache/Passenger solves the problem (for one clientrun), I don''t  
think that the problem lies with MySQL.

Do any of you have a clue of what can be wrong here?

--
Med venlig hilsen/Best regards
Juri Rischel Jensen

Fab:IT ApS
Vesterbrogade 50
DK-1620 København
Tlf: +45 70 202 407
www.fab-it.dk


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi,

First make sure you installed the mysql Ruby Gem (or the correct package 
on your system).

On 5/06/09 11:58, Juri Rischel Jensen wrote:
> Now, when I startup mysql and puppetmasterd, the first one or two runs 
> of puppetd on the client is fine. Puppetmasterd stores the configs, but 
> suddenly (and this is with only one host contacting the puppetmaster) 
> the log says:
> 
> Could not store configs: Mysql::Error: MySQL server has gone away: 
> SELECT * FROM `hosts`   WHERE (`hosts`.`name` =
''web01.mydomain.com'')
>  LIMIT 1
Server has gone away usually means MySQL has crashed. Check your 
mysql.err file or syslog on this system to see what happened.
Which version of MySQL is it?
> At first I thought it was MySQL drowning, but I''ve found that a
restart
> of puppetmasterd (with /etc/init.d/apache2 restart, as it run through 
> Passenger) fixes the problem - but only temporarily. Next time a host 
> contacts the puppetmaster, I get the error stated above. :-( And it 
> fails until I restart puppetmaster again.
Usually MySQL restarts itself after a crash (it runs through a wrapper 
called mysqld_safe usually).

But maybe, the mysql crash leaves the puppetmaster in an unstable state 
which prevents it to connect back to MySQL.
> I have the following optimizations in my passenger configuration (cut 
> from Nigel Kerstens mails about his Passengerbased setup):
> 
> PassengerLogLevel 1
> PassengerMaxRequests 3000
> PassengerMaxPoolSize 15
> PassengerStatThrottleRate 600
> 
> I''ve also tried to optimize the MySQL configuration, but as a
restart of
> Apache/Passenger solves the problem (for one clientrun), I don''t
think
> that the problem lies with MySQL.
> 
> Do any of you have a clue of what can be wrong here?
I''d suggest you run your master outside of passenger to see if you can 
replicate the issue.
Run the master on the console with --no-daemonize --debug --trace.
Then connect your clients.
If it crashes, you can rule out passenger as the culprit.
If it works, then it is a bad interaction between passenger, rails 
and/or MySQL.

Good luck,
-- 
Brice Figureau
My Blog: http://www.masterzen.fr/


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi Brice

Thanks for your answer - it helped me a bit further in the  
troubleshooting. Running puppetmaster outside of passenger gives no  
errors at all. All my hosts check in and puppetmaster stores their  
config. So it seems that there''s a bug when using Passenger together  
with storeconfigs. Maybe it''s the version of rails (2.0.5)
that''s the
culprit? Is it compatible with passenger?

My list of gems:

actionmailer (2.0.5)
actionpack (2.0.5)
activerecord (2.0.5)
activeresource (2.0.5)
activesupport (2.0.5)
cgi_multipart_eof_fix (2.5.0)
daemons (1.0.10)
fastthread (1.0.4, 1.0.1)
gem_plugin (0.2.3)
hoe (1.11.0)
mongrel (1.1.5)
mysql (2.7)
passenger (2.2.2, 2.1.2)
rack (0.4.0)
rails (2.0.5)
rake (0.8.7, 0.8.4)
rubyforge (1.0.3)
rubygems-update (1.3.3)

I''m using v. 0.4.0 of rack as the documentation describes
(http://reductivelabs.com/trac/puppet/wiki/UsingPassenger
). When calling puppetmaster directly on the console, I bypass Rack  
too, don''t I...?

On Jun 6, 2009, at 14:26 , Brice Figureau wrote:
> First make sure you installed the mysql Ruby Gem (or the correct  
> package
> on your system).
>
> On 5/06/09 11:58, Juri Rischel Jensen wrote:
>> Now, when I startup mysql and puppetmasterd, the first one or two  
>> runs
>> of puppetd on the client is fine. Puppetmasterd stores the configs,  
>> but
>> suddenly (and this is with only one host contacting the puppetmaster)
>> the log says:
>>
>> Could not store configs: Mysql::Error: MySQL server has gone away:
>> SELECT * FROM `hosts`   WHERE (`hosts`.`name` =
''web01.mydomain.com'')
>> LIMIT 1
>
> Server has gone away usually means MySQL has crashed. Check your
> mysql.err file or syslog on this system to see what happened.
> Which version of MySQL is it?
>
>> At first I thought it was MySQL drowning, but I''ve found that
a
>> restart
>> of puppetmasterd (with /etc/init.d/apache2 restart, as it run through
>> Passenger) fixes the problem - but only temporarily. Next time a host
>> contacts the puppetmaster, I get the error stated above. :-( And it
>> fails until I restart puppetmaster again.
>
> Usually MySQL restarts itself after a crash (it runs through a wrapper
> called mysqld_safe usually).
>
> But maybe, the mysql crash leaves the puppetmaster in an unstable  
> state
> which prevents it to connect back to MySQL.
>
>> I have the following optimizations in my passenger configuration (cut
>> from Nigel Kerstens mails about his Passengerbased setup):
>>
>> PassengerLogLevel 1
>> PassengerMaxRequests 3000
>> PassengerMaxPoolSize 15
>> PassengerStatThrottleRate 600
>>
>> I''ve also tried to optimize the MySQL configuration, but as a
>> restart of
>> Apache/Passenger solves the problem (for one clientrun), I
don''t
>> think
>> that the problem lies with MySQL.
>>
>> Do any of you have a clue of what can be wrong here?
>
> I''d suggest you run your master outside of passenger to see if you
can
> replicate the issue.
> Run the master on the console with --no-daemonize --debug --trace.
> Then connect your clients.
> If it crashes, you can rule out passenger as the culprit.
> If it works, then it is a bad interaction between passenger, rails
> and/or MySQL.
--
Med venlig hilsen/Best regards
Juri Rischel Jensen

Fab:IT ApS
Vesterbrogade 50
DK-1620 København
Tlf: +45 70 202 407
www.fab-it.dk


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi everyone

Am I the only one that has stumbled upon this bug....?

I''ve tried to raise the loglevel to info in config.ru, but it gives me
no further info, unfortunately. :-(

On Jun 6, 2009, at 23:05 , Juri Rischel Jensen wrote:
> Thanks for your answer - it helped me a bit further in the
> troubleshooting. Running puppetmaster outside of passenger gives no
> errors at all. All my hosts check in and puppetmaster stores their
> config. So it seems that there''s a bug when using Passenger
together
> with storeconfigs. Maybe it''s the version of rails (2.0.5)
that''s the
> culprit? Is it compatible with passenger?
--
Med venlig hilsen/Best regards
Juri Rischel Jensen

Fab:IT ApS
Vesterbrogade 50
DK-1620 København
Tlf: +45 70 202 407
www.fab-it.dk


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi everyone

I''ve now tried to change database to postgresql, and unfortunately: I  
get the same result:

Jun  9 12:03:13 puppet puppetmasterd[31869]: Could not store configs:  
PGError: server closed the connection unexpectedly#012#011This  
probably means the server terminated abnormally#012#011before or while  
processing the request.#012: BEGIN

So something in the combination of Passenger, activerecord,  
puppetmaster, rack and maybe more, does not work. Damn! I really don''t
know how I should proceed with this, as running puppetmasterd outside  
passenger doesn''t produce the error, and raising the errorlevel in  
config.ru doesn''t provide me with more info than the line above.

How can I track down such an error...?

On Jun 6, 2009, at 23:05 , Juri Rischel Jensen wrote:
> Thanks for your answer - it helped me a bit further in the
> troubleshooting. Running puppetmaster outside of passenger gives no
> errors at all. All my hosts check in and puppetmaster stores their
> config. So it seems that there''s a bug when using Passenger
together
> with storeconfigs. Maybe it''s the version of rails (2.0.5)
that''s the
> culprit? Is it compatible with passenger?
--
Med venlig hilsen/Best regards
Juri Rischel Jensen

Fab:IT ApS
Vesterbrogade 50
DK-1620 København
Tlf: +45 70 202 407
www.fab-it.dk


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---