Puppet users - Apr 2009 - Should we really expect puppetd to die randomly? (Was: Puppet watching puppet)

2009/4/6 Mike Renfro <renfro@tntech.edu>:
> I''d normally expect that to work, but I just have puppet keep cron
> running, and have a periodic cron job that checks if puppet has died,
> and if so, restarts it:
Interesting, but why would you expect Puppet to die? Would you expect
Apache, Nginx or worse MySQL to die randomly like this?

I''m a Puppet user since nearly two years now, and in the big picture
of my web servers I find that puppetd is not the most reliable piece
of software.  It dies every day, and my colleagues complain about this
regularly because their installed packages are not uptodate as they
expect.  So I have to start it again and again on all machines.  Is
Puppetd dying because of network problems?  I believe so, but I think
it should be fixed instead of finding creative ways to keep puppetd
running, especially since I request it to run every 5 minutes as a
daemon.

Is anyone using puppetd in a WAN setup with default Webrick server
successfully?  Shall I switch the HTTP server to Mongrel to gain
reliability?  I''ll test this setup.  But if puppetd fails on the
client side, I''m not certain that changing the server''s HTTP
server
would actually prevent the client to fail at all...

I''m a bit eager with this, and I''m really looking forward to
find a
solution, community-wise.  Your comments are welcome.
-- 
Jean-Baptiste Quenot
http://jbq.caraldi.com/

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

We are using puppetd though cron, and it seems to me much more reliable and
much less resource hungry..

if you have more than a few clients, switch to mongrel/passenger.
WAN is not a real issue if its stable, of course it takes longer runs but...

cheers,
Ohad

On Tue, Apr 21, 2009 at 3:19 PM, Jean-Baptiste Quenot
<caraldi@gmail.com>wrote:
>
> 2009/4/6 Mike Renfro <renfro@tntech.edu>:
>
> > I''d normally expect that to work, but I just have puppet keep
cron
> > running, and have a periodic cron job that checks if puppet has died,
> > and if so, restarts it:
>
> Interesting, but why would you expect Puppet to die? Would you expect
> Apache, Nginx or worse MySQL to die randomly like this?
>
> I''m a Puppet user since nearly two years now, and in the big
picture
> of my web servers I find that puppetd is not the most reliable piece
> of software.  It dies every day, and my colleagues complain about this
> regularly because their installed packages are not uptodate as they
> expect.  So I have to start it again and again on all machines.  Is
> Puppetd dying because of network problems?  I believe so, but I think
> it should be fixed instead of finding creative ways to keep puppetd
> running, especially since I request it to run every 5 minutes as a
> daemon.
>
> Is anyone using puppetd in a WAN setup with default Webrick server
> successfully?  Shall I switch the HTTP server to Mongrel to gain
> reliability?  I''ll test this setup.  But if puppetd fails on the
> client side, I''m not certain that changing the server''s
HTTP server
> would actually prevent the client to fail at all...
>
> I''m a bit eager with this, and I''m really looking forward
to find a
> solution, community-wise.  Your comments are welcome.
> --
> Jean-Baptiste Quenot
> http://jbq.caraldi.com/
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

I haven''t had any problems with the client stability.

But, I''m also not using the ''file'' type to copy files
over the
''puppet://'' protocol.

It might be related, but it might not.

Are you hitting the OOM killer?  It sounds like you might be.

I *would* expect Apache and MySQL to die like this (and I''ve seen it)
if they''re hitting resource limits and the OS decides that they need
to die.

The last time I saw it with Apache was due to a badly written plugin
that ate all of the memory in the system which caused Apache to be
nuked.

Trevor

On Tue, Apr 21, 2009 at 03:19, Jean-Baptiste Quenot <caraldi@gmail.com>
wrote:
>
> 2009/4/6 Mike Renfro <renfro@tntech.edu>:
>
>> I''d normally expect that to work, but I just have puppet keep
cron
>> running, and have a periodic cron job that checks if puppet has died,
>> and if so, restarts it:
>
> Interesting, but why would you expect Puppet to die? Would you expect
> Apache, Nginx or worse MySQL to die randomly like this?
>
> I''m a Puppet user since nearly two years now, and in the big
picture
> of my web servers I find that puppetd is not the most reliable piece
> of software.  It dies every day, and my colleagues complain about this
> regularly because their installed packages are not uptodate as they
> expect.  So I have to start it again and again on all machines.  Is
> Puppetd dying because of network problems?  I believe so, but I think
> it should be fixed instead of finding creative ways to keep puppetd
> running, especially since I request it to run every 5 minutes as a
> daemon.
>
> Is anyone using puppetd in a WAN setup with default Webrick server
> successfully?  Shall I switch the HTTP server to Mongrel to gain
> reliability?  I''ll test this setup.  But if puppetd fails on the
> client side, I''m not certain that changing the server''s
HTTP server
> would actually prevent the client to fail at all...
>
> I''m a bit eager with this, and I''m really looking forward
to find a
> solution, community-wise.  Your comments are welcome.
> --
> Jean-Baptiste Quenot
> http://jbq.caraldi.com/
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

We had this problem when we hit the scaling wall running webrick. We then
moved our puppetmaster to a beefy server (dual quadcore with 16 GB ram).
Here we run six puppetmasterds with mongrel and stored config. On the same
server we run nginx. With this setup we can loop over a lot of servers
restarting puppet client without any trouble at all. I tried doing more than
100 at the time without any trouble. So it certainly performs.

After setting this up the problem with dead puppet clients seemed to vanish.
I have no idea how many servers you are running, but when pulling every
fifth minute they sure should get some traffic. You will hit the scaling
wall earlier.  If you have 50 servers you need a setup that other people
would use for a two/three hundred server setup. Are you really sure you need
to pull that often?



2009/4/21 Ohad Levy <ohadlevy@gmail.com>
> We are using puppetd though cron, and it seems to me much more reliable and
> much less resource hungry..
>
> if you have more than a few clients, switch to mongrel/passenger.
> WAN is not a real issue if its stable, of course it takes longer runs
> but...
>
> cheers,
> Ohad
>
>
> On Tue, Apr 21, 2009 at 3:19 PM, Jean-Baptiste Quenot
<caraldi@gmail.com>wrote:
>
>>
>> 2009/4/6 Mike Renfro <renfro@tntech.edu>:
>>
>> > I''d normally expect that to work, but I just have puppet
keep cron
>> > running, and have a periodic cron job that checks if puppet has
died,
>> > and if so, restarts it:
>>
>> Interesting, but why would you expect Puppet to die? Would you expect
>> Apache, Nginx or worse MySQL to die randomly like this?
>>
>> I''m a Puppet user since nearly two years now, and in the big
picture
>> of my web servers I find that puppetd is not the most reliable piece
>> of software.  It dies every day, and my colleagues complain about this
>> regularly because their installed packages are not uptodate as they
>> expect.  So I have to start it again and again on all machines.  Is
>> Puppetd dying because of network problems?  I believe so, but I think
>> it should be fixed instead of finding creative ways to keep puppetd
>> running, especially since I request it to run every 5 minutes as a
>> daemon.
>>
>> Is anyone using puppetd in a WAN setup with default Webrick server
>> successfully?  Shall I switch the HTTP server to Mongrel to gain
>> reliability?  I''ll test this setup.  But if puppetd fails on
the
>> client side, I''m not certain that changing the
server''s HTTP server
>> would actually prevent the client to fail at all...
>>
>> I''m a bit eager with this, and I''m really looking
forward to find a
>> solution, community-wise.  Your comments are welcome.
>> --
>> Jean-Baptiste Quenot
>> http://jbq.caraldi.com/
>>
>>
>>
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Ohad, I remember you were saiyng something about running puppet from
inetd - can you share some info? I really think that cron+inetd
combination would suit my company better than puppetd (which is
restarted every hour via cron)

/br
Stanislaw

On Apr 21, 10:03 am, Ohad Levy <ohadl...@gmail.com>
wrote:
> We are using puppetd though cron, and it seems to me much more reliable and
> much less resource hungry..
>
> if you have more than a few clients, switch to mongrel/passenger.
> WAN is not a real issue if its stable, of course it takes longer runs
but...
>
> cheers,
> Ohad
>
> On Tue, Apr 21, 2009 at 3:19 PM, Jean-Baptiste Quenot
<cara...@gmail.com>wrote:
>
>
>
> > 2009/4/6 Mike Renfro <ren...@tntech.edu>:
>
> > > I''d normally expect that to work, but I just have puppet
keep cron
> > > running, and have a periodic cron job that checks if puppet has
died,
> > > and if so, restarts it:
>
> > Interesting, but why would you expect Puppet to die? Would you expect
> > Apache, Nginx or worse MySQL to die randomly like this?
>
> > I''m a Puppet user since nearly two years now, and in the big
picture
> > of my web servers I find that puppetd is not the most reliable piece
> > of software.  It dies every day, and my colleagues complain about this
> > regularly because their installed packages are not uptodate as they
> > expect.  So I have to start it again and again on all machines.  Is
> > Puppetd dying because of network problems?  I believe so, but I think
> > it should be fixed instead of finding creative ways to keep puppetd
> > running, especially since I request it to run every 5 minutes as a
> > daemon.
>
> > Is anyone using puppetd in a WAN setup with default Webrick server
> > successfully?  Shall I switch the HTTP server to Mongrel to gain
> > reliability?  I''ll test this setup.  But if puppetd fails on
the
> > client side, I''m not certain that changing the
server''s HTTP server
> > would actually prevent the client to fail at all...
>
> > I''m a bit eager with this, and I''m really looking
forward to find a
> > solution, community-wise.  Your comments are welcome.
> > --
> > Jean-Baptiste Quenot
> >http://jbq.caraldi.com/
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Jean-Baptiste Quenot wrote:
> 2009/4/6 Mike Renfro <renfro@tntech.edu>:
> 
>> I''d normally expect that to work, but I just have puppet keep
cron
>> running, and have a periodic cron job that checks if puppet has died,
>> and if so, restarts it:
> 
> Interesting, but why would you expect Puppet to die? Would you expect
> Apache, Nginx or worse MySQL to die randomly like this?
At the time I put together that recipe, I didn''t particularly expect 
puppet to die randomly, but I knew there was a similar practice for 
cfengine users. And though I wouldn''t *expect* any of those other 
services to die randomly, I''d certainly monitor all of them with Nagios
and write puppet manifests to restart them if they''re found to be not 
running.

I have had some puppetd''s die and get restarted from cron. On the few 
dozen systems I have with puppetd, I don''t think it happens even once 
per day for the entire group.

-- 
Mike Renfro  / R&D Engineer, Center for Manufacturing Research,
931 372-3601 / Tennessee Technological University -- renfro@tntech.edu

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Tue, Apr 21, 2009 at 6:56 AM, Mike Renfro <renfro@tntech.edu>
wrote:
>
> Jean-Baptiste Quenot wrote:
>> 2009/4/6 Mike Renfro <renfro@tntech.edu>:
>>
>>> I''d normally expect that to work, but I just have puppet
keep cron
>>> running, and have a periodic cron job that checks if puppet has
died,
>>> and if so, restarts it:
>>
>> Interesting, but why would you expect Puppet to die? Would you expect
>> Apache, Nginx or worse MySQL to die randomly like this?
Why not? Isn''t that why we have sitters and puppet ensuring services
are running? :)

my 2c is that we had persistent performance and reliability problems
with puppetd as a daemon, and switched to cron/launchd jobs that
simply run puppetd --onetime --no-daemonize instead.


>
> At the time I put together that recipe, I didn''t particularly
expect
> puppet to die randomly, but I knew there was a similar practice for
> cfengine users. And though I wouldn''t *expect* any of those other
> services to die randomly, I''d certainly monitor all of them with
Nagios
> and write puppet manifests to restart them if they''re found to be
not
> running.
>
> I have had some puppetd''s die and get restarted from cron. On the
few
> dozen systems I have with puppetd, I don''t think it happens even
once
> per day for the entire group.
>
> --
> Mike Renfro  / R&D Engineer, Center for Manufacturing Research,
> 931 372-3601 / Tennessee Technological University -- renfro@tntech.edu
>
> >
>


-- 
Nigel Kersten
nigelk@google.com
System Administrator
Google, Inc.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

2009/4/21 Trevor Vaughan <peiriannydd@gmail.com>:
>
> I haven''t had any problems with the client stability.
>
> But, I''m also not using the ''file'' type to copy
files over the
> ''puppet://'' protocol.
I''m using the puppet:// protocol extensively yes.  As I noticed it was
very slow especially with recurse=>true, I tend to put my files in
Debian packages when possible.
> Are you hitting the OOM killer?  It sounds like you might be.
I don''t think so because there is nothing in syslog.  The last message
from puppetd is "Starting catalog run".  And then it dies.
-- 
Jean-Baptiste Quenot
http://jbq.caraldi.com/

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

2009/4/21 Bjørn Dyre Dyresen <bjorn@dyresen.net>:
> We had this problem when we hit the scaling wall running webrick. We then
> moved our puppetmaster to a beefy server (dual quadcore with 16 GB ram).
> Here we run six puppetmasterds with mongrel and stored config. On the same
> server we run nginx. With this setup we can loop over a lot of servers
> restarting puppet client without any trouble at all. I tried doing more
than
> 100 at the time without any trouble. So it certainly performs.
>
> After setting this up the problem with dead puppet clients seemed to
vanish.
> I have no idea how many servers you are running, but when pulling every
> fifth minute they sure should get some traffic. You will hit the scaling
> wall earlier.  If you have 50 servers you need a setup that other people
> would use for a two/three hundred server setup. Are you really sure you
need
> to pull that often?
If scaling was involved I would expect Puppetd to be slow, not to die.
 I have about 12 servers that hit a single puppetmasterd through the
Internet, with a lot of puppet:// resources, and every 5 minutes.
Anyway, I''ll try that setup if it worked for you, especially as you
mention that "dead puppet clients" disappeared.

Thanks!
-- 
Jean-Baptiste Quenot
http://jbq.caraldi.com/

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Tue, Apr 21, 2009 at 05:30:54PM +0200, Jean-Baptiste Quenot
wrote:
> 
> > Are you hitting the OOM killer?  It sounds like you might be.
> 
> I don''t think so because there is nothing in syslog.  The last
> message from puppetd is "Starting catalog run".  And then it
dies.
The OOM killer isn''t a message from puppetd, and probably
isn''t even
in syslog; it''s a kernel message.  On my box I''d expect to see
it in
/var/log/kern.log

-Robin

-- 
They say:  "The first AIs will be built by the military as weapons."
And I''m  thinking:  "Does it even occur to you to try for
something
other  than  the default  outcome?"  See http://shrunklink.com/cdiz
http://www.digitalkingdom.org/~rlpowell/ *** http://www.lojban.org/

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Apr 21, 2009, at 2:19 AM, Jean-Baptiste Quenot wrote:
>
> 2009/4/6 Mike Renfro <renfro@tntech.edu>:
>
>> I''d normally expect that to work, but I just have puppet keep
cron
>> running, and have a periodic cron job that checks if puppet has died,
>> and if so, restarts it:
>
> Interesting, but why would you expect Puppet to die? Would you expect
> Apache, Nginx or worse MySQL to die randomly like this?
>
> I''m a Puppet user since nearly two years now, and in the big
picture
> of my web servers I find that puppetd is not the most reliable piece
> of software.  It dies every day, and my colleagues complain about this
> regularly because their installed packages are not uptodate as they
> expect.  So I have to start it again and again on all machines.  Is
> Puppetd dying because of network problems?  I believe so, but I think
> it should be fixed instead of finding creative ways to keep puppetd
> running, especially since I request it to run every 5 minutes as a
> daemon.
I agree, these problems shouldn''t exist, and when they do they should  
be fixed.

The problem is that we can''t seem to find anyone who has the time or  
interest in debugging the problem, and we not-too-surprisingly can''t  
reproduce it ourselves.  Everyone who has the problem just shrugs and  
switches to cron, and pretty much stops responding to our emails  
asking for help fixing the problem.

What we really need is someone who''s having these problems and is  
willing to spend some time investigating.  I expect it''s not that  
complicated, it just needs to be hunted down.

And, for the record, I think this got a lot better in 0.24.8 (or maybe  
0.24.7?).  We found a few cases where errors could propagate far  
enough to kill the client, and I think we''ve eradicated nearly all of  
them.
> Is anyone using puppetd in a WAN setup with default Webrick server
> successfully?  Shall I switch the HTTP server to Mongrel to gain
> reliability?  I''ll test this setup.  But if puppetd fails on the
> client side, I''m not certain that changing the server''s
HTTP server
> would actually prevent the client to fail at all...
We need to add logs or something to clarify this, but you should  
*definitely* switch away from Webrick if you''re having any kind of  
performance or reliability problems.  Or if you''re not.

-- 
Death and taxes are inevitable; at least death doesn''t get worse every
year.
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi,

You can find a "working" daemon here:
http://github.com/ohadlevy/puppet/tree/puppetlisten under the
ext/puppetlisten subdir.

It doesn''t use inetd as ruby socket implementation does not work inet,
I''m
considering rewriting it in c but no real reason / time for now...

search the mailing list for previous usage instruction (but its kinda
straight forward..)

Ohad

On Tue, Apr 21, 2009 at 7:35 PM, Stasheck
<stanislaw.kaminski@gmail.com>wrote:
>
> Ohad, I remember you were saiyng something about running puppet from
> inetd - can you share some info? I really think that cron+inetd
> combination would suit my company better than puppetd (which is
> restarted every hour via cron)
>
> /br
> Stanislaw
>
> On Apr 21, 10:03 am, Ohad Levy <ohadl...@gmail.com> wrote:
> > We are using puppetd though cron, and it seems to me much more
reliable
> and
> > much less resource hungry..
> >
> > if you have more than a few clients, switch to mongrel/passenger.
> > WAN is not a real issue if its stable, of course it takes longer runs
> but...
> >
> > cheers,
> > Ohad
> >
> > On Tue, Apr 21, 2009 at 3:19 PM, Jean-Baptiste Quenot
<cara...@gmail.com
> >wrote:
> >
> >
> >
> > > 2009/4/6 Mike Renfro <ren...@tntech.edu>:
> >
> > > > I''d normally expect that to work, but I just have
puppet keep cron
> > > > running, and have a periodic cron job that checks if puppet
has died,
> > > > and if so, restarts it:
> >
> > > Interesting, but why would you expect Puppet to die? Would you
expect
> > > Apache, Nginx or worse MySQL to die randomly like this?
> >
> > > I''m a Puppet user since nearly two years now, and in the
big picture
> > > of my web servers I find that puppetd is not the most reliable
piece
> > > of software.  It dies every day, and my colleagues complain about
this
> > > regularly because their installed packages are not uptodate as
they
> > > expect.  So I have to start it again and again on all machines. 
Is
> > > Puppetd dying because of network problems?  I believe so, but I
think
> > > it should be fixed instead of finding creative ways to keep
puppetd
> > > running, especially since I request it to run every 5 minutes as
a
> > > daemon.
> >
> > > Is anyone using puppetd in a WAN setup with default Webrick
server
> > > successfully?  Shall I switch the HTTP server to Mongrel to gain
> > > reliability?  I''ll test this setup.  But if puppetd
fails on the
> > > client side, I''m not certain that changing the
server''s HTTP server
> > > would actually prevent the client to fail at all...
> >
> > > I''m a bit eager with this, and I''m really
looking forward to find a
> > > solution, community-wise.  Your comments are welcome.
> > > --
> > > Jean-Baptiste Quenot
> > >http://jbq.caraldi.com/
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Luke,

I was the guy who brought up puppet client fails last June at the SF meetup.

The memory footprint of the puppetd client in idle mode (between
triggers) was enough
of a reason to dump the always-on client, but I also found that they
crashed regularly whenever
there was any difficulty contacting the puppet master.  (too many
concurrent, etc)

I just run my clients in cron like this:
/usr/bin/perl -e "sleep int(rand(60*30));" ; /usr/sbin/puppetd --test
1> /tmp/puppetd.log

I would be more than happy to help debug why the client crashes.
What do you suggest I run it with?  Is there a ruby debugger?

Unfortunately I''ll probably not want to upgrade to the newest puppet
for this
test harness, but I''d be happy to put 0.24.4 up for some crash and
burns.

--joel


On Tue, Apr 21, 2009 at 12:04 PM, Luke Kanies <luke@madstop.com>
wrote:
>
> On Apr 21, 2009, at 2:19 AM, Jean-Baptiste Quenot wrote:
>
>>
>> 2009/4/6 Mike Renfro <renfro@tntech.edu>:
>>
>>> I''d normally expect that to work, but I just have puppet
keep cron
>>> running, and have a periodic cron job that checks if puppet has
died,
>>> and if so, restarts it:
>>
>> Interesting, but why would you expect Puppet to die? Would you expect
>> Apache, Nginx or worse MySQL to die randomly like this?
>>
>> I''m a Puppet user since nearly two years now, and in the big
picture
>> of my web servers I find that puppetd is not the most reliable piece
>> of software.  It dies every day, and my colleagues complain about this
>> regularly because their installed packages are not uptodate as they
>> expect.  So I have to start it again and again on all machines.  Is
>> Puppetd dying because of network problems?  I believe so, but I think
>> it should be fixed instead of finding creative ways to keep puppetd
>> running, especially since I request it to run every 5 minutes as a
>> daemon.
>
> I agree, these problems shouldn''t exist, and when they do they
should
> be fixed.
>
> The problem is that we can''t seem to find anyone who has the time
or
> interest in debugging the problem, and we not-too-surprisingly
can''t
> reproduce it ourselves.  Everyone who has the problem just shrugs and
> switches to cron, and pretty much stops responding to our emails
> asking for help fixing the problem.
>
> What we really need is someone who''s having these problems and is
> willing to spend some time investigating.  I expect it''s not that
> complicated, it just needs to be hunted down.
>
> And, for the record, I think this got a lot better in 0.24.8 (or maybe
> 0.24.7?).  We found a few cases where errors could propagate far
> enough to kill the client, and I think we''ve eradicated nearly all
of
> them.
>
>> Is anyone using puppetd in a WAN setup with default Webrick server
>> successfully?  Shall I switch the HTTP server to Mongrel to gain
>> reliability?  I''ll test this setup.  But if puppetd fails on
the
>> client side, I''m not certain that changing the
server''s HTTP server
>> would actually prevent the client to fail at all...
>
> We need to add logs or something to clarify this, but you should
> *definitely* switch away from Webrick if you''re having any kind of
> performance or reliability problems.  Or if you''re not.
>
> --
> Death and taxes are inevitable; at least death doesn''t get worse
every
> year.
> ---------------------------------------------------------------------
> Luke Kanies | http://reductivelabs.com | http://madstop.com
>
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

2009/4/21 Luke Kanies <luke@madstop.com>:
> And, for the record, I think this got a lot better in 0.24.8 (or maybe
> 0.24.7?).  We found a few cases where errors could propagate far
> enough to kill the client, and I think we''ve eradicated nearly all
of
> them.
I''ll try to switch to 0.24.8, as I''m using 0.24.4 right now. 
Thanks
for your valuable answers.  If the problem persists I''ll be running
the daemons in debug mode to see what''s wrong.
-- 
Jean-Baptiste Quenot
http://jbq.caraldi.com/

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

2009/4/22 Jean-Baptiste Quenot
<caraldi@gmail.com>:
> 2009/4/21 Luke Kanies <luke@madstop.com>:
>
>> And, for the record, I think this got a lot better in 0.24.8 (or maybe
>> 0.24.7?).  We found a few cases where errors could propagate far
>> enough to kill the client, and I think we''ve eradicated nearly
all of
>> them.
>
> I''ll try to switch to 0.24.8, as I''m using 0.24.4 right
now.  Thanks
> for your valuable answers.  If the problem persists I''ll be
running
> the daemons in debug mode to see what''s wrong.
Seems good so far, none of my Puppet clients died since I upgraded to
0.24.8, still with default Webrick.

Good job!  I''m happy :-)
-- 
Jean-Baptiste Quenot
http://jbq.caraldi.com/

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Apr 21, 12:19 am, Jean-Baptiste Quenot <cara...@gmail.com>
wrote:
> 2009/4/6 Mike Renfro <ren...@tntech.edu>:
>
> > I''d normally expect that to work, but I just have puppet keep
cron
> > running, and have a periodic cron job that checks if puppet has died,
> > and if so, restarts it:
>
> Interesting, but why would you expect Puppet to die? Would you expect
> Apache, Nginx or worse MySQL to die randomly like this?
>
> I''m a Puppet user since nearly two years now, and in the big
picture
> of my web servers I find that puppetd is not the most reliable piece
> of software.  It dies every day, and my colleagues complain about this
> regularly because their installed packages are not uptodate as they
> expect.  So I have to start it again and again on all machines.  Is
> Puppetd dying because of network problems?  I believe so, but I think
> it should be fixed instead of finding creative ways to keep puppetd
> running, especially since I request it to run every 5 minutes as a
> daemon.
>
> Is anyone using puppetd in a WAN setup with default Webrick server
> successfully?  Shall I switch the HTTP server to Mongrel to gain
> reliability?  I''ll test this setup.  But if puppetd fails on the
> client side, I''m not certain that changing the server''s
HTTP server
> would actually prevent the client to fail at all...
>
> I''m a bit eager with this, and I''m really looking forward
to find a
> solution, community-wise.  Your comments are welcome.
> --
> Jean-Baptiste Quenothttp://jbq.caraldi.com/
This has already been mentioned in this thread, but I''ll add my $0.02
anyway. I use puppet to manage about 500 servers, and I started having
the same scalability problems everyone else is having with Webrick:
puppetd started randomly dying on clients, IIRC when I reached around
50 nodes. I initially thought about working around it by putting
puppet into cron, I didn''t want to sacrifice some of the features that
puppetd provides (scheduling, ability to monitor puppetd status from a
central location). I was also concerned that puppet might still have
problems pulling files from the fileserver with a Webrick-based
solution. I kicked Webrick to the curb and replaced it with Passenger/
Rack on my puppetmaster, using the instructions at
http://reductivelabs.com/trac/puppet/wiki/UsingPassenger. This dropped
the puppetd mortality rate on my client nodes to nearly zero, and I
get to keep using puppetd on the clients. :)
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Fri, Apr 24, 2009 at 6:25 AM, threetee <threetee@gmail.com>
wrote:
>
> On Apr 21, 12:19 am, Jean-Baptiste Quenot <cara...@gmail.com> wrote:
>> 2009/4/6 Mike Renfro <ren...@tntech.edu>:
>>
>> > I''d normally expect that to work, but I just have puppet
keep cron
>> > running, and have a periodic cron job that checks if puppet has
died,
>> > and if so, restarts it:
>>
>> Interesting, but why would you expect Puppet to die? Would you expect
>> Apache, Nginx or worse MySQL to die randomly like this?
>>
>> I''m a Puppet user since nearly two years now, and in the big
picture
>> of my web servers I find that puppetd is not the most reliable piece
>> of software.  It dies every day, and my colleagues complain about this
>> regularly because their installed packages are not uptodate as they
>> expect.  So I have to start it again and again on all machines.  Is
>> Puppetd dying because of network problems?  I believe so, but I think
>> it should be fixed instead of finding creative ways to keep puppetd
>> running, especially since I request it to run every 5 minutes as a
>> daemon.
>>
>> Is anyone using puppetd in a WAN setup with default Webrick server
>> successfully?  Shall I switch the HTTP server to Mongrel to gain
>> reliability?  I''ll test this setup.  But if puppetd fails on
the
>> client side, I''m not certain that changing the
server''s HTTP server
>> would actually prevent the client to fail at all...
>>
>> I''m a bit eager with this, and I''m really looking
forward to find a
>> solution, community-wise.  Your comments are welcome.
>> --
>> Jean-Baptiste Quenothttp://jbq.caraldi.com/
>
> This has already been mentioned in this thread, but I''ll add my
$0.02
> anyway. I use puppet to manage about 500 servers, and I started having
> the same scalability problems everyone else is having with Webrick:
> puppetd started randomly dying on clients, IIRC when I reached around
> 50 nodes. I initially thought about working around it by putting
> puppet into cron, I didn''t want to sacrifice some of the features
that
> puppetd provides (scheduling, ability to monitor puppetd status from a
> central location). I was also concerned that puppet might still have
> problems pulling files from the fileserver with a Webrick-based
> solution. I kicked Webrick to the curb and replaced it with Passenger/
> Rack on my puppetmaster, using the instructions at
> http://reductivelabs.com/trac/puppet/wiki/UsingPassenger. This dropped
> the puppetd mortality rate on my client nodes to nearly zero, and I
> get to keep using puppetd on the clients. :)
I have some python code that fires off a bunch of simultaneous threads
that mimic puppet clients supplying facts and requesting a compiled
manifest (planning to release this btw), and I''ve been amazed at just
how much better Passenger/Rack performs compared to
Pound/Nginx/Mongrel.



-- 
Nigel Kersten
nigelk@google.com
System Administrator
Google, Inc.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Fri, Apr 24, 2009 at 9:45 PM, Nigel Kersten <nigelk@google.com> wrote:
> ...
>
> manifest (planning to release this btw), and I''ve been amazed at
just
> how much better Passenger/Rack performs compared to
> Pound/Nginx/Mongrel.
>
How much did you tweak the passenger options? (e.g. PassengerMaxRequests,
PassengerPoolIdleTime  etc)

cheers,
Ohad

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Wed, Apr 29, 2009 at 11:29 PM, Ohad Levy <ohadlevy@gmail.com>
wrote:
>
>
> On Fri, Apr 24, 2009 at 9:45 PM, Nigel Kersten <nigelk@google.com>
wrote:
>>
>> ...
>> manifest (planning to release this btw), and I''ve been amazed
at just
>> how much better Passenger/Rack performs compared to
>> Pound/Nginx/Mongrel.
>
> How much did you tweak the passenger options? (e.g. PassengerMaxRequests,
> PassengerPoolIdleTime  etc)

I''m actually looking at this at the moment. I might start a new
thread....
>
> cheers,
> Ohad
>
> >
>


-- 
Nigel Kersten
nigelk@google.com
System Administrator
Google, Inc.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---