I''m still tinkering, moving over code from CFE to Puppet in a test environment. One item I did with CfE was manage services. If something wasn''t running, it was restarted. Puppet''s service integration makes this easy. The flip side, is how do you guarantee Puppet''s running? I pulled some tricks with CfE to ensure it was always running. I made CfE a direct child of init, set to respawn in a nonstandard runlevel. Last thing called in the normal boot sequence was telinit to the custom runlevel, so CfE didn''t try to restart things while they were still coming up. I called cfengine separately from a crontab to check on the main CfE process, and to ensure updates occurred. I also shot the main CfE process daily from a cron job. Can you tell I had one hang on me at one point? ;] I asked briefly regarding this on IRC, and I was told to look at daemontools or monit. I''ve reviewed these before, and I don''t want to manage services with two daemons. Either Puppet''s responsible for restarting services when they are down, or I configure Monit (or similar) to do so. My goal is to keep as much as possible in Puppet. This is made more complex by having two daemons with dependencies, puppetmasterd and puppetd. Since I''m using VC to distribute manifests, I could just cron that. I could also have cron trigger a simplistic check/restart one-liner for both puppet processes. Then I could make cron a child of init. Sounds like a real hack to me. I think it''d be more appropriate to make puppetd a child of init, to ensure it is running and allow it to monitor the remaining services. Any comments? ------------------------------------------------------------------ Russell Adams RLAdams@AdamsInfoServ.com PGP Key ID: 0x1160DCB3 http://www.adamsinfoserv.com/ Fingerprint: 1723 D8CA 4280 1EC9 557F 66E8 1154 E018 1160 DCB3
Hi Russel, 2007/10/1, Russell Adams <RLAdams@adamsinfoserv.com>:> > I''m still tinkering, moving over code from CFE to Puppet in a test > environment. > > One item I did with CfE was manage services. If something wasn''t > running, it was restarted. Puppet''s service integration makes this > easy. > > The flip side, is how do you guarantee Puppet''s running?[...] I think it''d be more appropriate to make puppetd a child of init, to> ensure it is running and allow it to monitor the remaining services. > > Any comments? >I usually make some "while [ 1 ]; do ... done" script to check every N seconds for the puppet service to be up and running, restarting it otherwise. This script is run from init, so that puppet runs normally as installed by default. Regards -- Lucas Di Pentima - http://lucas.di-pentima.com.ar GnuPG Public Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x6AA54FC9 Key fingerprint = BD3B 08C4 661A 8C3B 1855 740C 8F98 3FCF 6AA5 4FC9 _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
Russell Adams wrote:> One item I did with CfE was manage services. If something wasn''t > running, it was restarted. Puppet''s service integration makes this > easy. > > The flip side, is how do you guarantee Puppet''s running?This may not be up to your complete set of requirements, but it''s a start -- puppet ensures cron is running, and cron ensures puppetd and/or puppetmasterd are running: http://reductivelabs.com/trac/puppet/wiki/mutualRestartRecipe -- Mike Renfro / R&D Engineer, Center for Manufacturing Research, 931 372-3601 / Tennessee Technological University
On Mon, Oct 01, 2007 at 02:03:44PM -0500, Mike Renfro wrote:> This may not be up to your complete set of requirements, but it''s a > start -- puppet ensures cron is running, and cron ensures puppetd and/or > puppetmasterd are running: > > http://reductivelabs.com/trac/puppet/wiki/mutualRestartRecipe > > -- > Mike Renfro / R&D Engineer, Center for Manufacturing Research, > 931 372-3601 / Tennessee Technological UniversityThats a great start! The only change I would make is that cron should be a child of init, leaving little point in puppet ensuring cron is running. Instead of two processes that are mutually dependent, you have a clear chain from the kernel. If init quits respawning things, there''s no hope. ;] Thanks! ------------------------------------------------------------------ Russell Adams RLAdams@AdamsInfoServ.com PGP Key ID: 0x1160DCB3 http://www.adamsinfoserv.com/ Fingerprint: 1723 D8CA 4280 1EC9 557F 66E8 1154 E018 1160 DCB3
On Mon, Oct 01, 2007 at 11:44:59AM -0500, Russell Adams wrote:> I think it''d be more appropriate to make puppetd a child of init, to > ensure it is running and allow it to monitor the remaining services.Running Puppet from init didn''t work well for us on Fedora Core GNU/Linux, puppetd did something when it started that would cause init to lose it, so init would continually try to restart puppetd. We never investigated the cause in detail, choosing crond instead. Alex
On 01/10/2007, Russell Adams <RLAdams@adamsinfoserv.com> wrote:> One item I did with CfE was manage services. If something wasn''t > running, it was restarted. Puppet''s service integration makes this > easy. > > The flip side, is how do you guarantee Puppet''s running?You don''t mention your OS. On Solaris, the blastwave package of puppet installs an smf service and, unless told to do otherwise, will enable the puppetd service. Solaris smf will restart puppetd as required. I would suggest, if you''re not using Solaris, to consider using those frameworks provided by your OS vendor to do the same: launchd on MacOSX; upstart on Ubuntu etc. If you''re not using an OS that provides restart features consider upgrading to one that does. ;)> I asked briefly regarding this on IRC, and I was told to look at > daemontools or monit. I''ve reviewed these before, and I don''t want to > manage services with two daemons.Well, puppet is a sensible choice for managing all your services except one: puppetd. There isn''t really any getting around this. Daemontools (and I assume monit) is designed and tested for the express purpose of keeping things running, and has thousands of happy installs around the world. I don''t think you should reinvent the wheel if you don''t have to. Gary -- Gary Law
On Mon, Oct 01, 2007 at 11:30:34PM +0100, Gary Law wrote:> You don''t mention your OS.Debian Etch. ;]> > On Solaris, the blastwave package of puppet installs an smf service > and, unless told to do otherwise, will enable the puppetd service. > Solaris smf will restart puppetd as required.AIX has the srcmstr subsystem, same deal.> > I would suggest, if you''re not using Solaris, to consider using those > frameworks provided by your OS vendor to do the same: launchd on > MacOSX; upstart on Ubuntu etc. If you''re not using an OS that provides > restart features consider upgrading to one that does. ;)I''ve actually considered moving to Ubuntu before, but I like Debian''s tiny footprint.> Well, puppet is a sensible choice for managing all your services > except one: puppetd. There isn''t really any getting around this. > Daemontools (and I assume monit) is designed and tested for the > express purpose of keeping things running, and has thousands of happy > installs around the world. I don''t think you should reinvent the > wheel if you don''t have to.I agree it would be reinventing the wheel if I did multiple services. I''m just trying to patch the exception, puppetd, without involving another program or service to maintain. Slapping in a cron job or making puppetd a child of init is the quick answer, that would be incorrect for managing multiple services. Then again, one thing I''m wondering is how often puppetd applies the configuration. I want thing restarted within a few minutes (1-5) window, and I''m wondering how granular puppet could get. Thanks for the input. ------------------------------------------------------------------ Russell Adams RLAdams@AdamsInfoServ.com PGP Key ID: 0x1160DCB3 http://www.adamsinfoserv.com/ Fingerprint: 1723 D8CA 4280 1EC9 557F 66E8 1154 E018 1160 DCB3
On Mon, Oct 01, 2007 at 05:56:03PM -0500, Russell Adams wrote:> On Mon, Oct 01, 2007 at 11:30:34PM +0100, Gary Law wrote: > > I would suggest, if you''re not using Solaris, to consider using those > > frameworks provided by your OS vendor to do the same: launchd on > > MacOSX; upstart on Ubuntu etc. If you''re not using an OS that provides > > restart features consider upgrading to one that does. ;) > > I''ve actually considered moving to Ubuntu before, but I like Debian''s > tiny footprint.I''d use something *other* than Upstart, myself, for simple service monitoring, since that''s not really what it''s there for.> > Well, puppet is a sensible choice for managing all your services > > except one: puppetd. There isn''t really any getting around this. > > Daemontools (and I assume monit) is designed and tested for the > > express purpose of keeping things running, and has thousands of happy > > installs around the world. I don''t think you should reinvent the > > wheel if you don''t have to. > > Then again, one thing I''m wondering is how often puppetd applies the > configuration. I want thing restarted within a few minutes (1-5) > window, and I''m wondering how granular puppet could get.I''ve run puppet with a --runinterval of 10 seconds before for testing purposes, but I don''t think that would be real healthy for the Puppetmaster in the real world. Really, though, what you want is real site-wide monitoring system like Nagios checking all your services (including Puppet) to make sure they''re still healthy. A pair of Nagios boxes (at different sites) checking each other and reporting via SMS works nicely to minimise that SPOF feeling. - Matt