All,
I have puppet watching various services on RHEL systems and restarting
them as necessary.
Quite a while back I wrote some custom scripts as wrappers for various
services so that I can have a "custom" running status. These have
worked for quite a few months, but recently my iptables services have
been restarting every time puppet checks in.
Problem:
[me@host ~]$ sudo /usr/sbin/puppetd --test --noop
info: Caching catalog for host@fakedomain
info: Applying configuration version ''1312979662''
notice: //iptables/Service[build_iptables]/ensure: is stopped, should
be running (noop)
notice: Finished catalog run in 3.09 seconds
Excerpt from manifest that does this:
service { "build_iptables":
enable => true,
ensure => running,
hasstatus => true,
}
''build_iptables'' does a couple of things, but most importantly
for
this problem it runs a regex to look for a particular rule to see if
it exists. If said rule exists it returns that the service is
running, if that rule doesn''t exist then it says it is not running and
restarts the ''build_iptables'' service (which does things to
fix the
issue along with restarts the "real" iptables service).
But if I run the same build_iptables script I see that it is running:
[me@host ~]$ sudo /sbin/service build_iptables status
iptables is running
[me@host ~]$ echo $?
0
[me@host ~]$ sudo /usr/sbin/puppetd --test --noop
info: Caching catalog for host.fakedomain
info: Applying configuration version ''1312979662''
notice: //iptables/Service[build_iptables]/ensure: is stopped, should
be running (noop)
notice: Finished catalog run in 2.88 seconds
[me@host ~]$ sudo /sbin/service build_iptables status
iptables is running
[me@host ~]$ echo $?
0
This worked for a number of months, but recently broke. The issue is
that when iptables restarts it drops it''s state table and any
connections that would be allowed (through the ESTABLISHED or RELATED
rule) are no longer recognized and the packets are dropped.
Here is the relevant debug output:
debug: Service[build_iptables](provider=redhat): Executing ''/sbin/
service build_iptables status''
debug: Puppet::Type::Service::ProviderRedhat: Executing ''/sbin/
chkconfig build_iptables''
debug: //iptables/Service[build_iptables]: Changing ensure
debug: //iptables/Service[build_iptables]: 1 change(s)
notice: //iptables/Service[build_iptables]/ensure: is stopped, should
be running (noop)
And the log entry from the puppet service:
Aug 11 16:12:13 host/host puppetd[4113]: (//iptables/
Service[build_iptables]/ensure) ensure changed ''stopped'' to
''running
Any ideas?
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
On 08/12/2011 09:32 AM, Chad wrote:> Any ideas? > >For starters, see what happens when you run it in an exec { "build_iptables status": logoutput => true, } -- vagn -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
[me@host ~]$ sudo /usr/sbin/puppetd --test --noop info: Caching catalog for host.fakedomain info: Applying configuration version ''1313165427'' notice: //iptables/Service[build_iptables]/ensure: is stopped, should be running (noop) notice: //iptables/Exec[/etc/init.d/build_iptables status]/returns: is notrun, should be 0 (noop) notice: Finished catalog run in 2.85 seconds "is notrun"? I am not sure what that means... On Aug 12, 10:23 am, vagn scott <vagnsc...@gmail.com> wrote:> On 08/12/2011 09:32 AM, Chad wrote: > > > Any ideas? > > For starters, see what happens when you run it in an > > exec { "build_iptables status": logoutput => true, } > > -- > vagn-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Fri, Aug 12, 2011 at 06:32:57AM -0700, Chad wrote:> All, > > I have puppet watching various services on RHEL systems and restarting > them as necessary. > > Quite a while back I wrote some custom scripts as wrappers for various > services so that I can have a "custom" running status. These have > worked for quite a few months, but recently my iptables services have > been restarting every time puppet checks in. > > Problem: > > [me@host ~]$ sudo /usr/sbin/puppetd --test --noop > info: Caching catalog for host@fakedomain > info: Applying configuration version ''1312979662'' > notice: //iptables/Service[build_iptables]/ensure: is stopped, should > be running (noop) > notice: Finished catalog run in 3.09 seconds > > Excerpt from manifest that does this: > > service { "build_iptables": > enable => true, > ensure => running, > hasstatus => true, > } > > > ''build_iptables'' does a couple of things, but most importantly for > this problem it runs a regex to look for a particular rule to see if > it exists. If said rule exists it returns that the service is > running, if that rule doesn''t exist then it says it is not running and > restarts the ''build_iptables'' service (which does things to fix the > issue along with restarts the "real" iptables service). > > But if I run the same build_iptables script I see that it is running: > > [me@host ~]$ sudo /sbin/service build_iptables status > iptables is running > [me@host ~]$ echo $? > 0 > > [me@host ~]$ sudo /usr/sbin/puppetd --test --noop > info: Caching catalog for host.fakedomain > info: Applying configuration version ''1312979662'' > notice: //iptables/Service[build_iptables]/ensure: is stopped, should > be running (noop) > notice: Finished catalog run in 2.88 seconds > > [me@host ~]$ sudo /sbin/service build_iptables status > iptables is running > [me@host ~]$ echo $? > 0 > > > This worked for a number of months, but recently broke. The issue is > that when iptables restarts it drops it''s state table and any > connections that would be allowed (through the ESTABLISHED or RELATED > rule) are no longer recognized and the packets are dropped. > > Here is the relevant debug output: > > debug: Service[build_iptables](provider=redhat): Executing ''/sbin/ > service build_iptables status'' > debug: Puppet::Type::Service::ProviderRedhat: Executing ''/sbin/ > chkconfig build_iptables'' > debug: //iptables/Service[build_iptables]: Changing ensure > debug: //iptables/Service[build_iptables]: 1 change(s) > notice: //iptables/Service[build_iptables]/ensure: is stopped, should > be running (noop) >So puppet actually uses »/sbin/service build_iptables status« to check if your service is running. Is it possible that your initscript depends on a specific environment (like a special PATH variable?). Maybe your script doesnt work when run through puppet so it doesnt return with zero and puppet thinks the service is not running. -Stefan
> So puppet actually uses »/sbin/service build_iptables status« to check > if your service is running.Agreed. The service command works as well (and it also checks to make sure it is chkconfig''ed on) This has worked for months before recently. I just just checked and I added the status option on 2/23/11 (these are in a version control system, so I have logs). I just attempted to find in the logs when this started happening (suspect 2 weeks ago), but I had the loglevel set to debug on this from an old test I was doing and never removed it (oops!). Any other ideas? On Aug 12, 1:39 pm, Stefan Schulte <stefan.schu...@taunusstein.net> wrote:> On Fri, Aug 12, 2011 at 06:32:57AM -0700, Chad wrote: > > All, > > > I have puppet watching various services on RHEL systems and restarting > > them as necessary. > > > Quite a while back I wrote some custom scripts as wrappers for various > > services so that I can have a "custom" running status. These have > > worked for quite a few months, but recently my iptables services have > > been restarting every time puppet checks in. > > > Problem: > > > [me@host ~]$ sudo /usr/sbin/puppetd --test --noop > > info: Caching catalog for host@fakedomain > > info: Applying configuration version ''1312979662'' > > notice: //iptables/Service[build_iptables]/ensure: is stopped, should > > be running (noop) > > notice: Finished catalog run in 3.09 seconds > > > Excerpt from manifest that does this: > > > service { "build_iptables": > > enable => true, > > ensure => running, > > hasstatus => true, > > } > > > ''build_iptables'' does a couple of things, but most importantly for > > this problem it runs a regex to look for a particular rule to see if > > it exists. If said rule exists it returns that the service is > > running, if that rule doesn''t exist then it says it is not running and > > restarts the ''build_iptables'' service (which does things to fix the > > issue along with restarts the "real" iptables service). > > > But if I run the same build_iptables script I see that it is running: > > > [me@host ~]$ sudo /sbin/service build_iptables status > > iptables is running > > [me@host ~]$ echo $? > > 0 > > > [me@host ~]$ sudo /usr/sbin/puppetd --test --noop > > info: Caching catalog for host.fakedomain > > info: Applying configuration version ''1312979662'' > > notice: //iptables/Service[build_iptables]/ensure: is stopped, should > > be running (noop) > > notice: Finished catalog run in 2.88 seconds > > > [me@host ~]$ sudo /sbin/service build_iptables status > > iptables is running > > [me@host ~]$ echo $? > > 0 > > > This worked for a number of months, but recently broke. The issue is > > that when iptables restarts it drops it''s state table and any > > connections that would be allowed (through the ESTABLISHED or RELATED > > rule) are no longer recognized and the packets are dropped. > > > Here is the relevant debug output: > > > debug: Service[build_iptables](provider=redhat): Executing ''/sbin/ > > service build_iptables status'' > > debug: Puppet::Type::Service::ProviderRedhat: Executing ''/sbin/ > > chkconfig build_iptables'' > > debug: //iptables/Service[build_iptables]: Changing ensure > > debug: //iptables/Service[build_iptables]: 1 change(s) > > notice: //iptables/Service[build_iptables]/ensure: is stopped, should > > be running (noop) > > So puppet actually uses »/sbin/service build_iptables status« to check > if your service is running. > > Is it possible that your initscript depends on a specific environment > (like a special PATH variable?). Maybe your script doesnt work when run > through puppet so it doesnt return with zero and puppet thinks the > service is not running. > > -Stefan > > application_pgp-signature_part > < 1KViewDownload-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
> Is it possible that your initscript depends on a specific environment > (like a special PATH variable?).Sorry meant to comment on this as well. The status is actually a regular expression looking for a line from ''/sbin/iptables -L''. Here is the status section of the script (bash) ********** status) /sbin/iptables -L | /bin/egrep ''^DROP+\s+all.*NEW\s?+$'' > /dev/ null # A string has to start with "DROP" then whitespace, then "all", then anything, then NEW, then any amount of whitespace # For some reason iptables -L has a whitespace after NEW if [ "$?" = "0" ]; then echo "iptables is running" exit 0 else echo "iptables is stopped" exit 3 fi ;; ********** However, I re-ran the test again without the --noop and got a bit more output: info: Caching catalog for raspberry.argon.local info: Applying configuration version ''1313181263'' notice: //iptables/Service[build_iptables]/ensure: ensure changed ''stopped'' to ''running'' notice: //iptables/Exec[/sbin/service build_iptables status]/returns: iptables is stopped err: //iptables/Exec[/sbin/service build_iptables status]/returns: change from notrun to 0 failed: /sbin/service build_iptables status returned 3 instead of one of [0] at /etc/puppet/modules/iptables/ manifests/iptables.pp:73 notice: Finished catalog run in 5.77 seconds [me@host ~]$ sudo /sbin/service build_iptables status iptables is running [me@host ~]$ echo $? 0 Line 73 of iptables.pp is the exec that was mentioned from vagn (above) exec { "/sbin/service build_iptables status": logoutput => true, } So it is claiming it is returning with a value of 3, but that doesn''t line up with what I see. The command is pretty straightforward so it shouldn''t be hitting any environment variables with /sbin/iptables and /bin/egrep... But it''s very possible I am missing something here... On Aug 12, 4:26 pm, Chad <tru...@gmail.com> wrote:> > So puppet actually uses »/sbin/service build_iptables status« to check > > if your service is running. > > Agreed. The service command works as well (and it also checks to make > sure it is chkconfig''ed on) > > This has worked for months before recently. I just just checked and I > added the status option on 2/23/11 (these are in a version control > system, so I have logs). > > I just attempted to find in the logs when this started happening > (suspect 2 weeks ago), but I had the loglevel set to debug on this > from an old test I was doing and never removed it (oops!). > > Any other ideas? > > On Aug 12, 1:39 pm, Stefan Schulte <stefan.schu...@taunusstein.net> > wrote: > > > On Fri, Aug 12, 2011 at 06:32:57AM -0700, Chad wrote: > > > All, > > > > I have puppet watching various services on RHEL systems and restarting > > > them as necessary. > > > > Quite a while back I wrote some custom scripts as wrappers for various > > > services so that I can have a "custom" running status. These have > > > worked for quite a few months, but recently my iptables services have > > > been restarting every time puppet checks in. > > > > Problem: > > > > [me@host ~]$ sudo /usr/sbin/puppetd --test --noop > > > info: Caching catalog for host@fakedomain > > > info: Applying configuration version ''1312979662'' > > > notice: //iptables/Service[build_iptables]/ensure: is stopped, should > > > be running (noop) > > > notice: Finished catalog run in 3.09 seconds > > > > Excerpt from manifest that does this: > > > > service { "build_iptables": > > > enable => true, > > > ensure => running, > > > hasstatus => true, > > > } > > > > ''build_iptables'' does a couple of things, but most importantly for > > > this problem it runs a regex to look for a particular rule to see if > > > it exists. If said rule exists it returns that the service is > > > running, if that rule doesn''t exist then it says it is not running and > > > restarts the ''build_iptables'' service (which does things to fix the > > > issue along with restarts the "real" iptables service). > > > > But if I run the same build_iptables script I see that it is running: > > > > [me@host ~]$ sudo /sbin/service build_iptables status > > > iptables is running > > > [me@host ~]$ echo $? > > > 0 > > > > [me@host ~]$ sudo /usr/sbin/puppetd --test --noop > > > info: Caching catalog for host.fakedomain > > > info: Applying configuration version ''1312979662'' > > > notice: //iptables/Service[build_iptables]/ensure: is stopped, should > > > be running (noop) > > > notice: Finished catalog run in 2.88 seconds > > > > [me@host ~]$ sudo /sbin/service build_iptables status > > > iptables is running > > > [me@host ~]$ echo $? > > > 0 > > > > This worked for a number of months, but recently broke. The issue is > > > that when iptables restarts it drops it''s state table and any > > > connections that would be allowed (through the ESTABLISHED or RELATED > > > rule) are no longer recognized and the packets are dropped. > > > > Here is the relevant debug output: > > > > debug: Service[build_iptables](provider=redhat): Executing ''/sbin/ > > > service build_iptables status'' > > > debug: Puppet::Type::Service::ProviderRedhat: Executing ''/sbin/ > > > chkconfig build_iptables'' > > > debug: //iptables/Service[build_iptables]: Changing ensure > > > debug: //iptables/Service[build_iptables]: 1 change(s) > > > notice: //iptables/Service[build_iptables]/ensure: is stopped, should > > > be running (noop) > > > So puppet actually uses »/sbin/service build_iptables status« to check > > if your service is running. > > > Is it possible that your initscript depends on a specific environment > > (like a special PATH variable?). Maybe your script doesnt work when run > > through puppet so it doesnt return with zero and puppet thinks the > > service is not running. > > > -Stefan > > > application_pgp-signature_part > > < 1KViewDownload-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
In regard to: [Puppet Users] Re: hasstatus not working as expected, Chad...:> status) > /sbin/iptables -L | /bin/egrep ''^DROP+\s+all.*NEW\s?+$'' > /dev/ > nullI doubt this is the actual problem, but you could more portably and more correctly write that regex as ''^DROP[[:space:]]+all.*NEW[[:space:]]*$'' You don''t need the "+" after the P, since you don''t want to match things like DROPPPPP, and you don''t need the ? or the + after NEW, since you''re anchoring the end and there''s therefore really no reason to use non-greedy matching. Your comment also mentions "any amount of whitespace", which presumably includes 0 matches, hence the * instead of the +. Also, you should probably run (as you) env grep -i grep and see if something like GREP_OPTIONS is in your environment.> # A string has to start with "DROP" then whitespace, then "all", then > anything, then NEW, then any amount of whitespace > # For some reason iptables -L has a whitespace after NEW > if [ "$?" = "0" ]; then > echo "iptables is running" > exit 0 > else > echo "iptables is stopped" > exit 3 > fi > ;; > **********Tim -- Tim Mooney mooney@dogbert.cc.ndsu.NoDak.edu Enterprise Computing & Infrastructure 701-231-1076 (Voice) Room 242-J6, IACC Building 701-231-8541 (Fax) North Dakota State University, Fargo, ND 58105-5164 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Thanks for the regex fix. I am terrible at those, that probably took me 10 minutes to get that far... And my environment has nothing declared with grep. I thought more about this, and I am thinking the problem might have come up when RHEL 5.7 came out. I tested on a few RHEL 6 boxes and this works fine. But I can''t figure out what would have changed that still allows me to run it fine but puppet fails. On Aug 12, 5:10 pm, Tim Mooney <Tim.Moo...@ndsu.edu> wrote:> In regard to: [Puppet Users] Re: hasstatus not working as expected, Chad...: > > > status) > > /sbin/iptables -L | /bin/egrep ''^DROP+\s+all.*NEW\s?+$'' > /dev/ > > null > > I doubt this is the actual problem, but you could more portably and > more correctly write that regex as > > ''^DROP[[:space:]]+all.*NEW[[:space:]]*$'' > > You don''t need the "+" after the P, since you don''t want to match things > like DROPPPPP, and you don''t need the ? or the + after NEW, since you''re > anchoring the end and there''s therefore really no reason to use non-greedy > matching. Your comment also mentions "any amount of whitespace", which > presumably includes 0 matches, hence the * instead of the +. > > Also, you should probably run (as you) > > env grep -i grep > > and see if something like GREP_OPTIONS is in your environment. > > > # A string has to start with "DROP" then whitespace, then "all", then > > anything, then NEW, then any amount of whitespace > > # For some reason iptables -L has a whitespace after NEW > > if [ "$?" = "0" ]; then > > echo "iptables is running" > > exit 0 > > else > > echo "iptables is stopped" > > exit 3 > > fi > > ;; > > ********** > > Tim > -- > Tim Mooney moo...@dogbert.cc.ndsu.NoDak.edu > Enterprise Computing & Infrastructure 701-231-1076 (Voice) > Room 242-J6, IACC Building 701-231-8541 (Fax) > North Dakota State University, Fargo, ND 58105-5164-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.