Puppet users - Jun 2011 - Prerun, Postrun Commands, and Stages

Hi all,

I''m looking for background information about how bug #7127[1] should
be fixed: prerun_command don''t stop puppet on error

I think there''s general agreement that if the prerun command fails,
then the catalog should not be applied, but the report should be sent,
and the report''s status should be "failed".

However, what about the post-run command? In particular, if the
catalog is applied successfully, but the postrun command fails, should
the overall run be considered a failure? The documentation[2] says it
should be:

"A command to run after every agent run. If this command returns a
non-zero return code, the entire Puppet run will be considered to have
failed, even though it might have performed work during the normal
run."

But there are several problems with the way the code is currently implemented.

  * If the postrun command fails, puppet never sends the report.
  * Errors that occur while running the pre and postrun commands are
not captured in the report''s log.
  * If the catalog is applied successfully, but the postrun command
fails, the report status is not changed to "failed".

Right now it doesn''t matter because the report is never sent, but if I
fix that, it could matter.

Thoughts? The only use case I know of is etckeeper, but its postrun
command, etckeeper-commit-post[3], always returns 0 even if the
etckeeper command fails.

Finally, the prerun command is executed after dostorage,
download_plugins, download_fact_plugins. Is there reason for the
prerun command to occur first?

It''d be great to hear about your experience with the pre/post run
commands and what use cases you are trying to solve.

Also, is there anything that is being solved with pre/post run
commands that can''t be solved using stages? For example, if the prerun
command, catalog, and postrun commands are executed as stages, in that
order, with each stage depending on its predecessor(s), it would
ensure that:

  * An error in one stage would prevent the following stage(s) from executing.
  * The report would contain all errors from stages that were executed.
  * The report status, resource statuses, and metrics would be consistent.

Thanks,
Josh

[1] http://projects.puppetlabs.com/issues/7127
[2]
http://docs.puppetlabs.com/references/stable/configuration.html#postruncommand
[3] https://code.launchpad.net/~soren/ubuntu/lucid/puppet/etckeeper-integration

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Developers" group.
To post to this group, send email to puppet-dev@googlegroups.com.
To unsubscribe from this group, send email to
puppet-dev+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-dev?hl=en.

On Wed, Jun 08, 2011 at 05:50:58PM -0700, Josh Cooper wrote:
 
> It''d be great to hear about your experience with the pre/post run
> commands and what use cases you are trying to solve.
We use the feature to generate additional information about how the
puppet run has changed the system:

http://www.unixdaemon.net/tools/puppet/nagios-wrapped-puppet-runs.html

In our use case we''re only using the two stages as information hooks
for
their side-effects - not to alter the puppet run.
> Also, is there anything that is being solved with pre/post run
> commands that can''t be solved using stages? For example, if the
prerun
> command, catalog, and postrun commands are executed as stages, in that
> order, with each stage depending on its predecessor(s), it would
> ensure that:
For my example usage I could quite easily move the commands to be execs
in the pre and post stages.
 
  Dean
-- 
Dean Wilson
http://www.unixdaemon.net       @unixdaemon
http://www.puppetcookbook.com   @puppetcookbook

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thu, Jun 9, 2011 at 3:15 AM, Dean Wilson <dwilson@unixdaemon.net>
wrote:
> On Wed, Jun 08, 2011 at 05:50:58PM -0700, Josh Cooper wrote:
>
> > It''d be great to hear about your experience with the pre/post
run
> > commands and what use cases you are trying to solve.
>
> We use the feature to generate additional information about how the
> puppet run has changed the system:
>
> http://www.unixdaemon.net/tools/puppet/nagios-wrapped-puppet-runs.html
>
> In our use case we''re only using the two stages as information
hooks for
> their side-effects - not to alter the puppet run.
>
> > Also, is there anything that is being solved with pre/post run
> > commands that can''t be solved using stages? For example, if
the prerun
> > command, catalog, and postrun commands are executed as stages, in that
> > order, with each stage depending on its predecessor(s), it would
> > ensure that:
>
> For my example usage I could quite easily move the commands to be execs
> in the pre and post stages.
>
One use of pre commands that isn''t solved with stages is to check
"Should I
even do a Puppet run right now?" or anything else that is out of band in a
similar sense.

I used to do this in a wrapper script where the nodes would basically look
and see if the masters were all under too much load to serve this node or
not, or to check whether puppet runs had been disabled centrally, but
ideally this would just be a prerun command.




-- 
Nigel Kersten
Product, Puppet Labs
@nigelkersten

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Josh,

This might be out of scope of your bug fix, but why not make the
behaviour configurable? If prerun/postrun fails, give people the
option of continuing or dying.

prerun_command = /bin/meow
prerun_failure_fatal = true
> It''d be great to hear about your experience with the pre/post run
> commands and what use cases you are trying to solve.
>
> Also, is there anything that is being solved with pre/post run
> commands that can''t be solved using stages? For example, if the
prerun
> command, catalog, and postrun commands are executed as stages, in that
> order, with each stage depending on its predecessor(s), it would
> ensure that:
>
>   * An error in one stage would prevent the following stage(s) from
executing.
I have one use case to contribute. I''ve once had to use prerun and
postrun hooks to open up software firewall rules using iptables for
the duration of the puppet run to allow outbound http traffic for
package retrieval (don''t ask, was inserting Puppet into an existing
environment). So had the equivalent of:

prerun_command = /sbin/iptables -D rule
postrun_command = /sbin/iptables -A rule

Pretty sure I couldn''t use run stages for that because if the puppet
catalog failed to apply, the iptables rule would not be re-added.

-Luke

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thu, Jun 9, 2011 at 7:04 AM, Luke Bigum <Luke.Bigum@lmax.com> wrote:
> Hi Josh,
>
> This might be out of scope of your bug fix, but why not make the
> behaviour configurable? If prerun/postrun fails, give people the
> option of continuing or dying.
>
> prerun_command = /bin/meow
> prerun_failure_fatal = true
>
You should also be able to do something like this I assume.

prerun_command = /bin/meow || true

:)

>
> > It''d be great to hear about your experience with the pre/post
run
> > commands and what use cases you are trying to solve.
> >
> > Also, is there anything that is being solved with pre/post run
> > commands that can''t be solved using stages? For example, if
the prerun
> > command, catalog, and postrun commands are executed as stages, in that
> > order, with each stage depending on its predecessor(s), it would
> > ensure that:
> >
> >   * An error in one stage would prevent the following stage(s) from
> executing.
>
> I have one use case to contribute. I''ve once had to use prerun and
> postrun hooks to open up software firewall rules using iptables for
> the duration of the puppet run to allow outbound http traffic for
> package retrieval (don''t ask, was inserting Puppet into an
existing
> environment). So had the equivalent of:
>
> prerun_command = /sbin/iptables -D rule
> postrun_command = /sbin/iptables -A rule
>
> Pretty sure I couldn''t use run stages for that because if the
puppet
> catalog failed to apply, the iptables rule would not be re-added.
>
That''s a really good example Luke. Along similar lines would be
relaxing
SELinux rules, remounting filesystems read/write then read/only etc.


>
> -Luke
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>

-- 
Nigel Kersten
Product, Puppet Labs
@nigelkersten

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

> One use of pre commands that isn''t solved with stages is to check
"Should I
> even do a Puppet run right now?" or anything else that is out of band
in a
> similar sense.
This makes complete sense and is how the feature was intended to work,
but unfortunately, it never has worked that way (for the agent).
Currently, if the prerun command fails, puppet will attempt to apply
the catalog. Puppet will also always attempt to send a report (due to
#1054), which partially breaks the "master under too much load" use
case.

So there are several ways in which pre/post run failure states can be
handled. I''m curious to think what you think the default behavior
should be and whether you would like to see these other failure states
supported:

If the prerun_command fails:

  1. Ignore the failure, continue applying the catalog, send the report, etc.
  2. Stop puppet, don''t apply the catalog, don''t send the
report, and exit(1)
  3. Stop puppet, don''t apply the catalog, but do send the report,
including information about why the prerun command failed, and exit(1)

#1 is the current behavior, but could also be accomplished by
appending "|| true" to the prerun_command option, e.g. prerun_command
= /bin/meow || true.
#2 was how the feature was originally implemented and how it is
documented, but due to the merge with #1054, the default behavior was
changed to #1.
#3 can also be accomplished using stages. This would be best used in
cases where the prerun command should be "in-band" and its failure
should affect the overall report status, resource_statuses, metrics,
etc.

I''d like to propose that we change the default behavior for
prerun_command to #2 and document how to accomplish #1 and #3.

Similarly, if the postrun command fails, there are several different options:

  1. Ignore the failure, send the report with whatever status resulted
from applying the catalog, etc.
  2. Stop puppet, don''t send the report (even though the catalog may
have been applied), and exit(1)
  3. Add the postrun command error to the report, change the report
status to "failed", etc., and exit(1)

#1 can be accomplished by appending "|| true" to the postrun_command
option.
#2 is the current behavior.
#3 ideally could be handled using stages, but there is no way
currently to ensure a stage is run.

I''m not sure what the default should be here. For example, if the
postrun command "/sbin/iptables -A rule" fails, should the report have
a "failed" status? If we don''t send the report, will you ever
know,
will you care?

Josh

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thu, Jun 9, 2011 at 1:22 PM, Josh Cooper <josh@puppetlabs.com> wrote:
> > One use of pre commands that isn''t solved with stages is to
check "Should
> I
> > even do a Puppet run right now?" or anything else that is out of
band in
> a
> > similar sense.
>
> This makes complete sense and is how the feature was intended to work,
> but unfortunately, it never has worked that way (for the agent).
> Currently, if the prerun command fails, puppet will attempt to apply
> the catalog. Puppet will also always attempt to send a report (due to
> #1054), which partially breaks the "master under too much load"
use
> case.
>
> So there are several ways in which pre/post run failure states can be
> handled. I''m curious to think what you think the default behavior
> should be and whether you would like to see these other failure states
> supported:
>
> If the prerun_command fails:
>
>  1. Ignore the failure, continue applying the catalog, send the report,
> etc.
>  2. Stop puppet, don''t apply the catalog, don''t send the
report, and
> exit(1)
>  3. Stop puppet, don''t apply the catalog, but do send the report,
> including information about why the prerun command failed, and exit(1)
>
3.

We shouldn''t ask the master to compile a catalog or sync facts, but we
should report. In some cases the report server isn''t the same as the
master.
 As you say 1) is easily achieved.


>
> #1 is the current behavior, but could also be accomplished by
> appending "|| true" to the prerun_command option, e.g.
prerun_command
> = /bin/meow || true.
> #2 was how the feature was originally implemented and how it is
> documented, but due to the merge with #1054, the default behavior was
> changed to #1.
> #3 can also be accomplished using stages. This would be best used in
> cases where the prerun command should be "in-band" and its
failure
> should affect the overall report status, resource_statuses, metrics,
> etc.
>
> I''d like to propose that we change the default behavior for
> prerun_command to #2 and document how to accomplish #1 and #3.
>
> Similarly, if the postrun command fails, there are several different
> options:
>
>  1. Ignore the failure, send the report with whatever status resulted
> from applying the catalog, etc.
>  2. Stop puppet, don''t send the report (even though the catalog
may
> have been applied), and exit(1)
>  3. Add the postrun command error to the report, change the report
> status to "failed", etc., and exit(1)
>
> #1 can be accomplished by appending "|| true" to the
postrun_command
> option.
> #2 is the current behavior.
> #3 ideally could be handled using stages, but there is no way
> currently to ensure a stage is run.
>
> I''m not sure what the default should be here. For example, if the
> postrun command "/sbin/iptables -A rule" fails, should the report
have
> a "failed" status? If we don''t send the report, will you
ever know,
> will you care?
>
>
We had a chat about this in person, and my feeling is that the postrun
command shouldn''t change the status of the run (as it''s
already complete),
but the stdout/err/status should be captured in the report if possible.

If you want a command to change the status of the run, it can be an Exec in
a final Stage.


-- 
Nigel Kersten
Product, Puppet Labs
@nigelkersten

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.