Puppet users - Feb 2010 - Puppet's call to /sbin/service somehow different than calling from the command line?

Long shot question, but I''m running out of ideas...

I''m running puppet-0.25.2 on Fedora boxes to manage a simple Java
service started through an /etc/init.d script.  I can start, stop,
restart, and examine the service status using the standard /sbin/
service command.  Works like a champ.

The issue is that when I use puppet to start the service, the service
fails to run.  Specifically, I have a basic ''service {
"name": ensure
=> true }'' defined.  I run "puppetd --test" and I see it
changing the
status of the service from stopped to running, and yet my Java service
fails with random NoClassDefFound errors.

I keep coming back to the fact that I can run my example service with
a direct call to ''/sbin/service name start'' and everything
works
fine.  Yet when Puppet makes that same call (I''ve seen it in the debug
logs), my service fails.  What''s the difference in execution contexts?

Now, I know you''re immediately going to guess that I have some
classpath setup incorrectly or a permissions problem or something.
Believe me, I''ve gone crazy looking for something... anything. 
I''ve
stripped it down to the most basic set of classes.  I''ve added logging
every where.  I''ve looked at everything I can think of.  Comparing
puppet vs non-puppet, I can find no difference in the environment,
path, executable, limits, or anything else I can think of at the point
of service execution.

Does anyone have an ideas, hints, or even vague notions of why
Puppet''s service execution might be somehow different?  I''m
certainly
out of ideas to try.

Thanks,
Brian

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

>
> Does anyone have an ideas, hints, or even vague notions of why
> Puppet''s service execution might be somehow different? 
I''m certainly
> out of ideas to try.
One item you didn''t mention in your list of things you''ve
checked, so
I''ll bring it up:  Environment variables.

Don''t know which ones might be important though :); you could use
"set"
or "env" in the service startup script and dump the output to a text 
file, then compare the state between a terminal invocation and that from 
puppet.  See what''s different and go from there.

Craig Miskell

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 24, 1:17 am, Brian Ferris <bdfer...@gmail.com>
wrote:
>  I see it changing the
> status of the service from stopped to running, and yet my Java service
> fails with random NoClassDefFound errors.
For classes belonging to your application, to external jars, or to the
system library?

[...]
> Now, I know you''re immediately going to guess that I have some
> classpath setup incorrectly or a permissions problem or something.
It''s not a guess: you definitely have a classpath problem.  Following
Craig''s suggestion, you might consider looking at the CLASSPATH
environment variable.  I strongly recommend against relying on
CLASSPATH, by the way, as doing so is prone to problems exactly like
this.  Instead, specify your classpath on the command line.

Even better: package all the needed classes (except those from the
standard runtime library) into the same jar, identify the entry point
via a Main-Class: entry in its manifest, and run the sucker using
"java -jar".

You mention that you have checked permissions, etc..  I assume, then,
that you are aware that if the puppetd process does not have read
permission on one or more of the needed jars, or, recursively, the
directories containing them, then those jars are effectively not in
the classpath.  Also, do you give the *absolute* path to each jar in
your classpath?

One more thing: does your initscript use the absolute pathname of the
java runtime executable?  If not, and if there is more than one Java
runtime installed, then it may be that puppetd is running a different
java than you do from the command line.  That might contribute to your
issue.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 24, 1:17 am, Brian Ferris <bdfer...@gmail.com>
wrote:
> What''s the difference in execution contexts?
One more guess: are you running SELinux in enforcing mode?


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Sorry if I wasn''t clear in the original post.  I''ve definitely
looked at the
contents of the environment (dump of env) and there is no difference in the
two calling contexts.  I''ve also dumped the contents of
System.getProperties() (equivalent of Java''s environment) and again
there is
no difference in the two calling contexts.


On Tue, Feb 23, 2010 at 11:44 PM, Craig Miskell
<craig.miskell@opus.co.nz>wrote:
>
>
>> Does anyone have an ideas, hints, or even vague notions of why
>> Puppet''s service execution might be somehow different? 
I''m certainly
>> out of ideas to try.
>>
> One item you didn''t mention in your list of things you''ve
checked, so I''ll
> bring it up:  Environment variables.
>
> Don''t know which ones might be important though :); you could use
"set" or
> "env" in the service startup script and dump the output to a text
file, then
> compare the state between a terminal invocation and that from puppet.  See
> what''s different and go from there.
>
> Craig Miskell
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Yes to running in enforcing mode.  I just tried turning enforcing mode off
with a call to "setenforce 0" and I still get the same behavior.


On Wed, Feb 24, 2010 at 6:14 AM, jcbollinger
<John.Bollinger@stjude.org>wrote:
>
>
> On Feb 24, 1:17 am, Brian Ferris <bdfer...@gmail.com> wrote:
> > What''s the difference in execution contexts?
>
> One more guess: are you running SELinux in enforcing mode?
>
>
> John
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 24, 9:53 am, Brian Ferris <bdfer...@gmail.com>
wrote:
> Yes to running in enforcing mode.  I just tried turning enforcing mode off
> with a call to "setenforce 0" and I still get the same behavior.
Hmmm.  This still feels like a possibility.  The SELinux context of
the puppetd process is definitely different from the SELinux context
of an interactive shell, so that is a difference such as you asked
about.  It might be worth checking your logs for denial advices.

Alternatively, are you using ACLs on your filesystem?  It could be
that one or more of the needed jars have ACLs that prevent puppetd
access, even though the base file permissions appear to allow it.

Are any of the needed jars located on a remote file system?  That
could bring additional access rules into play that might discriminate
between you and puppetd.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

For what it''s worth, I finally debugged this issue.

When Puppet executes the service command, it forks and (by default) closes
the stdin, stdout, stderr and all other open file descriptors before
executing the service command.  This turned out to be my undoing, as closing
stdin seems to be what caused all the problems for my java daemon.  A simple
"cat /dev/null | java ..." in the daemon init script did the trick,
though I
haven''t debugged specifically what the issue was with java.

Hopefully this will help if anyone ever runs into a similar issue down the
road.

Thanks,
Brian


On Thu, Feb 25, 2010 at 6:52 AM, jcbollinger
<John.Bollinger@stjude.org>wrote:
>
>
> On Feb 24, 9:53 am, Brian Ferris <bdfer...@gmail.com> wrote:
> > Yes to running in enforcing mode.  I just tried turning enforcing mode
> off
> > with a call to "setenforce 0" and I still get the same
behavior.
>
> Hmmm.  This still feels like a possibility.  The SELinux context of
> the puppetd process is definitely different from the SELinux context
> of an interactive shell, so that is a difference such as you asked
> about.  It might be worth checking your logs for denial advices.
>
> Alternatively, are you using ACLs on your filesystem?  It could be
> that one or more of the needed jars have ACLs that prevent puppetd
> access, even though the base file permissions appear to allow it.
>
> Are any of the needed jars located on a remote file system?  That
> could bring additional access rules into play that might discriminate
> between you and puppetd.
>
>
> John
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mon, 01 Mar 2010, Brian Ferris wrote:
> When Puppet executes the service command, it forks and (by default) closes
> the stdin, stdout, stderr and all other open file descriptors before
> executing the service command.
It would be better if puppet redirected stdin, stdout, stderr to or
from /dev/null, instead of closing those file descriptors.

--apb (Alan Barrett)

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mar 1, 2:14 pm, Brian Ferris <bdfer...@gmail.com>
wrote:
> For what it''s worth, I finally debugged this issue.
Great!
> Hopefully this will help if anyone ever runs into a similar issue down the
> road.
Thanks for that.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.