Puppet users - Feb 2011 - Starting Tomcat Through Service Resource

I''m running into a bizarre issue.  What it boils down to is that when
I start tomcat through puppetd, certain UTF-8 settings do not appear
to take effect and the end result is garbage characters on a web
page.  However, when we run puppetd with --no-daemonize --debug --
trace --one-time, everything works fine.  Tomcat also starts
appropriately when started on boot, or manually with the service start
command.  Our initial thought was that potentially there was a
difference in the environment variables being passed to tomcat
depending on how it was started, but we''ve examined the environments
and altered the start command puppet is using to ensure that they are
identical.  I''ve included our service entry below, any help would be
appreicated.

   service { tomcat6:
     enable => false,
     #hasrestart => true,
     start => ''/bin/bash -c "unset LANGUAGE; unset LC_ALL;
unset
LC_MESSAGES; /usr/sbin/service tomcat6 start"'',
     require => [ Package[tomcat6] ],
     subscribe => [ File["/etc/tomcat6/tomcat-users.xml"],
File["/etc/
tomcat6/server.xml"], File["/etc/tomcat6/config.properties"],
File["/
etc/tomcat6/context.xml"] ],
     ensure => running,
    }

enable is set to false in this entry to ensure that the correct
configuration is in place before the service is started and picked up
by the load balancer.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 2, 1:12 pm, robertbogdon <robertbog...@gmail.com>
wrote:
> I''m running into a bizarre issue.  What it boils down to is that
when
> I start tomcat through puppetd, certain UTF-8 settings do not appear
> to take effect and the end result is garbage characters on a web
> page.  However, when we run puppetd with --no-daemonize --debug --
> trace --one-time, everything works fine.  Tomcat also starts
> appropriately when started on boot, or manually with the service start
> command.  Our initial thought was that potentially there was a
> difference in the environment variables being passed to tomcat
> depending on how it was started, but we''ve examined the
environments
> and altered the start command puppet is using to ensure that they are
> identical.  I''ve included our service entry below, any help would
be
> appreicated.
>
>    service { tomcat6:
>      enable => false,
>      #hasrestart => true,
>      start => ''/bin/bash -c "unset LANGUAGE; unset LC_ALL;
unset
> LC_MESSAGES; /usr/sbin/service tomcat6 start"'',
>      require => [ Package[tomcat6] ],
>      subscribe => [ File["/etc/tomcat6/tomcat-users.xml"],
File["/etc/
> tomcat6/server.xml"],
File["/etc/tomcat6/config.properties"], File["/
> etc/tomcat6/context.xml"] ],
>      ensure => running,
>     }
>
> enable is set to false in this entry to ensure that the correct
> configuration is in place before the service is started and picked up
> by the load balancer.
I concur that the difference is probably related to the environment.
How are you testing / verifying that the environments are identical?
For instance, have you inserted code in the init script to dump the
environment to a file at runtime?

Other than environment *variables*, the most suspect environmental
influence is the user identity (effective *and* real) on whose behalf
the service is started.  Do you perchance have SELinux running in
enforcing mode?  In that case, you need to expand the notion of
"identity" to security context.  These considerations could affect
whether Tomcat is able to read various configuration files, especially
any that have been locally modified.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

We''re looking at /proc/<pid>/environ to see the environment of
the
tomcat process.  We''re not using SELinux, and in this case everything
is being run by root, though tomcat does switch over to a tomcat user
at some point during it''s startup.  What really confuses me is why it
works in with all of the debug flags, but not when run normally as a
daemon.

On Thu, Feb 3, 2011 at 6:26 AM, jcbollinger <John.Bollinger@stjude.org>
wrote:
>
>
> On Feb 2, 1:12 pm, robertbogdon <robertbog...@gmail.com> wrote:
>> I''m running into a bizarre issue.  What it boils down to is
that when
>> I start tomcat through puppetd, certain UTF-8 settings do not appear
>> to take effect and the end result is garbage characters on a web
>> page.  However, when we run puppetd with --no-daemonize --debug --
>> trace --one-time, everything works fine.  Tomcat also starts
>> appropriately when started on boot, or manually with the service start
>> command.  Our initial thought was that potentially there was a
>> difference in the environment variables being passed to tomcat
>> depending on how it was started, but we''ve examined the
environments
>> and altered the start command puppet is using to ensure that they are
>> identical.  I''ve included our service entry below, any help
would be
>> appreicated.
>>
>>    service { tomcat6:
>>      enable => false,
>>      #hasrestart => true,
>>      start => ''/bin/bash -c "unset LANGUAGE; unset
LC_ALL; unset
>> LC_MESSAGES; /usr/sbin/service tomcat6 start"'',
>>      require => [ Package[tomcat6] ],
>>      subscribe => [ File["/etc/tomcat6/tomcat-users.xml"],
File["/etc/
>> tomcat6/server.xml"],
File["/etc/tomcat6/config.properties"], File["/
>> etc/tomcat6/context.xml"] ],
>>      ensure => running,
>>     }
>>
>> enable is set to false in this entry to ensure that the correct
>> configuration is in place before the service is started and picked up
>> by the load balancer.
>
> I concur that the difference is probably related to the environment.
> How are you testing / verifying that the environments are identical?
> For instance, have you inserted code in the init script to dump the
> environment to a file at runtime?
>
> Other than environment *variables*, the most suspect environmental
> influence is the user identity (effective *and* real) on whose behalf
> the service is started.  Do you perchance have SELinux running in
> enforcing mode?  In that case, you need to expand the notion of
> "identity" to security context.  These considerations could
affect
> whether Tomcat is able to read various configuration files, especially
> any that have been locally modified.
>
>
> John
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
>


-- 
---
Robert Bogdon
Sr. Systems Engineer, DivX
http://www.blah.net

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

The only substantial differences in execution environment between
those two should be that the daemon is running in a distinct session
from your shell, has a CWD of ''/'', and has std{in,out,err}
open to
/dev/null.

The rest of the code is identical; it might be interesting to find out
if the ''--debug'' or ''--trace'' flags are what
makes the difference, by
dropping them from the command-line while retaining
''--no-daemonize''
and see if that works or fails?

Regards,
    Daniel

On Thu, Feb 3, 2011 at 08:14, Robert Bogdon <robertbogdon@gmail.com>
wrote:
> We''re looking at /proc/<pid>/environ to see the environment
of the
> tomcat process.  We''re not using SELinux, and in this case
everything
> is being run by root, though tomcat does switch over to a tomcat user
> at some point during it''s startup.  What really confuses me is why
it
> works in with all of the debug flags, but not when run normally as a
> daemon.
>
> On Thu, Feb 3, 2011 at 6:26 AM, jcbollinger
<John.Bollinger@stjude.org> wrote:
>>
>>
>> On Feb 2, 1:12 pm, robertbogdon <robertbog...@gmail.com> wrote:
>>> I''m running into a bizarre issue.  What it boils down to
is that when
>>> I start tomcat through puppetd, certain UTF-8 settings do not
appear
>>> to take effect and the end result is garbage characters on a web
>>> page.  However, when we run puppetd with --no-daemonize --debug --
>>> trace --one-time, everything works fine.  Tomcat also starts
>>> appropriately when started on boot, or manually with the service
start
>>> command.  Our initial thought was that potentially there was a
>>> difference in the environment variables being passed to tomcat
>>> depending on how it was started, but we''ve examined the
environments
>>> and altered the start command puppet is using to ensure that they
are
>>> identical.  I''ve included our service entry below, any
help would be
>>> appreicated.
>>>
>>>    service { tomcat6:
>>>      enable => false,
>>>      #hasrestart => true,
>>>      start => ''/bin/bash -c "unset LANGUAGE; unset
LC_ALL; unset
>>> LC_MESSAGES; /usr/sbin/service tomcat6 start"'',
>>>      require => [ Package[tomcat6] ],
>>>      subscribe => [
File["/etc/tomcat6/tomcat-users.xml"], File["/etc/
>>> tomcat6/server.xml"],
File["/etc/tomcat6/config.properties"], File["/
>>> etc/tomcat6/context.xml"] ],
>>>      ensure => running,
>>>     }
>>>
>>> enable is set to false in this entry to ensure that the correct
>>> configuration is in place before the service is started and picked
up
>>> by the load balancer.
>>
>> I concur that the difference is probably related to the environment.
>> How are you testing / verifying that the environments are identical?
>> For instance, have you inserted code in the init script to dump the
>> environment to a file at runtime?
>>
>> Other than environment *variables*, the most suspect environmental
>> influence is the user identity (effective *and* real) on whose behalf
>> the service is started.  Do you perchance have SELinux running in
>> enforcing mode?  In that case, you need to expand the notion of
>> "identity" to security context.  These considerations could
affect
>> whether Tomcat is able to read various configuration files, especially
>> any that have been locally modified.
>>
>>
>> John
>>
>> --
>> You received this message because you are subscribed to the Google
Groups "Puppet Users" group.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
>> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>>
>>
>
>
>
> --
> ---
> Robert Bogdon
> Sr. Systems Engineer, DivX
> http://www.blah.net
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
>


-- 
⎋ Puppet Labs Developer – http://puppetlabs.com
✉ Daniel Pittman <daniel@puppetlabs.com>
✆ Contact me via gtalk, email, or phone: +1 (877) 575-9775
♲ Made with 100 percent post-consumer electrons

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Feb 2, 2011 at 11:12 AM, robertbogdon <robertbogdon@gmail.com>
wrote:
>
> I''m running into a bizarre issue.  What it boils down to is that
when
> I start tomcat through puppetd, certain UTF-8 settings do not appear
> to take effect and the end result is garbage characters on a web
> page.  However, when we run puppetd with --no-daemonize --debug --
> trace --one-time, everything works fine.  Tomcat also starts
> appropriately when started on boot, or manually with the service start
> command.  Our initial thought was that potentially there was a
> difference in the environment variables being passed to tomcat
> depending on how it was started, but we''ve examined the
environments
> and altered the start command puppet is using to ensure that they are
> identical.  I''ve included our service entry below, any help would
be
> appreicated.
>
>   service { tomcat6:
>     enable => false,
>     #hasrestart => true,
>     start => ''/bin/bash -c "unset LANGUAGE; unset LC_ALL;
unset
> LC_MESSAGES; /usr/sbin/service tomcat6 start"'',
>     require => [ Package[tomcat6] ],
>     subscribe => [ File["/etc/tomcat6/tomcat-users.xml"],
File["/etc/
> tomcat6/server.xml"],
File["/etc/tomcat6/config.properties"], File["/
> etc/tomcat6/context.xml"] ],
>     ensure => running,
>    }
There''s some subtle difference in environment variable between running
puppet agent as a daemon and onetime under the root user. Is your
script by any chance affected by environment variables? If so add an
exec that dumps this info to a file or check /proc/{proc_id}/environ.

Thanks,

Nan

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 3, 10:14 am, Robert Bogdon <robertbog...@gmail.com>
wrote:
> We''re looking at /proc/<pid>/environ to see the environment
of the
> tomcat process.
What about the environment in which the initscript runs?  It is not
safe to assume that''s the same as the environment of the Tomcat
process that gets spawned.  Among other things, the initscript
environment might affect the command-line options passed to Tomcat or
even the JRE with which it is run (if more than one is available),
which are some of the places that character encoding can be
influenced.  Have you compared the command lines with which Tomcat is
started?
>  We''re not using SELinux, and in this case everything
> is being run by root, though tomcat does switch over to a tomcat user
> at some point during it''s startup.  What really confuses me is why
it
> works in with all of the debug flags, but not when run normally as a
> daemon.
I agree that''s surprising.  I would be very interested in the outcome
of Daniel''s suggestion to determine which puppetd flags are sufficient
to cause the difference in behavior.

I would also be interested in the outcome of a companion experiment
wherein you use the ''env'' command to start puppetd
--no-daemonize --
debug --trace --one-time with the minimum sufficient environment to
get it to run (perhaps just a minimal path?).


Good Luck,

John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.