CentOS - Feb 2017 - Systemd debug logging turned on in CentOS 7

On Tue, Feb 28, 2017 at 10:49 AM, Valeri Galtsev <galtsev at
kicp.uchicago.edu>
wrote:
>
> On Tue, February 28, 2017 9:22 am, Rob DeSanno wrote:
> > Last time I saw it, I had just upgraded my CentOS 7 box with the
> > 3.10.0-514 kernel and it rebooted already configured into debug mode.
> Not
> > sure if this is a ???feature??  of the newer kernels or not but glad
to
> > see that i???m not the only one who had noticed this.
> >
> > # awk -F\' '$1=="menuentry " {print i++ " :
" $2}' /etc/grub2.cfg
> > 0 : CentOS Linux (3.10.0-514.6.2.el7.x86_64) 7 (Core)
> > 1 : CentOS Linux (3.10.0-514.6.2.el7.x86_64) 7 (Core) with debugging
> > 2 : CentOS Linux (0-rescue-7b37bcbe36eb420fb6426976c41b0aaf) 7 (Core)
> > 3 : CentOS Linux (0-rescue-7b37bcbe36eb420fb6426976c41b0aaf) 7 (Core)
> with
> > debugging
>
> I am not certain if there is real harm to have kernel with all debug stuff
> running on production machines. Probably no harm security wise, the only
> unpleasant stuff is: you really would prefer to run as slim kernel as
> possible on production systems. If I'm wrong about "no harm",
somebody
> chime in, I then will be really eager to address it on my boxes.
>
> Valeri
>
>
Main issue I've seen is that logs grow by an order of magnitude larger than
when it's off, due to systemd being systemd and now running in debug mode.
Other than disk space, it would affect any central logging system you have
with lots of unnecessary traffic, and would also add a lot of IO, amplified
if you have many machines running on a VM host.

~ Brian Mathis
@orev

On 02/28/2017 08:55 AM, Brian Mathis wrote:
> On Tue, Feb 28, 2017 at 10:49 AM, Valeri Galtsev <galtsev at
kicp.uchicago.edu>
> wrote:
> 
>>
>> On Tue, February 28, 2017 9:22 am, Rob DeSanno wrote:
>>> Last time I saw it, I had just upgraded my CentOS 7 box with the
>>> 3.10.0-514 kernel and it rebooted already configured into debug
mode.
>> Not
>>> sure if this is a ???feature??  of the newer kernels or not but
glad to
>>> see that i???m not the only one who had noticed this.
>>>
>>> # awk -F\' '$1=="menuentry " {print i++ " :
" $2}' /etc/grub2.cfg
>>> 0 : CentOS Linux (3.10.0-514.6.2.el7.x86_64) 7 (Core)
>>> 1 : CentOS Linux (3.10.0-514.6.2.el7.x86_64) 7 (Core) with
debugging
>>> 2 : CentOS Linux (0-rescue-7b37bcbe36eb420fb6426976c41b0aaf) 7
(Core)
>>> 3 : CentOS Linux (0-rescue-7b37bcbe36eb420fb6426976c41b0aaf) 7
(Core)
>> with
>>> debugging
>>
>> I am not certain if there is real harm to have kernel with all debug
stuff
>> running on production machines. Probably no harm security wise, the
only
>> unpleasant stuff is: you really would prefer to run as slim kernel as
>> possible on production systems. If I'm wrong about "no
harm", somebody
>> chime in, I then will be really eager to address it on my boxes.
>>
>> Valeri
>>
>>
> 
> Main issue I've seen is that logs grow by an order of magnitude larger
than
> when it's off, due to systemd being systemd and now running in debug
mode.
> Other than disk space, it would affect any central logging system you have
> with lots of unnecessary traffic, and would also add a lot of IO, amplified
> if you have many machines running on a VM host.
> 
> ~ Brian Mathis
> @orev

Just to put the record straight; it's not related to kernel debugging
being enabled or not. It's systemd debugging that is being turned on
for all menu entries, kernel debug or not.

Anyway, I think I have found a pattern. Only those machines that were
updated from 7.2 to 7.3 using the CR repo are showing this behaviour.
New 7.3 installs are fine. I'll just clean up the machines affected
and move on. There must have been some debug config left in an
installation script in one of the CR rpms.

thanks to everyone responding,

	Thomas

On Tue, Feb 28, 2017 at 1:44 PM, Thomas Eriksson <
thomas.eriksson at slac.stanford.edu> wrote:
> On 02/28/2017 08:55 AM, Brian Mathis wrote:
> >
> > Main issue I've seen is that logs grow by an order of magnitude
larger
> than
> > when it's off, due to systemd being systemd and now running in
debug
> mode.
> > Other than disk space, it would affect any central logging system you
> have
> > with lots of unnecessary traffic, and would also add a lot of IO,
> amplified
> > if you have many machines running on a VM host.
> >
> > ~ Brian Mathis
> > @orev
>
>
> Just to put the record straight; it's not related to kernel debugging
> being enabled or not. It's systemd debugging that is being turned on
> for all menu entries, kernel debug or not.
>
> Anyway, I think I have found a pattern. Only those machines that were
> updated from 7.2 to 7.3 using the CR repo are showing this behaviour.
> New 7.3 installs are fine. I'll just clean up the machines affected
> and move on. There must have been some debug config left in an
> installation script in one of the CR rpms.
>
> thanks to everyone responding,
>
        Thomas
>

Yes, true, this isn't technically the "kernel" debug mode, but
systemd
debugging is also enabled for the debug boot options, which seems to get
carried into a non-debug boot entry somehow.

I have seen this issue on a few machines, and I don't use the CR repos.
The ones I saw were during upgrades from 7.2 systems with all updates,
which were then upgraded to 7.3 with all updates as of 2 weeks ago.


~ Brian Mathis
@orev