CentOS - Apr 2016 - systemd-journald corruption

I have several recently-installed CentOS 7 servers that keep having
systemd-journald corruption (which stops ALL logging, including syslog).
Interestingly, they are all spam-scanning servers running amavisd-new
(so could be some particular pattern is triggering it).

Is there a "supported" way to just cut systemd-journald out of the
picture and have log entries go straight to rsyslogd?

Has anyone else seen this?
-- 
Chris Adams <linux at cmadams.net>

On Tue, Apr 26, 2016, 2:09 PM Chris Adams <linux at cmadams.net> wrote:
> I have several recently-installed CentOS 7 servers that keep having
> systemd-journald corruption


Determined with 'journalctl --verify' or another way?


(which stops ALL logging, including syslog).
> Interestingly, they are all spam-scanning servers running amavisd-new
> (so could be some particular pattern is triggering it).
>
> Is there a "supported" way to just cut systemd-journald out of
the
> picture and have log entries go straight to rsyslogd?
>
No. Everything reports to journald and rsyslog gets what it wants from
journald.

If you are referring to native journald logs corrupting, that should not
affect rsyslog. If you remove /var/log/journal then systemd-journald logs
will be stored volatile in /run.

> Has anyone else seen this?
>
Sortof, but not in a way that affects rsyslog. Usually journalctl just
skips over corrupt parts and systemd-journald will rotate logs when it
detects corruption to isolate corrupt files.


Chris Murphy

Once upon a time, Chris Murphy <lists at colorremedies.com>
said:
> On Tue, Apr 26, 2016, 2:09 PM Chris Adams <linux at cmadams.net>
wrote:
> > I have several recently-installed CentOS 7 servers that keep having
> > systemd-journald corruption
> 
> Determined with 'journalctl --verify' or another way?
I get messages like this in dmesg:

[4756650.489117] systemd-journald[21364]: Failed to write entry (21 items, 637
bytes), ignoring: Cannot assign requested address

I'll check journalctl --verify when it happens next (seems to happen
every day on at least one of the servers).  It does only seem to be
happening only on my spam-scanning VMs.  Some of them do have a
relatively high log rate (40-50 messages per second sometimes).

I dug into it a little more after my original message, and it appears to
be a recent issue; I have some VMs that were set up a little longer ago
(still running CentOS 7.1 I believe) that have not had this problem.
> If you are referring to native journald logs corrupting, that should not
> affect rsyslog. If you remove /var/log/journal then systemd-journald logs
> will be stored volatile in /run.
That appears to be where they're going (I don't have a /var/log/journal,
but I didn't do anything to remove it).  I have had to remove files from
/run/log/journal to get systemd-journald working again.
> > Has anyone else seen this?
> 
> Sortof, but not in a way that affects rsyslog. Usually journalctl just
> skips over corrupt parts and systemd-journald will rotate logs when it
> detects corruption to isolate corrupt files.
When it happens, all logs just stop; rsyslogd appears to not get any
more log entries (I have rsyslogd logging to central log hosts and they
get nothing).

-- 
Chris Adams <linux at cmadams.net>