Once upon a time, Chris Murphy <lists at colorremedies.com> said:> On Tue, Apr 26, 2016, 2:09 PM Chris Adams <linux at cmadams.net> wrote: > > I have several recently-installed CentOS 7 servers that keep having > > systemd-journald corruption > > Determined with 'journalctl --verify' or another way?I get messages like this in dmesg: [4756650.489117] systemd-journald[21364]: Failed to write entry (21 items, 637 bytes), ignoring: Cannot assign requested address I'll check journalctl --verify when it happens next (seems to happen every day on at least one of the servers). It does only seem to be happening only on my spam-scanning VMs. Some of them do have a relatively high log rate (40-50 messages per second sometimes). I dug into it a little more after my original message, and it appears to be a recent issue; I have some VMs that were set up a little longer ago (still running CentOS 7.1 I believe) that have not had this problem.> If you are referring to native journald logs corrupting, that should not > affect rsyslog. If you remove /var/log/journal then systemd-journald logs > will be stored volatile in /run.That appears to be where they're going (I don't have a /var/log/journal, but I didn't do anything to remove it). I have had to remove files from /run/log/journal to get systemd-journald working again.> > Has anyone else seen this? > > Sortof, but not in a way that affects rsyslog. Usually journalctl just > skips over corrupt parts and systemd-journald will rotate logs when it > detects corruption to isolate corrupt files.When it happens, all logs just stop; rsyslogd appears to not get any more log entries (I have rsyslogd logging to central log hosts and they get nothing). -- Chris Adams <linux at cmadams.net>
On Tue, Apr 26, 2016 at 3:01 PM, Chris Adams <linux at cmadams.net> wrote:> Once upon a time, Chris Murphy <lists at colorremedies.com> said: >> On Tue, Apr 26, 2016, 2:09 PM Chris Adams <linux at cmadams.net> wrote: >> > I have several recently-installed CentOS 7 servers that keep having >> > systemd-journald corruption >> >> Determined with 'journalctl --verify' or another way? > > I get messages like this in dmesg: > > [4756650.489117] systemd-journald[21364]: Failed to write entry (21 items, 637 bytes), ignoring: Cannot assign requested addressI haven't seen this. When I plug this text into a google search field, no quotes, there are 360 results. systemd-journald failed to write entry cannot assign requested address There's also this patch as a suggested fix: https://bugzilla.redhat.com/show_bug.cgi?id=1292447#c9 What version of systemd and rsyslog? systemd-219-19.el7_2.7 and rsyslog-7.4.7-12 are current. If you're there already you could ry editing /etc/systemd/journald.conf and uncommenting Compress=yes and changing it to no. -- Chris Murphy
Once upon a time, Chris Murphy <lists at colorremedies.com> said:> On Tue, Apr 26, 2016 at 3:01 PM, Chris Adams <linux at cmadams.net> wrote: > > Once upon a time, Chris Murphy <lists at colorremedies.com> said: > >> On Tue, Apr 26, 2016, 2:09 PM Chris Adams <linux at cmadams.net> wrote: > >> > I have several recently-installed CentOS 7 servers that keep having > >> > systemd-journald corruption > >> > >> Determined with 'journalctl --verify' or another way?One system did get into this state overnight, and that said: [root at spamscan3 ~]# journalctl --verify 15bd478: invalid object File corruption detected at /run/log/journal/f8ade260c5f84b8aa04095c233c041e0/system.journal:15bd478 (of 25165824 bytes, 90%). FAIL: /run/log/journal/f8ade260c5f84b8aa04095c233c041e0/system.journal (Cannot assign requested address) (and then a bunch of passes on the rest of the files)> There's also this patch as a suggested fix: > https://bugzilla.redhat.com/show_bug.cgi?id=1292447#c9I'll take a look at that.> What version of systemd and rsyslog? systemd-219-19.el7_2.7 and > rsyslog-7.4.7-12 are current.Those are the versions I have.> If you're there already you could ry editing > /etc/systemd/journald.conf and uncommenting Compress=yes and changing > it to no.Thanks, I'm trying that on these servers. -- Chris Adams <linux at cmadams.net>