CentOS - Feb 2009 - logs such as messages, boot.log, and kernel contained 0 size

Hi,

My both CentOS 5 servers have logging problems. Logs such as messages, boot.log,
kernel, spooler, and tallylog in /var/log directory are all 0 size.

The kernel is:  Linux 2.6.18-92.1.22.el5 #1 SMP.

Since the /var/log/messages contained no information it would be impossible to
troubleshoot the problem.

I am very sure both systems have not been hacked by others. 

Sincerely,

Frank Ling
--------------------------------------------------------------------------
-rw-------  1 root         root          0 Feb  8 04:02 messages
-rw-------  1 root         root          0 Feb  3 11:04 messages.1
-rw-------  1 root         root          0 Jan 25 04:02 messages.3
-rw-------  1 root         root          0 Jan 11 04:03 messages.4
-rw-------  1 root         root         10 Dec 27 13:00 messages.offset

-rwx------  1 root         root          0 Feb 11 19:12 kernel
-rwx------  1 root         root          0 Feb 11 16:53 kernel.1
-rwx------  1 root         root          0 Jan 25 04:02 kernel.3
-rwx------  1 root         root          0 Jan 11 04:03 kernel.4

-rw-------  1 root         root          0 Feb  8 04:02 spooler
-rw-------  1 root         root          0 Feb  3 07:51 spooler.1
-rw-------  1 root         root          0 Jan 25 04:02 spooler.3
-rw-------  1 root         root          0 Jan 11 04:03 spooler.4

-rw-------  1 root         root          0 Jun 24  2008 tallylog
--------------------------------------------------------------------------


      
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos/attachments/20090211/0126533f/attachment.html>

Frank Ling wrote:
> Hi,
> 
> My both CentOS 5 servers have logging problems. Logs such as messages, 
> boot.log, kernel, spooler, and tallylog in /var/log directory are all 0 
> size.
> 
> The kernel is:  Linux 2.6.18-92.1.22.el5 #1 SMP.
> 
> Since the /var/log/messages contained no information it would be 
> impossible to troubleshoot the problem.
> 
> I am very sure both systems have not been hacked by others.
> 
> Sincerely,
> 
> Frank Ling
> --------------------------------------------------------------------------
> -rw-------  1 root         root          0 Feb  8 04:02 messages
> -rw-------  1 root         root          0 Feb  3 11:04 messages.1
> -rw-------  1 root         root          0 Jan 25 04:02 messages.3
> -rw-------  1 root         root          0 Jan 11 04:03 messages.4
> -rw-------  1 root         root         10 Dec 27 13:00 messages.offset
> 
> -rwx------  1 root         root          0 Feb 11 19:12 kernel
> -rwx------  1 root         root          0 Feb 11 16:53 kernel.1
> -rwx------  1 root         root          0 Jan 25 04:02 kernel.3
> -rwx------  1 root         root          0 Jan 11 04:03 kernel.4
> 
> -rw-------  1 root         root          0 Feb  8 04:02 spooler
> -rw-------  1 root         root          0 Feb  3 07:51 spooler.1
> -rw-------  1 root         root          0 Jan 25 04:02 spooler.3
> -rw-------  1 root         root          0 Jan 11 04:03 spooler.4
> 
> -rw-------  1 root         root          0 Jun 24  2008 tallylog
> --------------------------------------------------------------------------
> 
I've had something similar happen a couple of times after an update.  In 
my case the /etc/services file got it's security context clobbered when 
some package tried to update it's contents.  When logrotate ran, the 
syslog daemon couldn't open /etc/services because of the error and I 
ended up with a bunch of empty log files.

The quickest way to check for this is the command:

     restorecon -v /etc/services

If nothing prints out in response, that's not the problem.  If it DOES, 
that might explain it.  I have been checking the contexts occasionally 
to try and trap exactly when it happens.  I use:

     restorecon -R -n -v /etc

which walks through the entire /etc tree looking for contexts to change 
but just reports any exceptions.

Just a thought!
-- 
Jay Leafey - Memphis, TN
jay.leafey at mindless.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5566 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20090211/3a9a6135/attachment.bin>

Marcelo,

I didn''t see open file for /var/log/messages.

Here is the screen output:
=======================================================================[root at
sun ~]# lsof /var/log/*
COMMAND  PID USER   FD   TYPE DEVICE    SIZE     NODE NAME
syslogd 2001 root    1w   REG    8,6   25323 17653325 /var/log/secure
syslogd 2001 root    2w   REG    8,6  117544 17653330 /var/log/maillog
syslogd 2001 root    3w   REG    8,6   13674 17653414 /var/log/cron
syslogd 2001 root    4w   REG    8,6       0 17653337 /var/log/spooler
syslogd 2001 root    5w   REG    8,6       0 17653412 /var/log/boot.log
syslogd 2001 root    9w   REG    8,6  166980 17653308 /var/log/syslog
acpid   2418 root    1w   REG    8,6   10758 17653423 /var/log/acpid
acpid   2418 root    2w   REG    8,6   10758 17653423 /var/log/acpid
python  3040 root    3w   REG    8,6 1187439 17653317 /var/log/denyhosts
python  3040 root    5r   REG    8,6   25323 17653325 /var/log/secure
[root at sun ~]# ps aux | grep syslog
root      2001  0.0  0.0   1720   604 ?        Ss   Feb12   0:00 syslogd -m 0 -r
 -x
root      8249  0.0  0.0   3908   660 pts/0    R+   07:04   0:00 grep syslog
================================================================================

Frank


________________________________
From: Marcelo Roccasalva <marcelo-centos at irrigacion.gov.ar>
To: CentOS mailing list <centos at centos.org>
Sent: Friday, February 13, 2009 5:35:51 AM
Subject: Re: [CentOS] logs such as messages, boot.log, and kernel contained 0 
size

On Thu, Feb 12, 2009 at 11:02 PM, Frank Ling <frankling77 at yahoo.com>
wrote:
> Hi Marcelo,
>
> Thanks for the comment. I had SELinux disabled. Anyway I tried your trick,
> and it didn''t work. Something must went wrong.
Are the files opened?:

# lsof /var/log/*

Can you "strace" the [syslog] pid?

-- 
Marcelo

"?No ser? acaso que ?sta vida moderna est? teniendo m?s de moderna que
de vida?" (Mafalda)
_______________________________________________
CentOS mailing list
CentOS at centos.org
http://lists.centos.org/mailman/listinfo/centos



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.centos.org/pipermail/centos/attachments/20090213/a7ec86af/attachment.html

On Fri, Feb 13, 2009 at 11:09 AM, Frank Ling <frankling77 at yahoo.com>
wrote:
> Marcelo,
>
> I didn''t see open file for /var/log/messages.
Have a look at your /etc/syslog.conf

-- 
Marcelo

"?No ser? acaso que ?sta vida moderna est? teniendo m?s de moderna que
de vida?" (Mafalda)

Here is my /etc/syslog.conf:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don''t log private authentication messages!
*.info;*.!warn;authpriv.none;cron.nome;mail.none;       -/var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*;mail.!err                                        -/var/log/maillog
mail.err                                               
-/var/log/mail.err*.info;*.!warn;authpriv.none;cron.nome;mail.none;      
-/var/log/messages

# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*..emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

#
# INN
#
news.=crit                                        /var/log/news/news.crit
news.=err                                         /var/log/news/news.err
news.notice                                       /var/log/news/news.notice

*.warn;authpriv.none;cron.none;mail.none;         -/var/log/syslog
*.kern                        /var/log/kernel
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Frank



________________________________

From: Marcelo Roccasalva <marcelo-centos at irrigacion.gov.ar>
To: CentOS mailing list <centos at centos.org>
Sent: Monday, February 16, 2009 5:59:35 AM
Subject: Re: [CentOS] logs such as messages, boot.log, and kernel contained 0 
size

On Fri, Feb 13, 2009 at 11:09 AM, Frank Ling <frankling77 at yahoo.com>
wrote:
> Marcelo,
>
> I didn''t see open file for /var/log/messages.
Have a look at your /etc/syslog.conf

-- 
Marcelo

"?No ser? acaso que ?sta vida moderna est? teniendo m?s de moderna que
de vida?" (Mafalda)
_______________________________________________
CentOS mailing list
CentOS at centos.org
http://lists..centos.org/mailman/listinfo/centos



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.centos.org/pipermail/centos/attachments/20090216/f6e556aa/attachment.html

Frank Ling <frankling77 at yahoo.com>
> *.info;*.!warn;authpriv.none;cron.nome;mail.none;       -/var/log/messages
I guess you alread tried to restart syslog.

From the manpage:
"You may prefix each entry with the minus ??-?? sign to omit syncing the
file after every
 logging.   . . ."

Maybe try to remove the ''-'' and restart syslog...

JD