CentOS - Nov 2008 - centralized logs server and also storing the logs on the local server

Hi Friends,

I am running most of my company's Linux Servers on Centos 4.x/5.x 32
and 64-bit. I am now trying to configure a centralized logging server
where logs of all the linux servers will be stored and also I want to
store all the logs on the local server aka means logs will be sent to
the central log server but also will be stored on the local server.
The reason for storing the logs locally is because we have offices in
different cities and few more offices are coming up and it is good to
store the logs locally so that when the connectivity b/w the offices
break the logs does not get lost. There are lots of configuration
available on internet which tells how to send the logs to the
centralized log server but I did not find any configuration where logs
can be stored locally as well as send to the centralized log.


Moreover I am also looking for logs analyzer tool which can generate
reports separately for each host for ex there are logs of 15 servers
are stored on the server and this logs analyzer tool should generate
reports separately for each host.


Thanks & Regards

Ankush

On Sun, Nov 9, 2008 at 8:57 AM, ankush grover <ankushcentos at gmail.com>
wrote:
> store all the logs on the local server aka means logs will be sent to
> the central log server but also will be stored on the local server.
I see two ways of doing this: 1) use rsyslog and multiple rules, one
for writing to local file and another for TCP forwarding to central
syslog server. 2) Do the logging locally, and let logrotate copy the
files to the central location.

BR Bent

Ankush,

--On 9. November 2008 13:27:42 +0530 ankush grover <ankushcentos at
gmail.com>
wrote:
> Hi Friends,
>
> I am running most of my company's Linux Servers on Centos 4.x/5.x 32
> and 64-bit. I am now trying to configure a centralized logging server
> where logs of all the linux servers will be stored and also I want to
> store all the logs on the local server aka means logs will be sent to
> the central log server but also will be stored on the local server.
> The reason for storing the logs locally is because we have offices in
> different cities and few more offices are coming up and it is good to
> store the logs locally so that when the connectivity b/w the offices
> break the logs does not get lost. There are lots of configuration
> available on internet which tells how to send the logs to the
> centralized log server but I did not find any configuration where logs
> can be stored locally as well as send to the centralized log.
If you use rsyslog from base repo that is easy to achieve. Rsyslog replaces 
syslog; thus you can copy your existing syslog.conf into rsyslog.conf and 
just add an enty for sending everything to the central log host 
additionally, like e.g.
	*.*	@@your.loghost.tld

On the loghost you switch to rsyslog as well and open it up for remote 
logging in /etc/sysconfig/rsyslog (it is documented there). Be aware to 
upen up your loghost's local firewall, if you use it.

On of the advantages of using rsyslog is that on the central loghost you 
can stuff everything in a mysql database (again additionally to plain log 
files) quite easy - that makes automated evaluation of log entries much 
more convenient.
>
>
> Moreover I am also looking for logs analyzer tool which can generate
> reports separately for each host for ex there are logs of 15 servers
> are stored on the server and this logs analyzer tool should generate
> reports separately for each host.
What exactly do yo want the analyzer to do? The simplest thing would be to 
use logwatch on the servers and customize it, but that depends on what kind 
of analyzation you have in mind.

Dirk

ankush grover wrote:
> Hi Friends,
> 
> I am running most of my company's Linux Servers on Centos 4.x/5.x 32
> and 64-bit. I am now trying to configure a centralized logging server
> where logs of all the linux servers will be stored and also I want to
> store all the logs on the local server aka means logs will be sent to
> the central log server but also will be stored on the local server.
> The reason for storing the logs locally is because we have offices in
> different cities and few more offices are coming up and it is good to
> store the logs locally so that when the connectivity b/w the offices
> break the logs does not get lost. There are lots of configuration
> available on internet which tells how to send the logs to the
> centralized log server but I did not find any configuration where logs
> can be stored locally as well as send to the centralized log.
If you don't mind being somewhat behind on the central copies, the 
simple-minded way is to run a scripted rsync nightly to pull in whatever 
you want from the remote sites.  For the ones that logrotate renames, 
you'll have to adjust accordingly - or fix it so the name always 
contains the date.
> Moreover I am also looking for logs analyzer tool which can generate
> reports separately for each host for ex there are logs of 15 servers
> are stored on the server and this logs analyzer tool should generate
> reports separately for each host.
Analog is very versatile for web logs.  But any tool should be able to 
accept command line options for the files and output location so you can 
script the runs you want.  If you want to go crazy with reporting and 
analysis, look at the tools from
http://community.pentaho.com/index.php and figure out how to get the log 
fields into a database.

-- 
   Les Mikesell
    lesmikesell at gmail.com