CentOS - Feb 2012 - Configuration Compliance auditing for many CentOS 5.x boxes

Hi CentOS experts,*

Short Version*

I would like to produce a weekly report in HTML for each CentOS 5.x 
server we have indicating configuration compliance with some industry 
benchmark. I am looking for a tool or tools to implement this, I am 
happy to use 3rd party proprietary stuff if necessary.
*
Long(er) Version*

    Current Situation.. I have a client with many (200x) CentOS 5.x 
servers deployed in various web, mail, database and file server roles, 
and these boxes have been variously administrated to a lessor or greater 
degree.

All the boxes have EPEL repository included as part of their 
base-install, and all boxes have cron jobs for "yum -y update" running
frequently, and are rebooted when kernels are available. (so they are 
not in a terrible state)

For network, local and external vulnerabilities - We use a 3rd party 
firm, who use WebInspect to monitor for external facing ports and 
vulnerable services and produce various regular reports to my boss. 
(hence am not looking at Nessus, OpenVAS or network based scanning tools 
right now, or indeed any vulnerability tools)

However we now have a New Big Boss in Town - who is an ex security 
compliance dude. The new rules are; that if its not being regularly 
tested, then its not in compliance, even if it is in compliance etc. (to 
be honest, I quite like that rule)

So now I am looking for a way to generate a report of server compliance 
with some compliance standard for all the boxes regularly.

We have a basic list of configuration settings, that is a weaker form of 
various compliance recommendations, so I am confident that most 
compliance benchmarks like CIS, EAL3 or the linux web STIG level would 
be sufficient.

We have chef installed on the CentOS instances, hence I can push out yum 
based packages, (and I can install from source tarballs, but it will 
make me cry, on these instances)

I Would like to have...  a tool that runs locally on each CentOS box and 
produces a reasonably comprehensive html report regarding configuration 
compliance

(and a massive bonus would be to send email alert for severe problems, 
but I can script that if required)

Ideally I could generate a weekly report that indicates compliance with 
1 or more of the recognised linux server benchmarks. I am happy to pay 
for a subscription for the checklist, but I suspect the kind per 
instance 100 USD licenses I see are going to blow my budget.

Current progress is...

I see that OPENSCAP and OVAL have tools in CentOS-base or EPEL, such as

     OpenSCAP-utils
     ovaldi - oval reference interpreter

Which can be used to create reports. However they seem a little unrefined.

For SCAP and OVAL content I have found the following.

1. NIST provide SCAP content for RHEL desktop, which is kinda close;
2. http://usgcb.nist.gov/usgcb/rhel_content.html
3. There is a tool called sectool in the fedora repos, but I can't get 
it to run on CentOS due to a missing python-slip module.

Any suggestions on functioning stacks for this problem would be helpful.

Thanks,
Tom


ps SORRY FOR THE LONG EMAIL

On Feb 1, 2012, at 2:54 PM, Tom H <tom at limepepper.co.uk> wrote:
> Hi CentOS experts,*
> 
> Short Version*
> 
> I would like to produce a weekly report in HTML for each CentOS 5.x 
> server we have indicating configuration compliance with some industry 
> benchmark. I am looking for a tool or tools to implement this, I am 
> happy to use 3rd party proprietary stuff if necessary.
> *
You could have a weekly cron job on all boxes that does a rpm for all package
config files, diff against a "snapshot" copy contained under /var
somewhere, email those diffs to a change management system,then save the current
files in the snapshot directory.

First run will send the complete configs, all subsequent runs will send the
diffs.

Of course you need a change management system that will hold an inventory of
systems, those systems' hardware/software inventories and configurations,
and track those changes with alerts and reports and such.

I don't know of a good system for doing all that unfortunately, but if you
do find one let me know.

-Ross

On Wed, Feb 1, 2012 at 2:54 PM, Tom H <tom at limepepper.co.uk> wrote:
> Hi CentOS experts,*
>
> Short Version*
>
> I would like to produce a weekly report in HTML for each CentOS 5.x
> server we have indicating configuration compliance with some industry
> benchmark. I am looking for a tool or tools to implement this, I am
> happy to use 3rd party proprietary stuff if necessary.

> [snip]
>
I'm in a similar situation. We have a growing infrastructure of over 300
instances of RHEL4/5/6.  Though not specifically CentOS the tools are the
same. My focus has been on PCI compliance. As of yet we don't have any SOX
systems on Linux, but I expect that will change in the near future.

For PCI compliance there are a few things that we do. The first thing was
to get a handle on the buildout process which we did via kickstart. This
ensured consistency in the builds which previously was done by different
engineers/operators with different skill levels. We validated the standard
image and then used Satellite/Spacewalk to keep track of the versions.

The next step was the daily bit rot and the damage from the application
folks whose sole experience was on desktop or laptop systems (i.e., they
never had to comply with any industry standards). We started by separating
OS from application. This meant not only separate volume groups and mount
points for application files, but also things like ensuring that apps did
not run as root (you'd be amazed how many developers insist that builds
must occur as root).  In just about every case where we allowed application
developers to have root access we ended up with systems that were wildly
out of compliance. In one case a developer installed an entire desktop
suite, including MP3 player and video editing tools, in order to satisfy a
dependency on a single widget library. We don't do that any more. :/

Next was auditing, which I think may apply to your question.

For the basic package setup, Spacewalk or Satellite can track the versions
and allow you to lock the package set. There are also existing scripts that
wrap variations of an 'rpm -qVa' and send the reports back. Tools such
as
tripwire are also useful for this. If you have deployed SELinux, you can
effectively even lock the root user from installing or modifying system
packages.

For the configurations, we are experimenting with cfengine and puppet. They
allow you to track configuration changes, reset changes, etc..  I've also
used CVS to track configuration files directly.  I.e., checkin the changes
onto a logged administration server then have the production servers
checkout the changes on an on-demand or scheduled basis. This minimizes
on-the-fly configurations that accumulate and take the server out of
compliance.  There are tools to generate reports from cfengine/puppet that
show which configurations have changed, etc..

We are also using the perl test harness to run validations. It's pretty
coding intensive so you'd possibly need a Perl developer initially to
create and to maintain the scripts. The idea is to create the test scripts
in lock step with changes to the kickstart. The harness generates a PASS or
FAIL response depending on the Perl test. For example, for PCI compliance
we have a standard login banner. The test does an MD5 sum against the
target machine's /etc/issue.net and checks it against the stored hash. If
the hashes correspond it passes the test (barring hash collisions of course
:D ).

We are still looking at other methods.

> -----Original Message-----
> From: centos-bounces at centos.org [mailto:centos-bounces at centos.org] On
> Behalf Of Tom H
> Sent: Wednesday, February 01, 2012 14:54
> To: CentOS mailing list
> Subject: [CentOS] Configuration Compliance auditing for many CentOS
5.x
> boxes
> 
> Hi CentOS experts,*
> 
> Short Version*
> 
> I would like to produce a weekly report in HTML for each CentOS 5.x
> server we have indicating configuration compliance with some industry
> benchmark. I am looking for a tool or tools to implement this, I am
> happy to use 3rd party proprietary stuff if necessary.
> Current progress is...
> 
> I see that OPENSCAP and OVAL have tools in CentOS-base or EPEL, such
as
> 
>      OpenSCAP-utils
>      ovaldi - oval reference interpreter
> 
> Which can be used to create reports. However they seem a little
> unrefined.
> 
> For SCAP and OVAL content I have found the following.
> 
> 1. NIST provide SCAP content for RHEL desktop, which is kinda close;
> 2. http://usgcb.nist.gov/usgcb/rhel_content.html
> 3. There is a tool called sectool in the fedora repos, but I can't get
> it to run on CentOS due to a missing python-slip module.
> 
> Any suggestions on functioning stacks for this problem would be
> helpful.
Sorry about no suggestions, but seeing where you are I have a question
back at you:
The http://usgcb.nist.gov/usgcb/rhel_content.html seemed to me to be a
newer schema than the openscap in RH/CentOS 5, did you find a way to run
it on 5?

And I sort of assume you have seen 
http://www.redhat.com/security/data/oval/?C=M;O=D
for the RHEL boxes...

Thanks for any pointers.