(Tried sending this before but it doesn't look like it went through;
apologies if you're seeing it twice.)
OK, a second machine hosted at the same hosting company has also apparently
been hacked. Since 2 of out of 3 machines hosted at that company have now
been hacked, but this hasn't happened to any of the other 37 dedicated
servers that I've got hosted at other hosting companies (also CentOS, same
version or almost), this makes me wonder if there's a security breach at
this company, like if they store customers' passwords in a place that's
been hacked. (Of course it could also be that whatever attacker found an
exploit, was just scanning that company's address space for hackable
machines, and didn't happen to scan the address space of the other hosting
companies.)
So, following people's suggestions, the machine is disconnected and hooked
up to a KVM so I can still examine the files. I've found this file:
-rw-r--r-- 1 root root 1358 Oct 21 17:40 /home/file.pl
which appears to be a copy of this exploit script:
http://archive.cert.uni-stuttgart.de/bugtraq/2006/11/msg00302.html
Note the last-mod date of October 21.
No other files on the system were last modified on October 21st. However
there was a security advisory dated October 20th which affected httpd:
http://mailinglist-archive.com/centos-announce/2011-10/00035-CentOSannounce+CESA20111392+Moderate+CentOS+5+i386+httpd+Update
https://rhn.redhat.com/errata/RHSA-2011-1392.html
and a large number of files on the machine, including lots of files in */
usr/lib64/httpd/modules/* and */lib/modules/2.6.18-274.7.1.el5/kernel/* ,
have a last-mod date of October 20th. So I assume that these are files
which were updated automatically by yum as a result of the patch that goes
with this advisory -- does that sound right?
So a couple of questions that I could use some help with:
1) The last patch affecting httpd was released on October 20th, and the
earliest evidence I can find of the machine being hacked is a file dated
October 21st. This could be just a coincidence, but could it also suggest
that the patch on October 20th introduced a new exploit, which the attacker
then used to get in on October 21st?
(Another possibility: I think that when yum installs updates, it
doesn't actually restart httpd. So maybe even after the patch was
installed, my old httpd instance kept running and was still vulnerable? As
for why it got hacked the very next day, maybe the attacker looked at the
newly released patch and reverse-engineered it to figure out where the
vulnerabilities were, that the patch fixed?)
2) Since the */var/log/httpd/* and /var/log/secure* logs only go back 4-5
weeks by default, it looks like any log entries related to how the attacker
would have gotten in on or before October 21st, are gone. (The secure*
logs do show multiple successful logins as "root" within the last 4
weeks,
mostly from IP addresses in Asia, but that's to be expected once the
machine was compromised -- it doesn't help track down how they originally
got in.) Anywhere else that the logs would contain useful data?
