I''d like to thank respondents for their useful advice and information -
I was able to recover from the intrusion with some grace and I was also
able to pass on accurate information to another respondent who was in
acute need of it. In time-honoured tradition - and in view of the
variety of experiences of bind-based attacks - I have summarised my
experience and respondents'' comments and am posting the summary back to
the list. Where respondents confirmed they too experienced attacks,
I have respected this confidence and have elided the name.
(I was a dilbert for 12 years at Hewlett-Packard Labs in Bristol before
going freelance and almost immediately got caught by an imapd buffer
overflow intrusion, the horse laugh from ex-colleagues nearly deafened
me).
In summary:
-----------
The appearance of a directory "ADMROCKS" is characteristic of an
exploit
of a known buffer overflow vulnerability in bind, the DNS name daemon.
The exploit has been discussed on the SecurityFocus Incidents email list
(LISTSERV@LISTS.SECURITYFOCUS.COM, "subscribe incidents") and the C
source for the exploit is available on:
http://packetstorm.securify.com/9911-exploits/adm-nxt.c
Opinion is divided whether bind-8.2.2-P3 (patchlevel 3) is vulnerable,
to this exploit. The general recommendation is to upgrade to
bind-8.2.2-P5 (patchlevel 5) which appears at the moment not to be
vulnerable to this exploit, although rumour may contest this.
Information on bind is available from: http://www.isc.org/products/BIND/
rpm packages are available via anonymous ftp from updates.redhat.com,
rawhide.redhat.com and (I can attest) from the redhat mirror on
ftp://src.doc.ic.ac.uk/pub/Mirrors/ftp.redhat.com/rawhide
As regards the robustness of the bind-8.2.2-P3 rpm listed in
RHSA-1999:054-01: I had installed this package but have no memory of
whether I actually stopped and started the daemon or merely sent it a(n
ineffectual) SIGHUP. I was sent a copy of the exploit C source (t666.c,
posted several days after the P3 advice from Redhat) but it''s been
deliberately broken and I can''t get it to work, otherwise I''d
have
tested it on P3 & P5 on my offline duplicate machine.
My experience:
--------------
I seem to have escaped lightly this time, partly due to the layered
security on my machine. Tripwire''s MD5 checksum matches okay with the
one on the offline duplicate. It and an rpm -V revealed nothing amiss,
with the exception of MD5 checksum mismatches for the openssl binary
(and a significant change in file size) and libcrypto.a (same filesize,
MD5 checksum differs) but a ''strings'' did not appear to reveal
anything
suspicious - although I replaced them both, just to be sure. The changed
files may have been a result of my activities - I had omitted to include
the ssl directory in the tripwire config (tsk, tsk). My eth0 was in
promiscuous mode - but that was probably a hang-over from my activities
during a recent wrangle with my ISP over a dismally slow Ethernet
performance (which turned out to be due to a dodgy cable), no
unexpected log files were revealed by (an uncompromised) find.
In this intrusion, the ADMROCKS directory appeared in /var/named which
is where the DNS databases are stored on my system. The directory was
completely empty. After satisfying myself of the integrity of the key
system binaries and library files (via tripwire and rpm -V), I used
''find / -newer /var/named/ADMROCKS -type f'' to probe for
recently-created files but found nothing untoward. A port scan with nmap
found nothing untoward nor were there any changes made to
/etc/inetd.conf, /etc/shadow or /etc/group.
My security setup:
------------------
My ISP''s firewall blocks all telnet connections except for their own
dial-up accounts. For emergencies, I run an s/key-enabled login which
only permits two specified uids to login via telnet and forces S/Key
authentication for both. For normal system admin, I connect with ssh
using a cert which effectively means the root password for my Linux
server never leaves my Macintosh. The S/Key arrangement means that the
root password which appears in /etc/shadow is never actually consulted
as s/key holds its encrypted passwords in another file entirely. inetd
services are stripped down to an absolute minimum, as recommended. I
have a PGP-encrypted, statically-linked version of ps which I keep
stashed away in a corner of the system - it was actually built from the
ps distributed as part of rootkit -- and it gives me an unsubverted list
of the processes currently running on the system. I run tripwire and
logcheck and used COPS + SATAN for pre-installation checking. I don''t
run a firewall.
What I learned:
---------------
I found that having tripwire installed was immensely helpful (in
conjunction with rpm -V) in concluding that my system had been at best
only minimally compromised, if at all. However, the tripwire binary can
be subverted in order to conceal unauthorised activity, so I thought I
would make use of the fact that I have a number of cron-driven Python
scripts which perform janitorial functions for several virtual host
websites. Seeing as the CPU is seriously underemployed, I added md5
checks of the tripwire binary and its database to one of the Python
scripts and set it to email me if either md5 checksum changes. As this
script runs every 15 mins, I get both early warning of possible
compromise and (in the absence of an alert), some degree of confidence
in tripwire''s continued integrity. When I get round to setting up a
CMUCL-driven CLHTTP, I can add further checks - it''s going to be a rare
intruder who can dig into Common Lisp byte-compiled files to find and
disable embedded system integrity checks.
As part of the recovery procedure, I checked several security-advice
websites and discovered that the CERT recommendation documents have aged
considerably and do not discuss the latest trend of agent-based
denial-of-service attacks.
There is some concern that the recent attacks are part of a preliminary
exercise aimed at recruiting a large number of compromised
''agent''
machines for a coordinated, large-scale denial-of-service attack. This
hypothesised attack could involve several hundred, or possibly several
thousand compromised machines which, acting in concert, could present a
major D-o-S threat. A recent CERT advisory on Bugtraq warns:
A distributed denial-of-service tool called "Stacheldraht" has
been
discovered on multiple compromised hosts at several organizations. In
addition, one organization reported what appears to be more than 100
different connections to various Stacheldraht agents. At the present
time, we have not been able to confirm that these are connections to
Stacheldraht agents, though they are consistent with an analysis
provided by Dave Dittrich of the University of Washington, available
at http://staff.washington.edu/dittrich/misc/stacheldraht.analysis
I checked my system for the presence of Stacheldraht agents with Dave
Dittrich''s ''gap''. The system is apparently free of
Stacheldraht agents
but absence of evidence is not evidence of absence.
Looks like this could be a good year to make sure that one''s intrusion
and D-o-S defences are robust. To this effect, I added
''portsentry'' to
my armoury (http://www.psionic.com/abacus/portsentry/) to see how we
get on.
Sysadmins of RedHat 6.x systems may wish to check the validity of their
binaries, a useful check is: for i in $(rpm -qa);do rpm -V $i;done
Respondents comments:
----------------------------------------------------------------------
----------
Antonio reports a Rootkit-based experience which saw changed binaries
such as inetd, in.fingerd, ps, ftpd and inetd.conf.
Antonio recommends examining /usr/sbin for executables which run during
startup and put eth0 into promiscuous mode to listen in on the local
network for passwords and creates a logfile recording the captured data.
----------------------------------------------------------------------
----------
8LGM was running bind-8.2.2_P3-1 and saw this attack on Dec 28 1999.
Having given up waiting on RedHat to release bind-8.2.2_P5, 8LGM
downloaded the scr. and comp, then got the attack tool that was used and
tested it on the servers - which now seem fine once again.
( ftp.isc.org for the latest bind-8x )
He notes that the attacker changed some of the system files, e.g. ls,
ps, netstat, login, df, sendmail, in.pop3, and named itself, so it is
worth checking the system for change date system files:
"ls -algctr /usr/sbin, /bin, /sbin, /usr/bin, /dev"
----------------------------------------------------------------------
----------
Tom Brown enquired whether I was sure that I was running the updated
version? i.e. that I rebooted after I did the rpm -U bind or executed a
''ps aux'' to make sure that the process had been stopped and
restarted.
Tom suggests that someone really did overflow the new daemon, the
isc.org folks would probably _really_ like to see any coredump you might
have floating around... (but then, if it was a successful overflow,
there probably wasn''t a segfault/coredump).
----------------------------------------------------------------------
----------
George noticed the hard way and now runs named as nobody in a chroot.
His system was attacked on Dec 11 and he reports that the issue has been
discussed on the SecurityFocus Incidents list. On his particular
machine, the intruders installed a trojan sshd with the alternate
password of ''phearsome3''. The fact that sshd was an i686
binary when
his machine was an i586 and he knew that he had compiled it himself was
a dead giveaway. Other than that, they left him alone. Other messages
from the incidents list indicate the replacements vary.
George notes that there''s a newer BIND RPM from rawhide but one person
on the incidents list mentioned he heard a rumor that even P5 might be
vulnerable without being in a chroot and suggests that I check the
archives there for more information.
----------------------------------------------------------------------
----------
J.Lewis enquired whether I was sure that I not only installed the new
bind RPM, but also did either ndc restart or /etc/rc.d/init.d/named
restart after installing it.
----------------------------------------------------------------------
----------
Dan warns me to take a look through the logs, he just found
/etc/named/AMDROCKS and in his logs found some identd messages at a time
he knew no legit users were logged in:
Jan 3 03:39:41 base identd[5807]: Connection from 209.148.161.242
Jan 3 03:39:41 base identd[5807]: from: 209.148.161.242
(209.148.161.242 ) for: 2823, 21
Jan 3 03:39:43 base identd[5808]: Connection from 209.148.161.242
Jan 3 03:39:43 base identd[5808]: from: 209.148.161.242
(209.148.161.242 ) for: 2823, 21
He also had a ''...'' dir in the httpd home dir from the same
time period.
The contents of this dir are:
drwxrwxr-x 2 root root 1024 Jan 3 03:31 .
drwxr-xr-x 6 root root 1024 Jan 3 03:31 ..
-rwsr-xr-x 1 root root 13344 Jan 3 03:31 gH.cgi
and gH.cgi is:
gH.cgi: setuid ELF 32-bit LSB executable, Intel 80386,
version 1, dynamically linked (uses shared libs),
not stripped
He remarks that he may be lucky in that this dir is not in the document root
on his machine.
strings on the binary presented him with some very interesting
info:
<ISINDEX prompt="Command to Execute: ">
<br><b>Command output:</b> [<em>%s</em>]
He runs named as user nobody on his Solaris machines and is now
switching to an unprivileged user (bind) on his Linux
machines. He is not using nobody on the Linux machines as
they also run web servers under nobody and a coworker
suggested that this may be a "bad thing." He thanked me for
mentioning this problem and reminding him to do this!
----------------------------------------------------------------------
----------
Paul Hart confirms that my machine has been root compromised. He
identifies the culpable exploit:
http://packetstorm.securify.com/9911-exploits/adm-nxt.c
One of the things the assembly code in this exploit does is create a
directory in the current directory named "ADMROCKS" that it uses when
breaking out of any possible chroot environments.
Paul suggested I read the CERT guidelines at:
http://www.cert.org/tech_tips/root_compromise.html
for tips on responding to the compromise.
----------------------------------------------------------------------
----------
Chuck Mead''s summary of the situation records that ISC has said that
patch level 5 is where one needs to be to stop the exploits. Bugtraq and
other reports of bind related scans and cracks occuring are on the rise
with two known cracks being reported on inet-access over the last five
days (and both were multiple cracks). The latest version of bind from
rawhide is at patch level 5) and installation of this version is
recommended. It appears to be stable and to work fine.
Chuck has built the src.rpm for i586 and i686 and make the original i386
rpms available via anonymous ftp on
ftp://server.moongroup.com/pub/bind-update. Or one can get them from
rawhide.redhat.com.
----------------------------------------------------------------------
----------
James experienced the BIND attack on an allegedly fixed in 8.2.2_P3-1
but on that occasion he did not see the same directory (it was called
something else) although little intrusion occurred. That has been fixed
and now he has experience no further problems since the 8.2.2_P3-1
binaries installed from rpmfind.net on his affected RH6.0 system.
James notes that his server is monitored by human eyes 24/7 just to be
sure and he (and colleagues) have noticed a surprising (to him anyway)
amount of suspect traffic but most of it is blocked by the firewall.
----------------------------------------------------------------------
----------
Greg confirms that he has also seen this attack and recommends that I
check /etc/inetd.conf for an additional entry that spawns a root shell.
He also suggests running a quick rpm binary check for any unexplained
changes.
for i in $(rpm -qa);do rpm -V $i;done
----------------------------------------------------------------------
----------
Keith Owens notes that the incidents list
(LISTSERV@LISTS.SECURITYFOCUS.COM, "subscribe incidents") is
discussing
this at the moment and that ADMROCKS is part of the t666.c rootkit.
----------------------------------------------------------------------
----------
Steve Jackson notes that since November he has been getting lame
attempts to connect to bind, but he has it blocked, although the
attempts are logged. His experience is that usually this means he will
see (and has just seen) a security alert soon, so he thinks I
experienced an intrusion.
Steve suggested I check the password and shadow and group files for
entries that have user id as 0.
----------------------------------------------------------------------
----------
Daniel confirms that his primary ns was attacked the exact same way a
couple of weeks ago, and the intruder arranged for connections to port
1337 to spawn a root shell. Nothing untoward appeared in the system logs
for him either.
Daniel is now running bind chrooted in hope of little less deadly
consequences in the case of subsequent successful intrusions. Daniel
wasn''t running tripwire, so decided that the safest course was to
reinstall the system.
----------------------------------------------------------------------
----------
Craig Rowland confirms that the "ADMROCKS" directory is created by the
ADM exploit code to break chroot() if run as root and he concludes that
my system may be compromised.
He has been seeing a large number of TCP probe attempts to his DNS
servers but hasn''t confirmed if it is a new exploit (he blocks
everything with filters). All he can confirm is that someone is looking
for a hole, he just doesn''t know if it is a new exploit or not.
----------------------------------------------------------------------
----------
Randy Church requested more precise information - I wrote that I went in
to the named directory, Randy asks: "Which directory? /var/named?" -
yes.
----------------------------------------------------------------------
----------
Neil Long notes that something just appeared on Incidents - t666.c is an
exploit for named and was just posted there. Neil wasn''t sure whether
it was relevant but he thought it likely to be the exploit I experienced
- or a similar exploit.
----------------------------------------------------------------------
----------
Dave Dittrich forwarded on Chris McNab''s posting (below)
----------------------------------------------------------------------
----------
Chris McNab recently subscribed to the Incidents list and saw the
discussion about ADMROCKS.
Chris confirms that ADMROCKS is a directory created by t666.c when it
breaks chroot. Apparently, this exploit has been publicly available for
some time, available from ADM and Chris kindly sent the C source code as
an attachment. Chris identifies the precise string (ADMROCKS) in the
shellcode (0x41,0x44,0x4d,0x52,0x4f,0x43,0x4b,0x53 == ADMROCKS).
Chris notes that the ADM exploit is all old news and recommends that I
upgrade the bind code and remember to restart it.
----------------------------------------------------------------------
----------
Eric reports a similar intrusion. However, after he upgraded to the
newer RPM, he saw the intruder return for several more attempts, each
one unsuccessful.
Eric appended his notes from his intrusion, noting that the intrusion
into my system may have had different results.
-------------------------------------------------
Here''s the list of stuff Eric found and what it does:
Replaced system utils:
/bin/login (file was also made immutable)
/bin/ls
/bin/ps
/usr/bin/du
/usr/bin/netstat
/usr/bin/syslogd
Modified config files:
/etc/services (added "working" service on port 1120)
/etc/inetd.conf (added "working" service that runs in.telnetd)
Added executables:
/usr/sbin/cronlogd
/usr/bin/xcat
/dev/portd/zap2
Added files:
/dev/.addro
/dev/ptyu
/dev/ptyv
/dev/ptyy
/dev/portd/.log
/dev/portd/.pid
Much of this seems to appear in the "lrk4" linux root kit. From the
directions on usage of lrk4 I was able to figure out what some of the
stuff left behind was.
zap2 is a utmp/wtmp/lastlog eraser so you can hide yourself from w/who
etc.
The file "/dev/.addro" was the config file for the trojaned netstat.
It
was configured to hide any connections from the following IP addresses:
192.117.xxx.xxx
139.92.xxx.xxx
207.91.xxx.xxx
The file "/dev/ptyu" was a config file for the trojaned "ps"
and tries
to hide programs with the following names: cronlogd, synk5, jess, smurf,
ipzoner, imapdx, namedx.
The file "/dev/ptyv" was a config file for the trojaned syslogd. They
don''t seem to have understood how to use it correctly, as it tries to
filter the following strings:il net edu com org.
The file "/dev/ptyy" was a config file for the trojaned "ls"
and "du"
and tries to hide the following filenames: .addro .log .pid portd ptyu
ptyv ptyy.
from running "strings cronlogd" I am guessing that cronlogd is a
packet
sniffer, unless the string "can''t set promiscuous mode" was
left there
as a red herring. It was using the file "/dev/portd/.log" to log the
sniffed traffic. It contained logs (including username/password) from
xxxx, xxxx, xxx, and xxxxx doing ftp or telnet sessions.
"/dev/portd/.pid" contained the process id (25560) of the sniffer. I
have no idea what the pid would be useful for, except maybe to tell the
intruder that it ran successfully.
"/usr/bin/xcat" is our original copy of "/bin/login" !
After the
trojaned version executes, it runs this one to let you really log in.
Cool. I have not found a log file from the trojaned /bin/login, so it
may only have been backdoored, not logging passwords. The lrk4 login
works this way (no logging).
I''m pretty sure that''s everything.
----------------------------------------------------------------------
----------
Steve confirms that he has also seen this exploit and notes that the
exploit is effected via a buffer overflow on the named daemon and
results in the intruder being able to run adduser to add a new user.
Steve suggests I check to see if there is a new user. In the attack on
his system the intruder replaced the tripwire binary, placed several
other binaries, created a directory called "ADMROCKS" and a hidden
zone
file (.something he believes). The intruder was traced to dialups
fraudulently set up by youths in the Houston area, although no
prosecutions were made.
----------------------------------------------------------------------
----------
Philipp Buehler confirms that the presence of the ADMROCKS directory is
a sign for an attacked bind. ADMROCKS is created [even] if the named was
in chroot. He notes that it is possible that my system is compromised.
----------------------------------------------------------------------
----------
Stephen Bickle enquires whether anything was left behind in the ADROCKS
directory and observes that if named spawned a "root" shell I might be
able to get history to show activity. Stephen also enquired whether
that was the only IP address that shows in any of the logs and asks
whether I have ftp setup to log xfers?
That was the only IP address communicated by logcheck, ftp transfers show
only the usual activity (mostly downloads of PyApache).
----------------------------------------------------------------------
----------
Chris Cogdon reminds me that it is worth bearing in mind that its
perfectly normal for a hackers program, or a hacked existing program, to
put its temporary directories in an unrelated directory.
This way, the sysadm may be fooled into attempting to fix the security
hole by recompiling, in the above case, named where the actual
vulnerability is elsewhere.
----------------------------------------------------------------------
----------
Jim confirms that he was attacked with something very similar perhaps.
Several days ago he discovered that something or someone had erased
/var/log/* (except the ftp logs, alert, and cron) and /bin/netstat.
Other than that, nothing was touched. Tripwire showed nothing new
either. In fact, other than that the only odd thing is that named was
down.
Jim notes that this reminded him of the really old named exploits where
remote arbitrary commands could be run on a vulnerable machine running
less than 8.x of bind. This last version he was running was 8.2.2 (no
patchlevel).
Since then Jim has upgraded to bind P5 and is running it as a non-root
user and chastises himself for not doing that before. He hasn''t had
any
issues since.
----------------------------------------------------------------------
----------
Jon reported a similar experience and a similar hardware/software setup.
He has seen half a dozen messages either on INCIDENTS or VULN-DEV at
securityfocus where people have had the same thing happen. Jon is also
concerned that P3 does not fix the problem and intends to try bind P5,
chrooted and running as a non-root
user.
----------------------------------------------------------------------
----------
Shawn observes that bind 8.2.2p5 is out (and has been for a while). He
seems to remember that there were still some problems with p3 and p4,
but they were fixed with p5.
http://www.isc.org/products/BIND/
----------------------------------------------------------------------
----------
James Griffin enquired whether I let anyone at Intel know that they may
have a compromised machine? He got some interesting results with dig.
----------------------------------------------------------------------
----------
Alan Mead kindly forwarded a posting to redhat-list by Chuck Mead who notes:
"Recently there have been a pile of RH6.x boxen cracked by various,
nefarious, persons. The general consensus appears to be that Bind is the
culprit so far as the exploited weakness goes. I can''t say for sure
myself as I have not been cracked though I do have some folks sniffing
around port 1080 on my firewall lately."
----------------------------------------------------------------------
----------
Carric has seen the same attack. He had to rebuild a box because of it,
because nothing he did to lock it down seemed to work (even updating the
binary). After a complete rebuild (and making an effort to actually secure
the box this time), the problem went away.
----------------------------------------------------------------------
----------
Thank you all again for your advice and information.
Cheers,
Graham Higgins
--------------
Bel EPA Bristol, UK.
http://bel-epa.com