Hello all: I have a couple CentOS 4 servers (all up-to-date) that are having strange command failures. I first noticed this with a perl script that uses lots of system calls. Basically, sometimes a command just won't run: thoth(52) /tmp> ls thoth(53) /tmp> ls thoth(54) /tmp> ls thoth(55) /tmp> ls learner lost+found/ thoth(56) /tmp> ls learner lost+found/ thoth(57) /tmp> ls learner lost+found/ thoth(58) /tmp> ls learner lost+found/ thoth(59) /tmp> ls learner lost+found/ thoth(60) /tmp> ls learner lost+found/ thoth(61) /tmp> ls learner lost+found/ thoth(62) /tmp> ls thoth(63) /tmp> ls thoth(64) /tmp> ls thoth(65) /tmp> ls thoth(66) /tmp> uname -a Linux thoth.ssc.wisc.edu 2.6.9-67.0.7.ELsmp #1 SMP Sat Mar 15 06:54:55 EDT 2008 i686 i686 i386 GNU/Linux Nothing in either dmesg or /var/log/messages seems to indicate any problems. It also doesn't seem to matter what the command is -- ls is the quickest test, but sshd will sometimes to fail to spawn children, etc. There aren't a large amount of processes on the machine either -- only 122 at the moment. Has anyone seen this behavior before? Have I been hit with some sort of cunning rootkit? This machine shouldn't be publicly accessible; it's behind our firewall. Thanks. -- Dan Bongert dbongert at wisc.edu
On Mon, Mar 24, 2008, Dan Bongert wrote:>Hello all: > >I have a couple CentOS 4 servers (all up-to-date) that are having strange >command failures. I first noticed this with a perl script that uses lots of >system calls. > >Basically, sometimes a command just won't run: > >thoth(52) /tmp> ls >...> >thoth(66) /tmp> uname -a >Linux thoth.ssc.wisc.edu 2.6.9-67.0.7.ELsmp #1 SMP Sat Mar 15 06:54:55 EDT >2008 i686 i686 i386 GNU/Linux > >Nothing in either dmesg or /var/log/messages seems to indicate any >problems. It also doesn't seem to matter what the command is -- ls is the >quickest test, but sshd will sometimes to fail to spawn children, etc. >There aren't a large amount of processes on the machine either -- only 122 >at the moment.There is a very good chance that the machine has been cracked, and the system's /bin/ls routine replaced by one hacked to hide the cracker's programs. ``rpm -V coreutils procps util-linux'' may well show several critical programs changed. You can also try running ``strace /bin/ls'' to see what is going on. Bill -- INTERNET: bill at celestial.com Bill Campbell; Celestial Software LLC URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way FAX: (206) 232-9186 Mercer Island, WA 98040-0820; (206) 236-1676 When I hear a man applauded by the mob I always feel a pang of pity for him. All he has to do to be hissed is to live long enough. -- H.L. Mencken, Minority Report
Dan Bongert wrote:> Hello all: > > I have a couple CentOS 4 servers (all up-to-date) that are having > strange command failures. I first noticed this with a perl script that > uses lots of system calls. > > Basically, sometimes a command just won't run: > > thoth(52) /tmp> ls > > thoth(53) /tmp> ls > > thoth(54) /tmp> ls > > thoth(55) /tmp> ls > learner lost+found/ > > thoth(56) /tmp> ls > learner lost+found/ > > thoth(57) /tmp> ls > learner lost+found/ > > thoth(58) /tmp> ls > learner lost+found/ > > thoth(59) /tmp> ls > learner lost+found/ > > thoth(60) /tmp> ls > learner lost+found/ > > thoth(61) /tmp> ls > learner lost+found/ > > thoth(62) /tmp> ls > > thoth(63) /tmp> ls > > thoth(64) /tmp> ls > > thoth(65) /tmp> ls > > thoth(66) /tmp> uname -a > Linux thoth.ssc.wisc.edu 2.6.9-67.0.7.ELsmp #1 SMP Sat Mar 15 06:54:55 > EDT 2008 i686 i686 i386 GNU/Linux > > Nothing in either dmesg or /var/log/messages seems to indicate any > problems. It also doesn't seem to matter what the command is -- ls is > the quickest test, but sshd will sometimes to fail to spawn children, > etc. There aren't a large amount of processes on the machine either -- > only 122 at the moment. > > Has anyone seen this behavior before? Have I been hit with some sort > of cunning rootkit? This machine shouldn't be publicly accessible; > it's behind our firewall.where is /tmp mounted? is this an external disk (usb, ...)? is it an nfs mount?