freebsd security - Oct 2003 - /var partition overflow (due to spyware?) in FreeBSD default install

All:

I'm posting this to FreeBSD-security (rather than FreeBSD-net) because 
the problems I'm seeing appear to have been caused by spyware, and 
because they constitute a possible avenue for denial of service on 
FreeBSD machines with default installs of the operating system.

Several of the FreeBSD machines on our network began to act strangely 
during the past week. Some have started to refuse mail; in other cases, 
important daemons have died without warning. All of the machines are 
running 4.x releases of FreeBSD with all recent patches installed, and 
all are running the version of BIND supplied with FreeBSD. The "top" 
command, when run on these machines, showed that BIND is consuming very 
large amounts of CPU time, but this by itself couldn't explain all of the 
symptoms we were seeing.

This afternoon, I examined the machines and discovered the problem: full 
/var partitions caused by huge /var/log/messages files.

Inspection of the files reveals hundreds of thousands of messages of the form:

Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
(ns0.opennic.glue)
Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
(ns1.opennic.glue)
Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
(ns3.opennic.glue)
Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
(ns4.opennic.glue)
Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
(ns6.opennic.glue)
Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
(ns7.opennic.glue)
Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
(ns8.opennic.glue)
Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
(ns11.opennic.glue)
Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
(ns10.opennic.glue)
Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
(ns11.opennic.glue)

The references to OpenNIC have caused me to suspect (though I have not 
verified it yet) that the problem is due to the New.Net spyware, which 
causes Windows machines to query OpenNIC's name servers. From what I've 
read so far, it appears that New.Net is "foistware" -- that is, it can
be
installed on innocent users' Windows machines without their consent via 
holes in Internet Explorer. But if New.Net is not what's responsible, 
SOMETHING certainly seems to be generating bogus DNS queries, which in 
turn are causing these messages.

FreeBSD currently comes configured, in the default install, to check 
/var/messages only once a day, and to rotate the log file if it's above a 
certain size. Unfortunately, these messages accumulate so rapidly that 
this is not sufficient; the /var partition in the default install can 
easily be overflowed long before the log is rotated, causing 
malfunctions. I've temporarily changed /etc/crontab so that newsyslog is 
run every 5 minutes instead of once a day (which may be a good idea to 
prevent other denials of service via this sort of overflow as well). But 
it also makes sense to patch the system so that it does not fill so many 
verbose messages -- and/or to ignore the bogus queries generated by the 
spyware. It may also pay to patch BIND to limit the overhead that is 
incurred when such queries occur. Ideas?

--Brett Glass

At 4:41 PM -0600 10/23/03, Brett Glass wrote:
>
>FreeBSD currently comes configured, in the default install,
>to check /var/messages only once a day, and to rotate the
>log file if it's above a certain size.
My /etc/newsyslog.conf indicates that /var/log/messages
should be rotated whenever it gets over 100K.
>I've temporarily changed /etc/crontab so that newsyslog is
>run every 5 minutes instead of once a day (which may be a
>good idea to prevent other denials of service via this sort
>of overflow as well).
On both my 4.x and 5.x systems, /etc/crontab will run
newsyslog once per hour.  I'm pretty sure that at least some
of the code in newsyslog assumes that the program is run only
once per hour.  Running it more frequently than that may
cause some problems.

I'm sure that /var can fill up even if /var/log/messages is
rotated every hour, if the error messages are coming in fast
enough.  But the file should be getting rotated once per hour
in the default install, not once per day.

I do not think that the correct solution is to rotate the
files at an even faster rate.  Just how large is /var on the
machine where you're seeing this problem?

-- 
Garance Alistair Drosehn            =   gad@gilead.netel.rpi.edu
Senior Systems Programmer           or  gad@freebsd.org
Rensselaer Polytechnic Institute    or  drosih@rpi.edu

Brett Glass wrote:

[snip]
> Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
> (ns11.opennic.glue)
> 
> The references to OpenNIC have caused me to suspect (though I have not 
> verified it yet) that the problem is due to the New.Net spyware, which 
Seeing how nobody else has noted it, OpenNIC is *not* New.net

[snip]

Brett Glass wrote:
> All:
> 
> I'm posting this to FreeBSD-security (rather than FreeBSD-net) because 
> the problems I'm seeing appear to have been caused by spyware, and 
> because they constitute a possible avenue for denial of service on 
> FreeBSD machines with default installs of the operating system.
> 
> Several of the FreeBSD machines on our network began to act strangely 
> during the past week. Some have started to refuse mail; in other cases, 
> important daemons have died without warning. All of the machines are 
> running 4.x releases of FreeBSD with all recent patches installed, and 
> all are running the version of BIND supplied with FreeBSD. The
"top"
> command, when run on these machines, showed that BIND is consuming very 
> large amounts of CPU time, but this by itself couldn't explain all of 
> the symptoms we were seeing.
> 
> This afternoon, I examined the machines and discovered the problem: full 
> /var partitions caused by huge /var/log/messages files.
> 
> Inspection of the files reveals hundreds of thousands of messages of the 
> form:
> 
> Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
> (ns0.opennic.glue)
> Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
> (ns1.opennic.glue)
> Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
> (ns3.opennic.glue)
> Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
> (ns4.opennic.glue)
> Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
> (ns6.opennic.glue)
> Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
> (ns7.opennic.glue)
> Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
> (ns8.opennic.glue)
> Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
> (ns11.opennic.glue)
> Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
> (ns10.opennic.glue)
> Oct 23 16:00:07 victim named[326]: sysquery: no addrs found for root NS 
> (ns11.opennic.glue)
> 
> The references to OpenNIC have caused me to suspect (though I have not 
> verified it yet) that the problem is due to the New.Net spyware, which 
> causes Windows machines to query OpenNIC's name servers. From what
I've
> read so far, it appears that New.Net is "foistware" -- that is,
it can
> be installed on innocent users' Windows machines without their consent 
> via holes in Internet Explorer. But if New.Net is not what's 
> responsible, SOMETHING certainly seems to be generating bogus DNS 
> queries, which in turn are causing these messages.
> 
> FreeBSD currently comes configured, in the default install, to check 
> /var/messages only once a day, and to rotate the log file if it's above
> a certain size. Unfortunately, these messages accumulate so rapidly that 
> this is not sufficient; the /var partition in the default install can 
> easily be overflowed long before the log is rotated, causing 
> malfunctions. I've temporarily changed /etc/crontab so that newsyslog
is
> run every 5 minutes instead of once a day (which may be a good idea to 
> prevent other denials of service via this sort of overflow as well). But 
> it also makes sense to patch the system so that it does not fill so many 
> verbose messages -- and/or to ignore the bogus queries generated by the 
> spyware. It may also pay to patch BIND to limit the overhead that is 
> incurred when such queries occur. Ideas?
> 
Wouldn't a better work-around be either add ns*.opennic.glue addresses 
to named.root or setup a dummy zone for .glue that just returns a 
localhost address to the client?

Or a possible solution would be to setup bind to log directly to its own 
log files and rotate them when needed and turn off logging to syslog.

Bind8&9 allow for logging of various messages to different files and 
letting bind rotate them when needed.  Check out the Bind documention. 
There is a helpful example available at: 
http://logreport.org/doc/gen/dns/bind8.php

Here's a quick example from bind9:
# This setups logging options
# general info is logged to both syslog and a local file
# info about lame-servers is sent to /dev/null
logging {
         channel named_log {
         file "/var/named/named.log" versions 5 size 1m;
         severity info;
         print-time yes;
         };

         channel null {
         null;
         };

category "default" { "named_log"; default_syslog; };
category "lame-servers" { "null"; };
};

I guess as an improvement on the default named.conf, it could include an 
example section on logging options.

greg