freebsd stable - Aug 2008 - machine hangs on occasion - correlated with ssh break-in attempts

Hello!

A machine I manage remotely for a friend comes under a distributed ssh 
break-in attack every once in a while. Annoyed (and alarmed) by the 
messages like:

Aug 12 10:21:17 symbion sshd[4333]: Invalid user mythtv from 85.234.158.180
Aug 12 10:21:18 symbion sshd[4335]: Invalid user mythtv from 85.234.158.180
Aug 12 10:21:20 symbion sshd[4337]: Invalid user mythtv from 85.234.158.180
Aug 12 10:21:21 symbion sshd[4339]: Invalid user mythtv from 85.234.158.180

I wrote an awk-script, which adds a block of the attacking IP-address to 
the ipfw-rules after three such "invalid user" attempts with:

    ipfw add 550 deny ip from ip

The script is fed by syslogd directly -- through a syslog.conf rule 
("|/opt/sbin/auth-log-watch").

Once in a while I manually flush these rules... I this a good (safe) 
reaction?
I'm asking, because the machine (currently running 7.0 as of July 7) 
hangs solid once every few weeks... My only guess is that a spike in 
attacks causes "too many" ipfw-entries created, which paralyzes the 
kernel due to some bug -- the machine is running natd and is the gateway 
for the rest of the network...
The hangs could, of course, be caused by something else entirely, but my 
self-defense mechanism is my first suspect...

Any comments? Thanks!

    -mi

On Thu, Aug 21, 2008 at 01:38:38PM -0400, Mikhail Teterin
wrote:
> ...
> I wrote an awk-script, which adds a block of the attacking IP-address to 
> the ipfw-rules after three such "invalid user" attempts with:
> 
>    ipfw add 550 deny ip from ip
> 
> The script is fed by syslogd directly -- through a syslog.conf rule 
> ("|/opt/sbin/auth-log-watch").
> ... 
At a previous employer, we were building mail relay boxen (FreeBSD
6.0 - 6.2 timeframe); at one point, It Was Decided that rather than
having /var/log/maillog written directly by syslogd(8), syslogd(8)
would feed a Perl script that would do some "Database Things" and
then get around to appending to /var/log/maillog itself.

While the amount of work involved was assuredly greater in that case
than in yours, those of us who were actually building and running the
relays in question were very unsurprised when Postfix performance
improved significantly following a redesign of the application, so that
/var/log/maillog was written by syslogd(8) and the Perl script was
effectively fed via "tail -F".
> Once in a while I manually flush these rules... I this a good (safe) 
> reaction?
I also see such things (on my home "firewall" machine); my approach
is quite a bit different.  If folks are interested, I could probably
discuss it a bit, but I believe that would be, at best, tangential
to your note, and thus ought not be crafted as if it were part of
the thread -- and definitely does not warrant the cross-post.
> ...
Peace,
david
-- 
David H. Wolfskill				david@catwhisker.org
Depriving a girl or boy of an opportunity for education is evil.

See http://www.catwhisker.org/~david/publickey.gpg for my public key.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
Url :
http://lists.freebsd.org/pipermail/freebsd-security/attachments/20080821/180a843a/attachment.pgp

> Date: Thu, 21 Aug 2008 13:38:38 -0400
> From: Mikhail Teterin <mi+mill@aldan.algebra.com>
> Sender: owner-freebsd-stable@freebsd.org
> 
> Hello!
> 
> A machine I manage remotely for a friend comes under a distributed ssh 
> break-in attack every once in a while. Annoyed (and alarmed) by the 
> messages like:
> 
> Aug 12 10:21:17 symbion sshd[4333]: Invalid user mythtv from 85.234.158.180
> Aug 12 10:21:18 symbion sshd[4335]: Invalid user mythtv from 85.234.158.180
> Aug 12 10:21:20 symbion sshd[4337]: Invalid user mythtv from 85.234.158.180
> Aug 12 10:21:21 symbion sshd[4339]: Invalid user mythtv from 85.234.158.180
> 
> I wrote an awk-script, which adds a block of the attacking IP-address to 
> the ipfw-rules after three such "invalid user" attempts with:
> 
>     ipfw add 550 deny ip from ip
> 
> The script is fed by syslogd directly -- through a syslog.conf rule 
> ("|/opt/sbin/auth-log-watch").
> 
> Once in a while I manually flush these rules... I this a good (safe) 
> reaction?
> I'm asking, because the machine (currently running 7.0 as of July 7) 
> hangs solid once every few weeks... My only guess is that a spike in 
> attacks causes "too many" ipfw-entries created, which paralyzes
the
> kernel due to some bug -- the machine is running natd and is the gateway 
> for the rest of the network...
> The hangs could, of course, be caused by something else entirely, but my 
> self-defense mechanism is my first suspect...
> 
> Any comments? Thanks!
Looks remarkably like sshguard (ports/security/sshguard-*). It does almost
exactly what you are doing but is written in C and has command-line
switches to set how long a system is blocked, how many attempts
constitute an attack and how long it should remember failed attempts. It
also allows the use of back-end scripts if you want it to do something
else such as generate reports (beyond an entry in /var/log/messages).

As far as the hangs, I don't believe it is from the large nu,ber of
brute force attempts as they will stop for a given host as soon as the
firewall is updated. I seldom see more than a handful of attack sources
over any short period.

Should you want to continue with your own tool, at least for IPv4,
consider using tables rather than a raft of rules. With tables, you need
only a single rule and it is there at boot time.
-- 
R. Kevin Oberman, Network Engineer
Energy Sciences Network (ESnet)
Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab)
E-mail: oberman@es.net			Phone: +1 510 486-8634
Key fingerprint:059B 2DDF 031C 9BA3 14A4  EADA 927D EBB3 987B 3751
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 224 bytes
Desc: not available
Url :
http://lists.freebsd.org/pipermail/freebsd-security/attachments/20080821/3ab8723f/attachment.pgp

David Wolfskill ???????(??):
> While the amount of work involved was assuredly greater in that case 
> than in yours, those of us who were actually building and running the 
> relays in question were very unsurprised when Postfix performance 
> improved significantly following a redesign of the application, so 
> that /var/log/maillog was written by syslogd(8) and the Perl script 
> was effectively fed via "tail -F".
In my setup, syslogd does both -- append the message to the appropriate 
log-file (in this case -- /var/log/auth.log) and feed it to the script's 
stdin. From syslogd.conf:

    auth.info;authpriv.info                         /var/log/auth.log
    auth.info;authpriv.info                        
    |/opt/sbin/auth-log-watch

"tail -F" seems just wrong :-)

    -mi

> Should you want to continue with your own tool, at least for IPv4,
> consider using tables rather than a raft of rules. With tables, you need
> only a single rule and it is there at boot time.
Also, you might want to consider switching to pf
which this functionality built-in.

	Jan

I haven't explored this issue enough to speak with any authority - but  
once upon a time I had an app doing tons of ipfw rule add/removes all  
the time and we had no end of performance and stability problems on  
that box (this would have been in 4.x or so timeline I expect).  As  
that approach wasn't really critical we abandoned it without really  
digging into the details.

Years later a need for lots of rapid firewall changes came up again  
and I drilled into it and found the use of tables was excellent for  
doing this and it does the job very well.  This is approach is on a  
FreeBSD 6.3 box.

ipfw add 00550 deny ip from 'table(1)' to any

Then just add remove entries to table 1 via:
ipfw table 1 add 10.1.1.22/32
ipfw table 1 delete 10.1.1.22/32

show all entries in table 1 with:
ipfw table 1 list

Clear out the whole of table 1
ipfw table 1 flush

I can't be sure if this relates to your particular issue, but I would  
recommend trying it out.

Neil Neely
http://neil-neely.blogspot.com




On Aug 21, 2008, at 11:38 AM, Mikhail Teterin wrote:
> Hello!
>
> A machine I manage remotely for a friend comes under a distributed  
> ssh break-in attack every once in a while. Annoyed (and alarmed) by  
> the messages like:
>
> Aug 12 10:21:17 symbion sshd[4333]: Invalid user mythtv from  
> 85.234.158.180
> Aug 12 10:21:18 symbion sshd[4335]: Invalid user mythtv from  
> 85.234.158.180
> Aug 12 10:21:20 symbion sshd[4337]: Invalid user mythtv from  
> 85.234.158.180
> Aug 12 10:21:21 symbion sshd[4339]: Invalid user mythtv from  
> 85.234.158.180
>
> I wrote an awk-script, which adds a block of the attacking IP- 
> address to the ipfw-rules after three such "invalid user"
attempts
> with:
>
>   ipfw add 550 deny ip from ip
>
> The script is fed by syslogd directly -- through a syslog.conf rule  
> ("|/opt/sbin/auth-log-watch").
>
> Once in a while I manually flush these rules... I this a good (safe)  
> reaction?
> I'm asking, because the machine (currently running 7.0 as of July 7)  
> hangs solid once every few weeks... My only guess is that a spike in  
> attacks causes "too many" ipfw-entries created, which paralyzes
the
> kernel due to some bug -- the machine is running natd and is the  
> gateway for the rest of the network...
> The hangs could, of course, be caused by something else entirely,  
> but my self-defense mechanism is my first suspect...
>
> Any comments? Thanks!
>
>   -mi
>
> _______________________________________________
> freebsd-security@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-security
> To unsubscribe, send any mail to
"freebsd-security-unsubscribe@freebsd.org
> "

Mikhail Teterin <mi+mill@aldan.algebra.com> writes:
> A machine I manage remotely for a friend comes under a distributed ssh
> break-in attack every once in a while. Annoyed (and alarmed) by the
> messages like:
>
> Aug 12 10:21:17 symbion sshd[4333]: Invalid user mythtv from 85.234.158.180
> Aug 12 10:21:18 symbion sshd[4335]: Invalid user mythtv from 85.234.158.180
> Aug 12 10:21:20 symbion sshd[4337]: Invalid user mythtv from 85.234.158.180
> Aug 12 10:21:21 symbion sshd[4339]: Invalid user mythtv from 85.234.158.180
>
> I wrote an awk-script, which adds a block of the attacking IP-address
> to the ipfw-rules after three such "invalid user" attempts with:
>
>    ipfw add 550 deny ip from ip
I don't know if it will make your problem go away, but using ipfw
tables for this seems to be a better idea than creating a new rule for
every IP address.

So you just need one rule:

    ipfw add 550 deny ip from table(1)

And then when you want to add an IP address to the table:

    ipfw table 1 add <ip>

You can add ranges too using the CIDR notation.

-- 
Christian Laursen

On Thu, Aug 21, 2008 at 01:38:38PM -0400, Mikhail Teterin
wrote:
> Hello!
>
> A machine I manage remotely for a friend comes under a distributed ssh  
> break-in attack every once in a while. Annoyed (and alarmed) by the  
> messages like:
>
> Aug 12 10:21:17 symbion sshd[4333]: Invalid user mythtv from 85.234.158.180
> Aug 12 10:21:18 symbion sshd[4335]: Invalid user mythtv from 85.234.158.180
> Aug 12 10:21:20 symbion sshd[4337]: Invalid user mythtv from 85.234.158.180
> Aug 12 10:21:21 symbion sshd[4339]: Invalid user mythtv from 85.234.158.180
>
> I wrote an awk-script, which adds a block of the attacking IP-address to  
> the ipfw-rules after three such "invalid user" attempts with:
>
>    ipfw add 550 deny ip from ip
>
> The script is fed by syslogd directly -- through a syslog.conf rule  
> ("|/opt/sbin/auth-log-watch").
>
> Once in a while I manually flush these rules... I this a good (safe)  
> reaction?
> I'm asking, because the machine (currently running 7.0 as of July 7)  
> hangs solid once every few weeks... My only guess is that a spike in  
> attacks causes "too many" ipfw-entries created, which paralyzes
the
> kernel due to some bug -- the machine is running natd and is the gateway  
> for the rest of the network...
> The hangs could, of course, be caused by something else entirely, but my  
> self-defense mechanism is my first suspect...
>
> Any comments? Thanks!
Yes, I have quite a few comments on this matter:

The above looks like sshguard.  I've personally never trusted something
that *automatically* adjusts firewall rules based on data read from text
logs or packets coming in off the Internet.  The risks involved are
insanely high.

Stop for a moment and think what would happen to your box if a
distributed brute-force attack (e.g. 300,000 different IPs) was launched
against it; someone executing 20-30 SSH login attempts per IP.  I'm
willing to bet adding 300,000 individual ipfw entries would cause some
serious havok on your machine (speculative: exhausted kernel memory, or
at a bare minimum, exhaust the number of remaining ipfw rule entries)

And yes, the liklihood of someone doing this is quite high.

Try re-thinking your firewall logic.  Instead of "allow any, deny
specific IPs dynamically", how about "allow specific IPs, deny all
others"?

Surely you don't have that many users who SSH into the NAT router from
random public IPs all over the world, rather than via the LAN?  Surely
if you yourself often SSH into your NAT router from a Blackberry device,
that you wouldn't have much of a problem adding a /19 to the allow list.
That's a hell of a lot better than allowing 0/0 and denying individual
/32s.

A different approach: consider putting sshd on a different port, rather
than the default of 22.  A lot of people I know do this, solely to
decrease the number of brute-force attempts you see above; I've never
seen any of those brute-force attacking programs portscan, then attack
against a port which returns a OpenSSH string.

Finally, consider moving to pf instead, if you really feel ipfw is
what's causing your machine to crash.  You might be pleasantly surprised
by the syntax, and overall administrative usability (it is significantly
superior to ipfw, IMHO).

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |

I do something related to this with fwlogwatch although it can probably
be adapted to any similar tool; when I hit the 'block' threshold, I
execute something like:

#!/bin/sh
HR=`date "+%-k"`
/sbin/ipfw table 0 add $3 ${HR}

.. so each entry has a tag indicating the hour at which the block was
initiated.

At 5 to the hour, I run a simple cron job which does this to clean out
everything older than 24 hours ..

#!/bin/sh
HR=`date -v+1H "+%-k"`
/sbin/ipfw table 0 list >/tmp/xx.$$
cat /tmp/xx.$$ |
while read LINE
do
        set $LINE
        case "$2" in
        ${HR})
                /sbin/ipfw table 0 delete $1
                echo -n `date +"%H:%M:%S"`
>>/var/log/fwlw_clean_log
                echo " fwlw_clean: removed $1 from table
0"
>>/var/log/fwlw_clean_log
        esac
done
rm /tmp/xx.$$

I also have a script in /usr/local/etc/rc.d which saves the current
state in the event of an orderly shutdown and restores it on boot:

#!/bin/sh
case "$1" in
start)
        cat /var/db/ipfw/cache0 | while read LINE
        do
                set $LINE
                /sbin/ipfw table 0 add $1 $2
        done
        ;;
stop)
        /sbin/ipfw table 0 list >/var/db/ipfw/cache0
        ;;
restart)
        $0 $DEBUG stop
        $0 $DEBUG start
        exit $?
        ;;
*)
        echo "Usage: $0 {start|stop|restart}"
        exit 1
        ;;
esac
exit 0

Of course, this only works for ipv4 because of the restriction on the
ipfw table data but it's just an example,

    Michael

Mikhail Teterin pisze:
> Hello!
> 
> A machine I manage remotely for a friend comes under a distributed ssh 
> break-in attack every once in a while. Annoyed (and alarmed) by the 
> messages like:
> 
> Aug 12 10:21:17 symbion sshd[4333]: Invalid user mythtv from 85.234.158.180
> Aug 12 10:21:18 symbion sshd[4335]: Invalid user mythtv from 85.234.158.180
> Aug 12 10:21:20 symbion sshd[4337]: Invalid user mythtv from 85.234.158.180
> Aug 12 10:21:21 symbion sshd[4339]: Invalid user mythtv from 85.234.158.180
> 
> I wrote an awk-script, which adds a block of the attacking IP-address to 
> the ipfw-rules after three such "invalid user" attempts with:
> 
>    ipfw add 550 deny ip from ip
> 
> The script is fed by syslogd directly -- through a syslog.conf rule 
> ("|/opt/sbin/auth-log-watch").
Hi,

   You should look at 'bruteblock' (ports/security), it has similar 
fuctionality. It also provides daemon process, bruteblockd, which is
responsible for removing entries from ipfw table.

Best regards,
-- 
_/_/   .. Eugene Butusov
  _/_/  ... www.devilka.info
   _/_/ .... ebutusov(at)gmail(dot)com

May or may not be an option, but changing the default port that ssh runs on
worked well enough for our needs. --It greatly reduced the number of automated
attacks against the servers.
It might work well enough to allow your DIY script to keep up without problems.

-dan
-----Original Message-----
From: Mikhail Teterin <mi+mill@aldan.algebra.com>

Date: Thu, 21 Aug 2008 13:38:38 
To: <freebsd-security@freebsd.org>; <freebsd-stable@FreeBSD.org>
Subject: machine hangs on occasion - correlated with ssh break-in attempts


Hello!

A machine I manage remotely for a friend comes under a distributed ssh
break-in attack every once in a while. Annoyed (and alarmed) by the
messages like:

Aug 12 10:21:17 symbion sshd[4333]: Invalid user mythtv from 85.234.158.180
Aug 12 10:21:18 symbion sshd[4335]: Invalid user mythtv from 85.234.158.180
Aug 12 10:21:20 symbion sshd[4337]: Invalid user mythtv from 85.234.158.180
Aug 12 10:21:21 symbion sshd[4339]: Invalid user mythtv from 85.234.158.180

I wrote an awk-script, which adds a block of the attacking IP-address to
the ipfw-rules after three such "invalid user" attempts with:

    ipfw add 550 deny ip from ip

The script is fed by syslogd directly -- through a syslog.conf rule
("|/opt/sbin/auth-log-watch").

Once in a while I manually flush these rules... I this a good (safe)
reaction?
I'm asking, because the machine (currently running 7.0 as of July 7)
hangs solid once every few weeks... My only guess is that a spike in
attacks causes "too many" ipfw-entries created, which paralyzes the
kernel due to some bug -- the machine is running natd and is the gateway
for the rest of the network...
The hangs could, of course, be caused by something else entirely, but my
self-defense mechanism is my first suspect...

Any comments? Thanks!

    -mi

_______________________________________________
freebsd-security@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-security
To unsubscribe, send any mail to
"freebsd-security-unsubscribe@freebsd.org"

On Thu, 21 Aug 2008 13:03:09 -0700
Jeremy Chadwick <koitsu@FreeBSD.org> wrote:
> A different approach: consider putting sshd on a different port, rather
> than the default of 22.  A lot of people I know do this, solely to
> decrease the number of brute-force attempts you see above; I've never
> seen any of those brute-force attacking programs portscan, then attack
> against a port which returns a OpenSSH string.
+1 - obscurity definitely doesn't ADD to security , but it removes all the
noise from your system.

Alternatively, you try port knocking ;)
> Finally, consider moving to pf instead, if you really feel ipfw is
> what's causing your machine to crash.  You might be pleasantly
surprised
> by the syntax, and overall administrative usability (it is significantly
> superior to ipfw, IMHO).
+1 
_________________________
{Beto|Norberto|Numard} Meijome

If Bill Gates had a dollar for every time a Windows box crashed...
.. Oh, wait a minute, he already does.

I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.

At 12:38 PM 8/21/2008, Mikhail Teterin wrote:
>Hello!
>
>A machine I manage remotely for a friend comes under a distributed ssh 
>break-in attack every once in a while. Annoyed (and alarmed) by the 
>messages like:
>
>Aug 12 10:21:17 symbion sshd[4333]: Invalid user mythtv from 85.234.158.180
>Aug 12 10:21:18 symbion sshd[4335]: Invalid user mythtv from 85.234.158.180
>Aug 12 10:21:20 symbion sshd[4337]: Invalid user mythtv from 85.234.158.180
>Aug 12 10:21:21 symbion sshd[4339]: Invalid user mythtv from 85.234.158.180
>
>I wrote an awk-script, which adds a block of the attacking IP-address to 
>the ipfw-rules after three such "invalid user" attempts with:
>
>    ipfw add 550 deny ip from ip
>
>The script is fed by syslogd directly -- through a syslog.conf rule 
>("|/opt/sbin/auth-log-watch").
>
>Once in a while I manually flush these rules... I this a good (safe)
reaction?
>I'm asking, because the machine (currently running 7.0 as of July 7)
hangs
>solid once every few weeks... My only guess is that a spike in attacks 
>causes "too many" ipfw-entries created, which paralyzes the kernel
due to
>some bug -- the machine is running natd and is the gateway for the rest of 
>the network...
>The hangs could, of course, be caused by something else entirely, but my 
>self-defense mechanism is my first suspect...
>
>Any comments? Thanks!
>
>    -mi
I doubt it is your script, or syslog causing the crash.  It is likely a 
hardware problem of some type if you have this server completely patched 
and up-to-date for security patches.  I would look at the memory, ethernet, 
hard disk, or power supply as the most likely candidates.

         -Derek

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.