Shorewall users - Aug 2003 - DDoS attacks, what can be done?

Hi,

I run two live c-class subnets on the internet. Last
Sunday morning I was hit with a DDoS attack and it
hasn''t stopped.

I made modifications on my shorewall firewall during
Sunday to lesson the impact, as they were hammering me
with 180k/5sec traffic both ways (inbound and
outbound).

One of the primary things which helped reduce their
DDoS was enabling "norfc1918" on the interfaces (this
stopped about 95% of the barrage), but I also enabled
routefilter,dropunclean,blacklist and tcpflags.

The blacklist feature unfortunately doesn''t do much
since the source addresses are faked, but I was able
to determine most of the attack was coming from a site
in Taiwan.

The barrage hasn''t stopped and monitoring it over the
last 5 days it''s been consistently consuming about
5-9k/5sec both ways - which indicates to me I''m still
sending some responses out to the attackers although
checking the iptables logs they''re mostly ICMP 8
packets.

I''ve seeked help from my provider also but they''ve
said the only thing they can do is change my static
IP, which won''t do anything as they''re attacking my
subnets.

What else can I do?

Thanks.

Michael.


http://search.yahoo.com.au - Yahoo! Search
- Looking for more? Try the new Yahoo! Search

On Thu, 28 Aug 2003, Michael Mansour wrote:
> The blacklist feature unfortunately doesn''t do much
> since the source addresses are faked, but I was able
> to determine most of the attack was coming from a site
> in Taiwan.
If the IP addresses are faked how did you determine the attack was/is
coming from Taiwan?

If you give me the sites coordinates I''ll walk over, assuming its in
Taipei, and pull their plug.   :-)

Ed
>
> The barrage hasn''t stopped and monitoring it over the
> last 5 days it''s been consistently consuming about
> 5-9k/5sec both ways - which indicates to me I''m still
> sending some responses out to the attackers although
> checking the iptables logs they''re mostly ICMP 8
> packets.
>
> I''ve seeked help from my provider also but they''ve
> said the only thing they can do is change my static
> IP, which won''t do anything as they''re attacking my
> subnets.
>
> What else can I do?
>
> Thanks.
>
> Michael.
>
>
> http://search.yahoo.com.au - Yahoo! Search
> - Looking for more? Try the new Yahoo! Search
> _______________________________________________
> Shorewall-users mailing list
> Post: Shorewall-users@lists.shorewall.net
> Subscribe/Unsubscribe:
http://lists.shorewall.net/mailman/listinfo/shorewall-users
> Support: http://www.shorewall.net/support.htm
> FAQ: http://www.shorewall.net/FAQ.htm
>
-- 
SARS - The only virus not spread by Outlook
http://www.shorewall.net/  for all your firewall needs

--- Michael Mansour <micoots@yahoo.com> wrote:
> Hi,
> 
> I run two live c-class subnets on the internet. Last
> Sunday morning I was hit with a DDoS attack and it
> hasn''t stopped.
What kind of DOS attack specifically are you talking about?
> I made modifications on my shorewall firewall during
> Sunday to lesson the impact, as they were hammering me
> with 180k/5sec traffic both ways (inbound and
> outbound).
I can understand inbound but what do you mean outbound? This would tell me that
you have some
systems behind the firewall that are hacked with trojans running on them...
> One of the primary things which helped reduce their
> DDoS was enabling "norfc1918" on the interfaces (this
> stopped about 95% of the barrage), but I also enabled
> routefilter,dropunclean,blacklist and tcpflags.
RFC1918 addresses??..How was that being routed?
> The blacklist feature unfortunately doesn''t do much
> since the source addresses are faked, but I was able
> to determine most of the attack was coming from a site
> in Taiwan.
Were the source addresses public ip''s? How did you determine that this
was coming from Taiwan??
Curious as to what a TCPdump of this would look like. Even better and ethereal
capture log to see
the ip options flags set within these packets.

Check this out. I think that this is whats going on. Loose Source Routing. 

http://www.synacklabs.net/OOB/LSR.html

Tom isn''t there a way to control LSR''ing in iptables? If this
in fact whats happening?
> The barrage hasn''t stopped and monitoring it over the
> last 5 days it''s been consistently consuming about
> 5-9k/5sec both ways - which indicates to me I''m still
> sending some responses out to the attackers although
> checking the iptables logs they''re mostly ICMP 8
> packets.
 
> I''ve seeked help from my provider also but they''ve
> said the only thing they can do is change my static
> IP, which won''t do anything as they''re attacking my
> subnets.
> 
> What else can I do?
I would like to know what the outcome of this is? Hope everything settles down
soon for you
Michael.

Joshua Banks

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com

> > The blacklist feature unfortunately doesn''t do
> much
> > since the source addresses are faked, but I was
> able
> > to determine most of the attack was coming from a
> site
> > in Taiwan.
> 
> If the IP addresses are faked how did you determine
> the attack was/is
> coming from Taiwan?
About 98% of the packets are faked, but the remaining
2 % are real. The way I knew this was looking at the
outbound DROP''s on my firewall, they all have the same
internet IP address (in Taiwan). The normal activity
I''ve noticed for this is random source addresses, but
with outbound bursts of the same IP happening every
few hours, that tells me the authors of the attack are
trying to get some info from my systems.
> If you give me the sites coordinates I''ll walk over,
> assuming its in
> Taipei, and pull their plug.   :-)
Thanks :) .. the best I can do is give you their IP
address though.

Michael.


http://search.yahoo.com.au - Yahoo! Search
- Looking for more? Try the new Yahoo! Search

On Thu, 28 Aug 2003, Michael Mansour wrote:
> About 98% of the packets are faked, but the remaining
> 2 % are real. The way I knew this was looking at the
> outbound DROP''s on my firewall, they all have the same
> internet IP address (in Taiwan). The normal activity
> I''ve noticed for this is random source addresses, but
> with outbound bursts of the same IP happening every
> few hours, that tells me the authors of the attack are
> trying to get some info from my systems.
I guess I''m still a bit confused about your definition of
"faked"...
> > If you give me the sites coordinates I''ll walk over,
> > assuming its in
> > Taipei, and pull their plug.   :-)
>
> Thanks :) .. the best I can do is give you their IP
> address though.
Sure, why not.  :-)

BTW, I''ve been seeing lots of PROTO=ICMP TYPE=8 drops and they are
coming
from all over the globe...

Ed

-- 
SARS - The only virus not spread by Outlook
http://www.shorewall.net/  for all your firewall needs

Hey Michael,

What is the destination port in the packets. Is it simply ICMP or something
else?

And what machines are responding outbound specifically? 

Are these servers that you have setup for the public to get to?..Http...Smtp..??

When you say "Faked" addresses, do you mean random spoofed public ip
addresses?

It sounds as though you could be suffering from acouple of different types of
attacks basically
using the same type of method.


1)
Smurf /relector attack: as well as Smurf /amplified relfector attack.
If I put your external firewall address or any public ip address that you own in
the source of a
packet and randomly generated legit destination addresses in the same packets,
for say ICMP echo
request, then all legitimate destanation addresses will respond with an echo
reply to your source
address. Flooding your network with Garbage icmp echo-reply packets.

In an amplified refelctor attack the same thing happens but the destination is a
broadcast
subnetwork in the destination field. If 100''s of computers are alive
and public on the specified
subnet then you will get hammered 1000''s of times. Ouch...

2)
Spoofed Source Syn packets..This is very common Syn flood attack to take down
web servers and
such. Randomly generated public source addresses destined to your public
address(es) with the syn
flag set in the packets. Your servers respond with a flood of "Syn
Acks" to an ip address(es) that
really never sent anything legit... This will burn up cpu cycles galore and
bandwidth....

I wish I knew more about how to accomplish combating this with Linux/Unix so I
could help. I''ll be
doing my homework. I''ve always had firewalls that had built in
capabilites to handle these
different types of signature attacks.


This is why you need to figure out, 1st, what kind of attack(s) are you dealing
with. Most of
these are well documented on the internet with solutions and prevention
techniques. A dood IDS
system helps.

Joshua Banks


__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com

--On Thursday, August 28, 2003 1:20 AM -0700 Joshua Banks 
<l0f33t@yahoo.com> wrote:
>
> This is why you need to figure out, 1st, what kind of attack(s) are you
> dealing with. Most of these are well documented on the internet with
> solutions and prevention techniques. A dood IDS system helps.
I''m pretty sure he meant good here, and he''s serious.  the IDS
needs to
help not hurt.  If the IDS just chokes on the syn flood same as the other 
things it''s not a good IDS.  Snort is pretty good, it has it''s
moments, but
I''ve yet to see it get into a failure mode that caused massive havoc
like I
have some IDSen (commercial ones mostly...we did a lab of about half a 
dozen different IDSes, the commercial ones fared, on average, worse).

Be careful though if you use the acronym IDS on the snort list....recent 
thread was rather strongly against that LOL :)

HTH HAND!

> > About 98% of the packets are faked, but the
> remaining
> > 2 % are real. The way I knew this was looking at
> the
> > outbound DROP''s on my firewall, they all have the
> same
> > internet IP address (in Taiwan). The normal
> activity
> > I''ve noticed for this is random source addresses,
> but
> > with outbound bursts of the same IP happening
> every
> > few hours, that tells me the authors of the attack
> are
> > trying to get some info from my systems.
> 
> I guess I''m still a bit confused about your
> definition of "faked"...
> 
> > > If you give me the sites coordinates I''ll walk
> over,
> > > assuming its in
> > > Taipei, and pull their plug.   :-)
> >
> > Thanks :) .. the best I can do is give you their
> IP
> > address though.
> 
> Sure, why not.  :-)
What I''ll do is the next time I see bursts of this
stuff, and I''ve determined that I believe them to be
from the authors of the attack, I''ll email the IPs
here.
> BTW, I''ve been seeing lots of PROTO=ICMP TYPE=8
> drops and they are coming
> from all over the globe...
I agree, but I have been running my systems for just
over 9 years, and it''s never been this bad.

BTW, for the past hour I''ve been hit with this garbage
continually:

Aug 31 11:11:45 cheetah named[2284]: lame server
resolving ''bgl1dns-a.sancharnet.in'' (in
''in''?):
198.6.1.182#53
Aug 31 11:11:45 cheetah named[2284]: lame server
resolving ''ndl1dns-a.sancharnet.in'' (in
''in''?):
198.6.1.182#53
Aug 31 11:11:45 cheetah named[2284]: lame server
resolving ''ndl1nms-a.sancharnet.in'' (in
''in''?):
198.6.1.182#53
Aug 31 11:11:45 cheetah named[2284]: lame server
resolving ''bgl1dns-a.sancharnet.in'' (in
''in''?):
137.39.1.3#53
Aug 31 11:11:45 cheetah named[2284]: lame server
resolving ''ndl1dns-a.sancharnet.in'' (in
''in''?):
137.39.1.3#53
Aug 31 11:11:45 cheetah named[2284]: lame server
resolving ''ndl1nms-a.sancharnet.in'' (in
''in''?):
137.39.1.3#53
Aug 31 11:11:46 cheetah named[2284]: lame server
resolving ''bgl1dns-a.sancharnet.in'' (in
''in''?):
198.6.1.65#53
Aug 31 11:11:46 cheetah named[2284]: lame server
resolving ''ndl1dns-a.sancharnet.in'' (in
''in''?):
198.6.1.65#53
Aug 31 11:11:46 cheetah named[2284]: lame server
resolving ''ndl1nms-a.sancharnet.in'' (in
''in''?):
198.6.1.65#53
Aug 31 11:11:46 cheetah named[2284]: lame server
resolving ''230.19.113.202.in-addr.arpa'' (in
''19.113.202.in-addr.arpa''?): 202.113.16.10#53
Aug 31 11:11:47 cheetah named[2284]: lame server
resolving ''230.19.113.202.in-addr.arpa'' (in
''19.113.202.in-addr.arpa''?): 202.113.16.10#53
Aug 31 11:11:47 cheetah named[2284]: lame server
resolving ''218.28.208.134.in-addr.arpa'' (in
''28.208.134.in-addr.arpa''?): 134.208.10.11#53
Aug 31 11:11:48 cheetah named[2284]: lame server
resolving ''19.243.253.211.in-addr.arpa'' (in
''243.253.211.in-addr.arpa''?): 210.104.1.3#53
Aug 31 11:11:49 cheetah named[2284]: lame server
resolving ''8.254.182.211.in-addr.arpa'' (in
''254.182.211.in-addr.arpa''?): 210.104.1.3#53
Aug 31 11:11:50 cheetah kernel:
Shorewall:net2all:DROP:IN=ppp0 OUT=eth0
SRC=203.197.157.57 DST=203.14.210.87 LEN=48 TOS=0x00
PREC=0x00 TTL=107 ID=14149 PROTO=TCP SPT=4085 DPT=80
WINDOW=16384 RES=0x00 SYN URGP=0
Aug 31 11:11:50 cheetah named[2284]: lame server
resolving ''9.254.182.211.in-addr.arpa'' (in
''254.182.211.in-addr.arpa''?): 210.204.251.22#53
Aug 31 11:11:51 cheetah named[2284]: lame server
resolving ''9.254.182.211.in-addr.arpa'' (in
''254.182.211.in-addr.arpa''?): 210.104.1.3#53
Aug 31 11:11:51 cheetah named[2284]: lame server
resolving ''8.254.182.211.in-addr.arpa'' (in
''254.182.211.in-addr.arpa''?): 210.204.251.22#53
Aug 31 11:11:51 cheetah named[2284]: lame server
resolving ''9.254.182.211.in-addr.arpa'' (in
''254.182.211.in-addr.arpa''?): 210.104.1.3#53
Aug 31 11:11:52 cheetah named[2284]: lame server
resolving ''9.254.182.211.in-addr.arpa'' (in
''254.182.211.in-addr.arpa''?): 210.204.251.22#53

about 20 of these entries a second. My guess is they
know I answer queries on port 53, from their port
scans of my network, and they''re now attacking that,
although shutting down named does not affect these
entires.

Any ideas here?

Michael.


http://search.yahoo.com.au - Yahoo! Search
- Looking for more? Try the new Yahoo! Search

Michael Mansour wrote:
[snip]
> Aug 31 11:11:52 cheetah named[2284]: lame server
> resolving ''9.254.182.211.in-addr.arpa'' (in
> ''254.182.211.in-addr.arpa''?): 210.204.251.22#53
> 
> about 20 of these entries a second. My guess is they
> know I answer queries on port 53, from their port
> scans of my network, and they''re now attacking that,
> although shutting down named does not affect these
> entires.
> 
> Any ideas here?
Michael,

There is  not much I can add to this thread that has not already been
discussed. I can only confirm that in the past few weeks, I have seen a huge
increase in icmp/8 rejections at this end. Like on the order of 200-300+
logged in a 5 minute period (I chart rejections at my firewall using MRTG).
I eventually had to add a shorewall rule to stop the logging of icmp/8, but
still reject these packets.

As for the lame server entries...

Type: dig +trace 9.254.182.211.in-addr.arpa ptr

Note the infinite loop from the .kr DNS servers. <groan> At my end, the
above dig resulted in a "To many lookups" message. Plus, when I remove
the
+trace option, I see the same lame server message being logged when I
re-enable lame server logging (I normally disable lame server logging in
named.conf).

Hopefully this madness will slow down in the near future. At least I hope
so.

Steve Cowles