Hello, I''ve been a Shorewall user and supporter for many years and it has been a great tool. But recently our Web servers have been under attack and I can figure out how to stop it. The problem is that the attacks are coming in on port 80 all from different IPs. I''m talking thousands of requests per hour. I can''t find any information on how to stop this kind of attack. What I''m doing right now is redirecting these from cgi to a page using mod rewrite, but this isn''t stopping all these requests from being initiated and it''s killing our server. Any ideas on what to do? 216.109.73.21 - - [24/Aug/2010:19:21:25 -0700] "GET /cgi-bin/sitesearch.cgi?t=XXXdUwYrtYXXXdU HTTP/1.0" 302 298 "http://vanhanhphuc.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9" 200.43.141.173 - - [24/Aug/2010:19:21:25 -0700] "GET /cgi-bin/sitesearch.cgi?t=XXXdUwYrtYXXXdU HTTP/1.0" 302 298 "http://vanhanhphuc.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9" 210.13.105.7 - - [24/Aug/2010:19:21:26 -0700] "GET /cgi-bin/sitesearch.cgi?t=XXXdUwYrtYXXXdU HTTP/1.1" 302 298 "http://vanhanhphuc.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9" I have verified that there is nothing on the "http://vanhanhphuc.com/" page pointing to us (no frames, script or ?). What you will notice is that all these requests have the same user-agent (millions of them exactly the same) which leads me to believe this is a worm of some sort. Is there anything Shorewall can do to help us? If not, any ideas of what we can do? Thanks in advance, John ------------------------------------------------------------------------------ Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d
On 8/24/10 7:33 PM, J and T wrote:> Is there anything Shorewall can do to help us?You can try per-IP rate limiting but that might end up running your kernel out of memory if there are truly 1000s of hosts attacking your system. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d
Thanks Tom. I also thought of that, but you''re right, that would crash us as well. I would think this would be a common problem, but I can''t seem to find any solution. The sad thing is similar attacks are super easy to setup using iframes with a combination of javascript to make for loop calls in a target window completely hidden from their visitors. Put that on a few dozen servers and Web pages and make 100 requests per minute per visitor per page and you''ll take down a server in no time. That sucks! Oh well, I guess we''ll just have to limit our port 80 requests to keep it under the crashing point and just block those legit visitors when that limit is reached. Too bad there are these kinds of people out there. Thanks again for your time Tom, John Date: Tue, 24 Aug 2010 20:30:33 -0700 From: teastep@shorewall.net To: shorewall-users@lists.sourceforge.net Subject: Re: [Shorewall-users] Can Shorewall Help Me? On 8/24/10 7:33 PM, J and T wrote:> Is there anything Shorewall can do to help us?You can try per-IP rate limiting but that might end up running your kernel out of memory if there are truly 1000s of hosts attacking your system. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d _______________________________________________ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users ------------------------------------------------------------------------------ Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d
You should try implementing IDS, snort or something. That might help detecting an attack and write a script to block that IP. Also keep your web server updated with all the security patches. Swapnil Sent from my iPhone On 25-Aug-2010, at 9:26 AM, J and T <j_and_t@hotmail.com> wrote: Thanks Tom. I also thought of that, but you''re right, that would crash us as well. I would think this would be a common problem, but I can''t seem to find any solution. The sad thing is similar attacks are super easy to setup using iframes with a combination of javascript to make for loop calls in a target window completely hidden from their visitors. Put that on a few dozen servers and Web pages and make 100 requests per minute per visitor per page and you''ll take down a server in no time. That sucks! Oh well, I guess we''ll just have to limit our port 80 requests to keep it under the crashing point and just block those legit visitors when that limit is reached. Too bad there are these kinds of people out there. Thanks again for your time Tom, John Date: Tue, 24 Aug 2010 20:30:33 -0700 From: teastep@shorewall.net To: shorewall-users@lists.sourceforge.net Subject: Re: [Shorewall-users] Can Shorewall Help Me? On 8/24/10 7:33 PM, J and T wrote:> Is there anything Shorewall can do to help us?You can try per-IP rate limiting but that might end up running your kernel out of memory if there are truly 1000s of hosts attacking your system. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d _______________________________________________ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users ------------------------------------------------------------------------------ Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d _______________________________________________ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users ------------------------------------------------------------------------------ Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d
On Tue, 2010-08-24 at 20:56 -0700, J and T wrote:> Thanks Tom. I also thought of that, but you''re right, that would crash > us as well. I would think this would be a common problem, but I can''t > seem to find any solution. >Would creating a blackhole or prohibit route on the web-server itself help? Yea, a bit heavy handed but if the offending ip address is not really access any of your sites... something like: for i in `cat <path/to/logfile> | grep sitesearch.cgi?t=XXXdUwYrtYXXXdU | awk ''{print $1}''`; do sudo ip route add prohibit $i; done The web-server should bail on the request as there is no route back and close the connection. Jerry ------------------------------------------------------------------------------ Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d
On 08/24/2010 09:33 PM, J and T wrote:> Hello, > > I''ve been a Shorewall user and supporter for many years and it has been > a great tool. But recently our Web servers have been under attack and I > can figure out how to stop it. The problem is that the attacks are > coming in on port 80 all from different IPs. I''m talking thousands of > requests per hour. I can''t find any information on how to stop this kind > of attack. What I''m doing right now is redirecting these from cgi to a > page using mod rewrite, but this isn''t stopping all these requests from > being initiated and it''s killing our server. Any ideas on what to do? > > 216.109.73.21 - - [24/Aug/2010:19:21:25 -0700] "GET > /cgi-bin/sitesearch.cgi?t=XXXdUwYrtYXXXdU HTTP/1.0" 302 298 > "http://vanhanhphuc.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; > en-US; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9" > 200.43.141.173 - - [24/Aug/2010:19:21:25 -0700] "GET > /cgi-bin/sitesearch.cgi?t=XXXdUwYrtYXXXdU HTTP/1.0" 302 298 > "http://vanhanhphuc.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; > en-US; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9" > 210.13.105.7 - - [24/Aug/2010:19:21:26 -0700] "GET > /cgi-bin/sitesearch.cgi?t=XXXdUwYrtYXXXdU HTTP/1.1" 302 298 > "http://vanhanhphuc.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; > en-US; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9" > > I have verified that there is nothing on the "http://vanhanhphuc.com/" > page pointing to us (no frames, script or ?). What you will notice is > that all these requests have the same user-agent (millions of them > exactly the same) which leads me to believe this is a worm of some sort. > > Is there anything Shorewall can do to help us? If not, any ideas of what > we can do? > > Thanks in advance, > John > > >You could try using fail2ban with a regex for "vanhanhphuc" or something. And then once matched, ban the ip address. http://www.fail2ban.org/wiki/index.php/Main_Page Sam ------------------------------------------------------------------------------ Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d
On 8/25/2010 5:33 AM, J and T wrote:> Hello, > > I''ve been a Shorewall user and supporter for many years and it has > been a great tool. But recently our Web servers have been under attack > and I can figure out how to stop it. The problem is that the attacks > are coming in on port 80 all from different IPs. I''m talking thousands > of requests per hour. I can''t find any information on how to stop this > kind of attack. What I''m doing right now is redirecting these from cgi > to a page using mod rewrite, but this isn''t stopping all these > requests from being initiated and it''s killing our server. Any ideas > on what to do?I''ve been using with very good results the script from here: http://deflate.medialayer.com/ I would recommend using the following line in it though: netstat -ntu | grep ":80" | awk ''{print $5}'' | sed s/::ffff:// | cut -d: -f1 | sort | uniq -c | sort -nr > $BAD_IP_LIST Good luck, Marius ------------------------------------------------------------------------------ Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d
Michael Weickel - iQom Business Services GmbH
2010-Aug-25 06:37 UTC
Re: Can Shorewall Help Me?
You will never manage to stop an attack! All what you can do is either to
wait until its finished, pay the attacker or prepare your environment to be
able to handle attacks beside regular traffic successfully. The last way is
obviously the one we are interested in.
Since they will come from thousands by thousands different ips blocking is a
possible workaround but be sure that you may will cut good traffic as well
since the source ips are in most cases spoofed and my belong to other
parties.
Shorewall can successfully block those traffic but it has to be processed
(rejected, dropped, whatever) anyway.
But thousands per hour sounds not as a problem, I guess you mean thousands
per second. The problem in most cases is a bad handshake or broken header in
any other way and they send thousands by thousands of them.
There are some workarounds how you can solve your problem.
1.) LVS
2.) A proxy server
3.) Talk to your provider they may have Arbors, Brocades or similar in
their network
So at all you have to ask yourself a question. Are you earning money with
your website which means is it a shop or only informational? If it is a shop
you will definitively become a target once the time is ripe to it. Normally
beside the attack you receives e-mails where you are requested on a very
polite way to pay 100 or 200 Euros. Not much but anyway not what we want.
We have learned that a Webservice such as a Webserver should not be behind a
Firewall. There is really no reason to do it if you have multiple visitors.
Better to put it beside but behind a loadbalancer or a proxy. They easily
can handle hundret thousands of sessions and are able to have filter sets to
eat the bad sessions and only to let the good ones to your real server.
So at all its only a problem of amount of queries. From a given time our
session table is too full and since you have bad syn´s or whatever it can
take some time to timeout but nes sessions are arriving anyway each second.
So at all its not a Shorewall problem and usually also not a bandwidth
problem. So even if you would use a medium sized hardware firewall such as
Juniper or Cisco you wont be able to fight against them.
I hope I was able to give some ideas how to move on.
Cheers
Michael
_____
Von: Marius Stan [mailto:mstan@asesoft.ro]
Gesendet: Mittwoch, 25. August 2010 07:46
An: Shorewall Users
Betreff: Re: [Shorewall-users] Can Shorewall Help Me?
On 8/25/2010 5:33 AM, J and T wrote:
Hello,
I''ve been a Shorewall user and supporter for many years and it has been
a
great tool. But recently our Web servers have been under attack and I can
figure out how to stop it. The problem is that the attacks are coming in on
port 80 all from different IPs. I''m talking thousands of requests per
hour.
I can''t find any information on how to stop this kind of attack. What
I''m
doing right now is redirecting these from cgi to a page using mod rewrite,
but this isn''t stopping all these requests from being initiated and
it''s
killing our server. Any ideas on what to do?
I''ve been using with very good results the script from here:
http://deflate.medialayer.com/
I would recommend using the following line in it though:
netstat -ntu | grep ":80" | awk ''{print $5}'' | sed
s/::ffff:// | cut -d: -f1
| sort | uniq -c | sort -nr > $BAD_IP_LIST
Good luck,
Marius
------------------------------------------------------------------------------
Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
Be part of this innovative community and reach millions of netbook users
worldwide. Take advantage of special opportunities to increase revenue and
speed time-to-market. Join now, and jumpstart your future.
http://p.sf.net/sfu/intel-atom-d2d
I want to thank everyone for all their suggestions. Everything has been of help to me. With this said, you''ll notice that all requests result in a 302 because I detect them first and redirect them so I have already captured their IP. So what I decided to do for now is instead of redirecting, I am passing the IPs off to netfilter to block them and then releasing those IPs from the filter after some time. This is working OK for the time being and certainly better than nothing. Question: is there a way to block IP''s using netfilter/shorewall with a "time-to-live"? That would be an awesome feature if there is one. I was not able to find anything on that at the site. Right now I''m storing the IPs in a text file and then purging from the filter. If there was a ttl this would be much easier. Thanks again everyone, John From: j_and_t@hotmail.com To: shorewall-users@lists.sourceforge.net Date: Tue, 24 Aug 2010 19:33:28 -0700 Subject: [Shorewall-users] Can Shorewall Help Me? Hello, I''ve been a Shorewall user and supporter for many years and it has been a great tool. But recently our Web servers have been under attack and I can figure out how to stop it. The problem is that the attacks are coming in on port 80 all from different IPs. I''m talking thousands of requests per hour. I can''t find any information on how to stop this kind of attack. What I''m doing right now is redirecting these from cgi to a page using mod rewrite, but this isn''t stopping all these requests from being initiated and it''s killing our server. Any ideas on what to do? 216.109.73.21 - - [24/Aug/2010:19:21:25 -0700] "GET /cgi-bin/sitesearch.cgi?t=XXXdUwYrtYXXXdU HTTP/1.0" 302 298 "http://vanhanhphuc.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9" 200.43.141.173 - - [24/Aug/2010:19:21:25 -0700] "GET /cgi-bin/sitesearch.cgi?t=XXXdUwYrtYXXXdU HTTP/1.0" 302 298 "http://vanhanhphuc.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9" 210.13.105.7 - - [24/Aug/2010:19:21:26 -0700] "GET /cgi-bin/sitesearch.cgi?t=XXXdUwYrtYXXXdU HTTP/1.1" 302 298 "http://vanhanhphuc.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9" I have verified that there is nothing on the "http://vanhanhphuc.com/" page pointing to us (no frames, script or ?). What you will notice is that all these requests have the same user-agent (millions of them exactly the same) which leads me to believe this is a worm of some sort. Is there anything Shorewall can do to help us? If not, any ideas of what we can do? Thanks in advance, John ------------------------------------------------------------------------------ Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d _______________________________________________ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users ------------------------------------------------------------------------------ Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d
On 8/25/10 7:56 AM, J and T wrote:> > Question: is there a way to block IP''s using netfilter/shorewall with a > "time-to-live"? That would be an awesome feature if there is one. I was > not able to find anything on that at the site. Right now I''m storing the > IPs in a text file and then purging from the filter. If there was a ttl > this would be much easier.You can use an ''iptree'' ipset which allows a timeout value for entries to be specified. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d