Hello List! I got a small (50mbits or so) application layer ddos attack against a few name servers (thousands of IPs sending lots of bogus A record requests - weird) - one of the name servers was behind a shorewall firewall. That firewall was running a 2.6.18-194.11.1.el5 kernel and shorewall-4.4.11.1-1. I noticed that the shorewall host had ksoftirqd using 100% of the CPU during the attack and was kind of slow in general as a result - I think this may have affected traffic to other hosts behind that firewall as well. Any ideas what would cause this? I was hoping to avoid this scenario in the future if possible since I plan on deploying some other name servers behind shorewall (latest stable on 2.6.32-358.0.1.el6.x86_64) as a result of this incident, but would ideally have a fix for this in place. I should probably point out that the blacklist file had around 500 entries in it - not sure that would have any effect on things. During the attack, the kernel logged a bunch of these: ip_conntrack: table full, dropping packet - Possibly the result of connection tracking? Does netfilter even track UDP "connections"? I thought UDP was connectionless. Is the only workaround for cases like this just to have larger connection tracking values in the kernel? Does that help with the ksoftirqd CPU use? Or is it best in this case to just not have it track connection state for DNS traffic at all and just forward the packets along? How is the ideal solution for this case implemented? Any help is appreciated! Michael P.S. The attack ended up coming from a bunch of networks mostly in Taiwan - had my provider drop traffic from those networks and the problem was solved. ------------------------------------------------------------------------------ AlienVault Unified Security Management (USM) platform delivers complete security visibility with the essential security capabilities. Easily and efficiently configure, manage, and operate all of your security controls from a single console and one unified framework. Download a free trial. http://p.sf.net/sfu/alienvault_d2d
Michael McCallister wrote, On 5/16/2013 12:05 AM:> Hello List! > > I got a small (50mbits or so) application layer ddos attack against a > few name servers (thousands of IPs sending lots of bogus A record > requests - weird) - one of the name servers was behind a shorewall > firewall. That firewall was running a 2.6.18-194.11.1.el5 kernel and > shorewall-4.4.11.1-1. I noticed that the shorewall host had ksoftirqd > using 100% of the CPU during the attack and was kind of slow in general > as a result - I think this may have affected traffic to other hosts > behind that firewall as well. Any ideas what would cause this? I was > hoping to avoid this scenario in the future if possible since I plan on > deploying some other name servers behind shorewall (latest stable on > 2.6.32-358.0.1.el6.x86_64) as a result of this incident, but would > ideally have a fix for this in place. I should probably point out that > the blacklist file had around 500 entries in it - not sure that would > have any effect on things. > > During the attack, the kernel logged a bunch of these: ip_conntrack: > table full, dropping packet - Possibly the result of connection > tracking? Does netfilter even track UDP "connections"? I thought UDP > was connectionless. Is the only workaround for cases like this just to > have larger connection tracking values in the kernel? Does that help > with the ksoftirqd CPU use? Or is it best in this case to just not have > it track connection state for DNS traffic at all and just forward the > packets along? How is the ideal solution for this case implemented? > > Any help is appreciated! > > Michael > > P.S. The attack ended up coming from a bunch of networks mostly in > Taiwan - had my provider drop traffic from those networks and the > problem was solved.BTW: I realize all bets are off with a DDoS attack - but this one was only 50mbit and an application layer attack - I just want to beef things up to better handle smaller attacks (like this one) - I am fully aware that if they saturate the link, there is nothing you can do. ------------------------------------------------------------------------------ AlienVault Unified Security Management (USM) platform delivers complete security visibility with the essential security capabilities. Easily and efficiently configure, manage, and operate all of your security controls from a single console and one unified framework. Download a free trial. http://p.sf.net/sfu/alienvault_d2d
On 05/16/2013 09:50 AM, Michael McCallister wrote:> BTW: I realize all bets are off with a DDoS attack - but this one was > only 50mbit and an application layer attack - I just want to beef things > up to better handle smaller attacks (like this one) - I am fully aware > that if they saturate the link, there is nothing you can do.If you can characterize the attack, I have found that adding NOTRACK rules in /etc/shorewall/conntrack that match the attack profile is effective in reducing CPU utilization and conntrack table overflow. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ AlienVault Unified Security Management (USM) platform delivers complete security visibility with the essential security capabilities. Easily and efficiently configure, manage, and operate all of your security controls from a single console and one unified framework. Download a free trial. http://p.sf.net/sfu/alienvault_d2d
Michael McCallister wrote, On 5/16/2013 9:50 AM:> Michael McCallister wrote, On 5/16/2013 12:05 AM: >> Hello List! >> >> I got a small (50mbits or so) application layer ddos attack against a >> few name servers (thousands of IPs sending lots of bogus A record >> requests - weird) - one of the name servers was behind a shorewall >> firewall. That firewall was running a 2.6.18-194.11.1.el5 kernel and >> shorewall-4.4.11.1-1. I noticed that the shorewall host had ksoftirqd >> using 100% of the CPU during the attack and was kind of slow in general >> as a result - I think this may have affected traffic to other hosts >> behind that firewall as well. Any ideas what would cause this? I was >> hoping to avoid this scenario in the future if possible since I plan on >> deploying some other name servers behind shorewall (latest stable on >> 2.6.32-358.0.1.el6.x86_64) as a result of this incident, but would >> ideally have a fix for this in place. I should probably point out that >> the blacklist file had around 500 entries in it - not sure that would >> have any effect on things. >> >> During the attack, the kernel logged a bunch of these: ip_conntrack: >> table full, dropping packet - Possibly the result of connection >> tracking? Does netfilter even track UDP "connections"? I thought UDP >> was connectionless. Is the only workaround for cases like this just to >> have larger connection tracking values in the kernel? Does that help >> with the ksoftirqd CPU use? Or is it best in this case to just not have >> it track connection state for DNS traffic at all and just forward the >> packets along? How is the ideal solution for this case implemented? >> >> Any help is appreciated! >> >> Michael >> >> P.S. The attack ended up coming from a bunch of networks mostly in >> Taiwan - had my provider drop traffic from those networks and the >> problem was solved. > BTW: I realize all bets are off with a DDoS attack - but this one was > only 50mbit and an application layer attack - I just want to beef things > up to better handle smaller attacks (like this one) - I am fully aware > that if they saturate the link, there is nothing you can do. >After running a series of tests that to some extent... emulate a @#$% ton of incoming UDP/DNS connections, I found that dropping shorewall altogether worked best for me - 2 things seemed to help, 1) the less/simpler the iptables rules, the better, and 2) not loading the connection tracking modules - at all, helped a lot. In this setup, I saw CPU use drop a lot, and it does not overrun connection tracking. Of course, this really limits what you can do with the firewall, but if your needs can be trimmed down and you can live without connection tracking and the convenience of shorewall, then this seems to help with this problem. Of course, I am not sure to what extent this is specific to my setup/kernel/network/hardware - so mileage may vary considerably. ------------------------------------------------------------------------------ AlienVault Unified Security Management (USM) platform delivers complete security visibility with the essential security capabilities. Easily and efficiently configure, manage, and operate all of your security controls from a single console and one unified framework. Download a free trial. http://p.sf.net/sfu/alienvault_d2d
Tom Eastep wrote, On 5/17/2013 8:14 AM:> On 05/16/2013 09:50 AM, Michael McCallister wrote: > >> BTW: I realize all bets are off with a DDoS attack - but this one was >> only 50mbit and an application layer attack - I just want to beef things >> up to better handle smaller attacks (like this one) - I am fully aware >> that if they saturate the link, there is nothing you can do. > If you can characterize the attack, I have found that adding NOTRACK > rules in /etc/shorewall/conntrack that match the attack profile is > effective in reducing CPU utilization and conntrack table overflow. > > -TomThanks! I will check that out - that could possibly help for the one place I was unable to lose shorewall due to the sheer complexity of the rules. ------------------------------------------------------------------------------ AlienVault Unified Security Management (USM) platform delivers complete security visibility with the essential security capabilities. Easily and efficiently configure, manage, and operate all of your security controls from a single console and one unified framework. Download a free trial. http://p.sf.net/sfu/alienvault_d2d
On 5/17/13 1:48 PM, "Michael McCallister" <mikemc-shorewall@terabytemedia.com> wrote:>Tom Eastep wrote, On 5/17/2013 8:14 AM: >> On 05/16/2013 09:50 AM, Michael McCallister wrote: >> >>> BTW: I realize all bets are off with a DDoS attack - but this one was >>> only 50mbit and an application layer attack - I just want to beef >>>things >>> up to better handle smaller attacks (like this one) - I am fully aware >>> that if they saturate the link, there is nothing you can do. >> If you can characterize the attack, I have found that adding NOTRACK >> rules in /etc/shorewall/conntrack that match the attack profile is >> effective in reducing CPU utilization and conntrack table overflow. >> >> -Tom > >Thanks! I will check that out - that could possibly help for the one >place I was unable to lose shorewall due to the sheer complexity of the >rules.One correction -- use DROP rules rather than NOTRACK. -Tom You do not need a parachute to skydive. You only need a parachute to skydive twice. ------------------------------------------------------------------------------ AlienVault Unified Security Management (USM) platform delivers complete security visibility with the essential security capabilities. Easily and efficiently configure, manage, and operate all of your security controls from a single console and one unified framework. Download a free trial. http://p.sf.net/sfu/alienvault_d2d