Bob Hoffman
2012-Feb-09 23:54 UTC
[CentOS] oops, or how to bring a datacenter router down with one setting
so I gave up on bonding. I found about 300 posts showing eth0 and eth1 both pointing to br0 (bridge) as interfaces. I followed them correctly, or so I thought. I pointed both ethx to the bridge, restarted network and bam...!!! entire ip block went out. when I called datacenter they told me the router was under attack and I was like 'uh oh' and told them to just shut off my computer I would be there to fix it. They did not believe me. An hour later I was there and deleted the eth1 point to the br0 and all was fine. Meanwhile they were all around the router trying to stop the attack. (it was just the router for me and others in that room....oops) I wonder if they will boot me from the center now? How is it possible that it did that so quickly? Such an easy way to bring down routers, wow, a hacker could have a field day. Apparently there is more to making to eth ports go to the same bridge than a simple point. I have since tried bridge_ports command as listed online, however that must be deprecated. I think I am just gonna stay with multiple bridges with one eth on each for a while until I can test this stuff in a safe environ. I never had a chance to recover, the second the network came up I lost all contact with my ip block. The ratelimit number got this high by the time I got there. Feb 9 04:22:41 main kernel: __ratelimit: 100807 callbacks suppressed Feb 9 04:22:41 main kernel: eth1: received packet with own address as source address Feb 9 04:22:41 main kernel: eth1: received packet with own address as source address Feb 9 04:22:41 main kernel: eth1: received packet with own address as source address Feb 9 04:22:41 main kernel: eth1: received packet with own address as source address Feb 9 04:22:41 main kernel: eth0: received packet with own address as source address Feb 9 04:22:41 main kernel: eth0: received packet with own address as source address Feb 9 04:22:41 main kernel: eth0: received packet with own address as source address Feb 9 04:22:41 main kernel: eth0: received packet with own address as source address Feb 9 04:22:41 main kernel: eth0: received packet with own address as source address Feb 9 04:22:41 main kernel: eth0: received packet with own address as source address
Tony Mountifield
2012-Feb-10 10:18 UTC
[CentOS] oops, or how to bring a datacenter router down with one setting
In article <4F345CD3.4060604 at bobhoffman.com>, Bob Hoffman <bob at bobhoffman.com> wrote:> so I gave up on bonding. > I found about 300 posts showing eth0 and eth1 both pointing to br0 (bridge) > as interfaces. > I followed them correctly, or so I thought. > I pointed both ethx to the bridge, restarted network and bam...!!! > > entire ip block went out. > > [...] > > Feb 9 04:22:41 main kernel: __ratelimit: 100807 callbacks suppressed > Feb 9 04:22:41 main kernel: eth1: received packet with own address as > source addressI think to do this you also need to be connected to a managed switch which supports interface bonding. You would have to tell it that the two switch ports are bonded to the same machine. That should prevent it from forwarding packets received on one of the ports out via the other port. The key phrase to look for appears to be "IEEE 802.3ad Dynamic Link Aggregation". Cheers Tony -- Tony Mountifield Work: tony at softins.co.uk - http://www.softins.co.uk Play: tony at mountifield.org - http://tony.mountifield.org
Dennis Jacobfeuerborn
2012-Feb-10 11:47 UTC
[CentOS] oops, or how to bring a datacenter router down with one setting
On 02/10/2012 12:54 AM, Bob Hoffman wrote:> so I gave up on bonding. > I found about 300 posts showing eth0 and eth1 both pointing to br0 (bridge) > as interfaces. > I followed them correctly, or so I thought. > I pointed both ethx to the bridge, restarted network and bam...!!!Bonding and bridging are completely different things. If you want to start bonding then you should first start with simply bonding the two interfaces and only once you got that going add the bridge and then add the bond0 device to it. Regards, Dennis
Lamar Owen
2012-Feb-10 20:01 UTC
[CentOS] oops, or how to bring a datacenter router down with one setting
On Feb 9, 2012, at 6:54 PM, Bob Hoffman wrote:> entire ip block went out. > > when I called datacenter they told me the router was under attack > and I > was like 'uh oh' and told them to just shut off my computer I would be > there to fix it. They did not believe me. > An hour later I was there and deleted the eth1 point to the br0 and > all > was fine. > Meanwhile they were all around the router trying to stop the attack. > (it was just the router for me and others in that room....oops) > > I wonder if they will boot me from the center now? > How is it possible that it did that so quickly? > Such an easy way to bring down routers, wow, a hacker could have a > field > day.If you weren't running a spanning-tree on your Linux bridge, and their switch ports aren't sending you BPDU's for STP, then you found out what happens when you activate a bridging (from the point of view of the switch, not the Linux bridging) loop. Been there, done that. Most monitoring tools are written to track layer-3 happenings, and this is happening at layer 2. And it will take down that whole layer 2 broadcast domain, that's for sure. And since many, if not most, tools are working at layer 3 and dealing with IP flows and not actual ethernet traffic, none of the typical layer 3 tools will give any indication why the network just bogged down to a halt; you just about have to have a network probe (like wireshark) on a SPAN port to catch it, unless you know some of the telltale signs. On a gigabit switch a fully saturating bridge loop can form in less than a second, and bring things close to a halt. Most datacenter switches have configurable parameters to guard against loops (Cisco even has a feature called, appropriately enough, loopguard, but this may or may not fix this case).