Here are some notes I have about Linux bridging. I''ll try to separate what I know I know from what I think I know. Let''s say I want to bridge eth0, eth1, and eth2 together, all with an IP Address of, say, 1.2.3.2. This is how to do it: echo "Setting up br0 to bridge eth0 with eth1 and eth2" /usr/sbin/brctl addbr br0 /usr/sbin/brctl addif br0 eth0 /usr/sbin/brctl addif br0 eth1 /usr/sbin/brctl addif br0 eth2 /sbin/ip addr add 1.2.3.2/24 dev br0 /sbin/ip link set br0 up Continuing with the above example, most of the writeups also say to remove any IP Addresses from eth0, eth1, and eth2. But I''ve found this doesn''t seem necessary - well, sort of. Let''s say that eth0 is at IP Address 1.2.3.2, and now I bridge eth0, eth1, and eth2 together and give bridge br0 the same IP Address of 1.2.3.2. Now I have a mess because both eth0 and br0 have the same IP Address. Doing this: ip addr del 1.2.3.2/24 dev eth0 cleans up the mess. But let''s say that physical interface eth1 has IP Address 10.0.0.1.>From testing, it looks like other systems can ping 10.0.0.1 just fine,assuming they have a route to it. So I **think** I know that I can assign an IP Address to a raw interface, as long as it''s a different IP Address than what I assigned to the overall bridge. But I haven''t seen this capability documented anywhere. Let''s say the bridge is up and working at IP Address 1.2.3.2. I have a system at IP Address 1.2.3.1 connected via eth0. That system can ping 1.2.3.2 easily. If I disconnect the Ethernet cable from eth0 and plug into eth1 or eth2, after about 30 seconds, that bridged system begins answering pings again. As indicated in the writeups, that spare PC with a bunch of NICs is now acting like a managed Ethernet switch. Cool! Filtering iptables is a super-sophisticated toolset to filter IP packets. ebtables is another toolset to filter at the OSI layer 2 (datalink) layer. iptables concerns itself (mostly) with routing across an IP network, computer to computer. ebtables concerns itself (mostly) with filtering packets across physical NIC interfaces in the same computer. Here is a great writeup on using ebtables and iptables together: http://ebtables.sourceforge.net/br_fw_ia/br_fw_ia.html But - like everything I''ve been able to find so far, I don''t think this writeup is completely accurate. iptables has a module called physdev. According to the writeup, I can use the iptables physdev module to filter among the raw interfaces in a bridge. But a discussion in the netfilter list essentially says that physdev is being removed because it creates all kinds of other problems. At least, I think that''s what it says. The relevant discussion took place in early July 2006. Here is a pointer to the beginning of the discussion: https://lists.netfilter.org/pipermail/netfilter-devel/2006-July/024896.h tml So it looks like when filtering at the network layer, (IP in this case) use iptables. When filtering at the data link layer, use ebtables and maybe arptables. Avoid using -m physdev in iptables because it''s going away. You can add IP Addresses to bridged eth-- interfaces as long as they don''t conflict with the bridge IP Address(es). Next up will be to try some filtering scenarios with ebtables and iptables. - Greg Scott
On 6/6/2007 6:51 PM, Greg Scott wrote:> Continuing with the above example, most of the write ups also say to > remove any IP Addresses from eth0, eth1, and eth2. But I''ve found > this doesn''t seem necessary - well, sort of.No, you don''t have to remove IP addresses from the bridge member interfaces, though it does make things (IMHO) cleaner and consistent. You could put one IP address on each of the (physical) bridge member interfaces, or you could put the IP addresses (as aliases) on the bridge interface. Although I''m not sure how well EBTables would be able to filter addresses on a raw interface verses on the bridge interface.> Let''s say that eth0 is at IP Address 1.2.3.2, and now I bridge eth0, > eth1, and eth2 together and give bridge br0 the same IP Address of > 1.2.3.2. Now I have a mess because both eth0 and br0 have the same > IP Address. Doing this:Yes, I would consider an IP address conflict a mess.> But let''s say that physical interface eth1 has IP Address 10.0.0.1. > From testing, it looks like other systems can ping 10.0.0.1 just > fine, assuming they have a route to it. So I **think** I know that > I can assign an IP Address to a raw interface, as long as it''s a > different IP Address than what I assigned to the overall bridge. But > I haven''t seen this capability documented anywhere.Just because you can does not mean that you should. Most people that are working (read: using in production) bridging would not do so unless they had a need to do so. If you do not need bridging, generally it is not used. Thus if you can get by with putting the IP address(es) on the physical interfaces, then usually you will not be using bridging. This does not mean that it can not be done.> Let''s say the bridge is up and working at IP Address 1.2.3.2. I have > a system at IP Address 1.2.3.1 connected via eth0. That system can > ping 1.2.3.2 easily. If I disconnect the Ethernet cable from eth0 > and plug into eth1 or eth2, after about 30 seconds, that bridged > system begins answering pings again. As indicated in the write ups, > that spare PC with a bunch of NICs is now acting like a managed > Ethernet switch. Cool!In Spanning Tree Protocol, this is considered the listening / learning delay to prevent loops in the network. This delay should be tunable (I''m not 100% certain of this fact).> iptables is a super-sophisticated toolset to filter IP packets. > ebtables is another toolset to filter at the OSI layer 2 (datalink) > layer. iptables concerns itself (mostly) with routing across an IP > network, computer to computer. ebtables concerns itself (mostly) > with filtering packets across physical NIC interfaces in the same > computer.Close, but not quite. (I''ll preface this paragraph by saying that by default, this is how IPTables behaves, as it can be changed.) IPTables is a GREAT filtering framework that operates on OSI Layer 3 and higher (match extensions and what not). EBTables is a VERY GOOD filtering framework that operates on OSI Layer 2. I.e. one filters in the IP stack and the other filters in the bridging code (respectively). Granted both IPTables and EBTables can filter on some things like source / destination IP address, IP protocol, TCP / UDP port, MAC address, and a few other things. However IPTables has access to a LOT more information than EBTables does. Now, for the non default config. You can turn on "Bridge Netfilter" code in the kernel to allow the bridging code to use IPTables in addition to EBTables. What this means is that you can use all of IPTables matches extensions to make OSI Layer 3 and higher decisions on OSI Layer 2. I.e. you can have a bridging firewall do just about any thing that you want it to do.> Here is a great writeup on using ebtables and iptables together: > http://ebtables.sourceforge.net/br_fw_ia/br_fw_ia.html But - like > everything I''ve been able to find so far, I don''t think this writeup > is completely accurate.(It has been too long since I read that. Please refresh my memory.) What about that write up do you think is in-accurate? (I''ll re-read the parts you point out.) I''ll see if I agree with you or the write up and if so try to help explain why.> iptables has a module called physdev. According to the writeup, I > can use the iptables physdev module to filter among the raw > interfaces in a bridge. But a discussion in the netfilter list > essentially says that physdev is being removed because it creates all > kinds of other problems. At least, I think that''s what it says. The > relevant discussion took place in early July 2006. Here is a pointer > to the beginning of the discussion: > https://lists.netfilter.org/pipermail/netfilter-devel/2006-July/024896.html > > So it looks like when filtering at the network layer, (IP in this > case) use iptables. When filtering at the data link layer, use > ebtables and maybe arptables. Avoid using -m physdev in iptables > because it''s going away. You can add IP Addresses to bridged eth-- > interfaces as long as they don''t conflict with the bridge IP > Address(es).With out reading too far in to it, it looks like the existing physdev match will be deprecated. I however do not think this will be a permanent thing. I believe that either a re-worked physdev, or something like it will re-appear in due time. Also keep in mind that there is a LOT of re-working going on in the later 2.6.x kernels, including renaming variables and changing the internal workings. (Too much so if you ask me for a .even release.)> Next up will be to try some filtering scenarios with ebtables and > iptables.... Grant. . . .
> Although I''m not sure how well EBTables would be able > to filter addresses on a raw interface verses on the > bridge interface.I ran some tests and it seemed to work. Here is a little test script: #!/bin/sh /firewall-scripts/allow-all EBTABLES="/usr/local/sbin/ebtables" IPTABLES="/sbin/iptables" $EBTABLES -F $EBTABLES -A INPUT -i eth0 -j DROP $EBTABLES -A INPUT -i eth1 -j ACCEPT $EBTABLES -A INPUT -i eth2 -j ACCEPT #$IPTABLES -A FORWARD -i eth0 -j DROP #$IPTABLES -A FORWARD -i eth1 -j ACCEPT #$IPTABLES -A FORWARD -i eth2 -j ACCEPT #$IPTABLES -A INPUT -i eth0 -j DROP #$IPTABLES -A INPUT -i eth1 -j ACCEPT #$IPTABLES -A INPUT -i eth2 -j ACCEPT [root@IXFactor-fw gregs]# The IP Address for br0 is 1.2.3.2. I setup a system at 1.2.3.1 and had it ping 1.2.3.2. Everything behaved as expected. With bridging turned on, none of the iptables rules made any difference, so I commented them out. With the cable plugged into eth0, none of the pings replied. Tcpdump on the 1.2.3.2 box didn''t even show anything coming into br0. Connecting to eth1 or eth2, pings replied and tcpdump showed the ICMP going back and forth on br0. Setting br0 to 1.2.3.2 and eth1 to 10.0.0.1, my external host could ping both IP Addresses when I fudged in appropriate routes on my test system. Here are a couple of my challenges. I want to block anything coming in from the Internet claiming to come from 10/8, or 172.16/12 or 192.168/16. But anything from these address blocks from any trusted eth-- interface should be OK. Easy to do with pure routing, seems a little more challenging with bridging. I think I can whip up some rules, but they depend on ebtables filtering by physical eth-- port. I have a public /24 block, call it 1.2.3.nnn. Some of these are on eth0, some on eth1, a few others on eth2. Bridging seems to handle this nicely. eth0 faces the Internet. I am a little nervous in packaging all this. Let''s say something goes haywire in startup, before I run the brctl stuff. Or let''s say something bad happens and bridging goes haywire. If eth0 has no IP Address, I have no remote access to the box. So I''d like to at least give an IP Address to eth0. This seems to work, but like I said, it''s not documented to work. It just hit me what''s bugging me - it bugs me to give an IP Address to a logical device and cut off any kind of fallback access to the physical device. Maybe I''m looking for trouble that''s not there and creating problems I don''t need just because bridging is so new in my little world. I wonder what others have done in this situation? And then there''s my PPTP clients. I give these guys a 10.0.0.xxx IP Address with a gateway of 10.0.0.1. This was all part of the eth1 LAN when it was a pure router. I suppose in this bridging setup, I could make 10.0.0.1 an alias for br0 and leave eth1 with no IP Address. This just takes a little getting used to I guess. - Greg
On 6/7/2007 12:12 AM, Greg Scott wrote:> The IP Address for br0 is 1.2.3.2. I setup a system at 1.2.3.1 and > had it ping 1.2.3.2. Everything behaved as expected. With bridging > turned on, none of the iptables rules made any difference, so I > commented them out. With the cable plugged into eth0, none of the > pings replied. Tcpdump on the 1.2.3.2 box didn''t even show anything > coming into br0. Connecting to eth1 or eth2, pings replied and > tcpdump showed the ICMP going back and forth on br0. Setting br0 to > 1.2.3.2 and eth1 to 10.0.0.1, my external host could ping both IP > Addresses when I fudged in appropriate routes on my test system.Ok, good to know. The only other thing that I can think of why you might bind your IP address(es) to the physical interface verses the bridge interface is if you were not wanting to bridge IP, but some other non routeable protocol, say NetBEUI. (But why would you want to do that???) In this scenario, it would be perfectly normal to use a bridging router to bridge any thing but IP and route IP.> Here are a couple of my challenges. > > I want to block anything coming in from the Internet claiming to come > from 10/8, or 172.16/12 or 192.168/16. But anything from these > address blocks from any trusted eth-- interface should be OK. Easy > to do with pure routing, seems a little more challenging with > bridging. I think I can whip up some rules, but they depend on > ebtables filtering by physical eth-- port.You should be able to easily do this with a few EBTables rules. Just check the source IP of the packets as they come in eth0. ebtables -t filter -N bogonsFromNet -P RETURN ebtables -t filter -A bogonsFromNet -s 10.0.0.0/8 -j DROP ebtables -t filter -A bogonsFromNet -s 172.16.0.0/12 -j DROP ebtables -t filter -A bogonsFromNet -s 192.168.0.0/16 -j DROP ebtables -t filter -A INPUT -j bogonsFromNet ebtables -t filter -A FORWARD -j bogonsFromNet (Note: This is untested.) If I have things correct, this should be a simple chain that checks the source IP and blocks drops them if they match. The (new) chain its self has a default policy of RETURN to return back to the chain that jumped to it. You might want to expand your bogons list a bit more to include some more forbidden networks. I.e. Test Net 192.0.2.x/24... RFC 3330 is your friend in this matter.> I have a public /24 block, call it 1.2.3.nnn. Some of these are on > eth0, some on eth1, a few others on eth2. Bridging seems to handle > this nicely. eth0 faces the Internet. I am a little nervous in > packaging all this. Let''s say something goes haywire in startup, > before I run the brctl stuff. Or let''s say something bad happens and > bridging goes haywire. If eth0 has no IP Address, I have no remote > access to the box. So I''d like to at least give an IP Address to > eth0. This seems to work, but like I said, it''s not documented to > work. It just hit me what''s bugging me - it bugs me to give an IP > Address to a logical device and cut off any kind of fallback access > to the physical device. Maybe I''m looking for trouble that''s not > there and creating problems I don''t need just because bridging is so > new in my little world. I wonder what others have done in this > situation?I don''t think you are being too paranoid, but I think you should be aware of something I have run in to in my own testing. If I have eth0 up with an IP address w.x.y.z and I am connected to it (via ssh) and I enslave it to a bridge (bri0) (I like 3 letter / 1 number device names) my connection goes dead. I have to try reconnecting or log in to console and move the IP to the bridge. Well, at least I think that''s what I have to do, it''s been too long sense I last did it and I don''t remember the details. You may want to play with this. The reason that I bring it up is that I think an IP address on the ether interface is different than an IP address on the enslaved ether interface. Your mileage may vary. What I''m worried about is that if the bridging code is seeing the traffic before the system''s IP stack sees it (supported by the fact that you can use EBTables to filter it), you are still passing through the bridging code even with the IP address on the enslaved ether interface. I have personally used bridging extensively for the past 4+ years and I''ve been ABSOLUTELY THRILLED with it. I have some systems with four interfaces bridged together with all IPs on the bri0 interface. I have another system that is 802.1q trunking to a Layer 2 managed switch with 25+ VLAN interfaces on the trunk. I''m then bridging the 25+ VLANs plus the enslaved ether interface in one bridge with the IP bound to bri0. I then have EBTables dividing up what access each VLAN has. Specifically, each VLAN can communicate with the bri0 interface and the bri0 interface can communicate with all VLANs, but the VLANs can not communicate with each other. I even threw in ARP tables and EBTables to masquerade MAC addresses b/c the managed switch that I was working with could only see any given MAC on one VLAN. Let''s just say this was fun to set up. In the end, the system has been up and running with out problems at multi megabit speeds for 3+ years.> And then there''s my PPTP clients. I give these guys a 10.0.0.xxx IP > Address with a gateway of 10.0.0.1. This was all part of the eth1 > LAN when it was a pure router. I suppose in this bridging setup, I > could make 10.0.0.1 an alias for br0 and leave eth1 with no IP > Address. This just takes a little getting used to I guess.I don''t think you should be scared of bridging and / or EBTables. With them you can do a LOT of things you could not do other wise. Grant. . . .
>> I want to block anything coming in from the Internet claiming to come>> from 10/8, or 172.16/12 or 192.168/16.> You should be able to easily do this with a few EBTables rules.Yup - I was off putting together something similar to what you did and just now saw your reply. It all tested good. I''ll paste it in below: # # ebtables rules for bridging # echo "ebtables rules" echo " Directing anything between the Internet and private IP Addresses to the bogus_ip chain" $EBTABLES -A FORWARD -i $INET_IFACE -p IPv4 --ip-src 10.0.0.0/8 -j bogus_ip $EBTABLES -A FORWARD -i $INET_IFACE -p IPv4 --ip-src 172.16.0.0/12 -j bogus_ip $EBTABLES -A FORWARD -i $INET_IFACE -p IPv4 --ip-src 192.168.0.0/16 -j bogus_ip $EBTABLES -A FORWARD -i $INET_IFACE -p IPv4 --ip-dst 10.0.0.0/8 -j bogus_ip $EBTABLES -A FORWARD -i $INET_IFACE -p IPv4 --ip-dst 172.16.0.0/12 -j bogus_ip $EBTABLES -A FORWARD -i $INET_IFACE -p IPv4 --ip-dst 192.168.0.0/16 -j bogus_ip $EBTABLES -A INPUT -i $INET_IFACE -p IPv4 --ip-src 10.0.0.0/8 -j bogus_ip $EBTABLES -A INPUT -i $INET_IFACE -p IPv4 --ip-src 172.16.0.0/12 -j bogus_ip $EBTABLES -A INPUT -i $INET_IFACE -p IPv4 --ip-src 192.168.0.0/16 -j bogus_ip $EBTABLES -A INPUT -i $INET_IFACE -p IPv4 --ip-dst 10.0.0.0/8 -j bogus_ip $EBTABLES -A INPUT -i $INET_IFACE -p IPv4 --ip-dst 172.16.0.0/12 -j bogus_ip $EBTABLES -A INPUT -i $INET_IFACE -p IPv4 --ip-dst 192.168.0.0/16 -j bogus_ip # # Set up the bogus_ip chain to log and drop packets to/from private IP addresses # echo "Setting up the bogus_ip chain to LOG and DROP spoofed packets" $EBTABLES -A bogus_ip --log-prefix " spoofed packet" $EBTABLES -A bogus_ip -j DROP I might see if I can do this with one set of rules in the PREROUTING chain.> I don''t think you are being too paranoid, but > I think you should be aware of something I have > run in to in my own testing. . . .Yup - I noticed similar behavior. So this is how I''ll handle it. eth0 gets an IP Address of 1.2.3.6 during normal bootup. And then when I do the brctl stuff, br0 gets 1.2.3.2. That way there''s never a conflict between the physical and logical. All the physical interfaces have unique addresses so I can route based on the address when it makes sense, or bridge based on the interface when that makes sense. I''m feeling lots better about all this. Hopefully this discussion can help others out there. - Greg
On 06/07/07 01:55, Greg Scott wrote:> I might see if I can do this with one set of rules in the PREROUTING > chain.That is exactly why I put the rules in a new chain and jumped to the chain. One place to maintain them. With the ability to jump to the chain and then return to the chain that you jumped from, things act more like a sub-routine in programming. - Create new chains with a default policy of RETURN. - DROP on failures with in the new chain. - Default policy of RETURN will return packet flow to the calling chain. - Jump to the new chain from INPUT and return from it if everything is ok. - Jump to the new chain from FORWARD and return from it if everything is ok. In fact, you could use the new chain as a (sub)chain from any chain with in the filter table.> Yup - I noticed similar behavior. So this is how I''ll handle it. > eth0 gets an IP Address of 1.2.3.6 during normal bootup. And then > when I do the brctl stuff, br0 gets 1.2.3.2. That way there''s never > a conflict between the physical and logical. All the physical > interfaces have unique addresses so I can route based on the address > when it makes sense, or bridge based on the interface when that makes > sense. I''m feeling lots better about all this. Hopefully this > discussion can help others out there.This is one way to do it. Just be careful of race conditions while you are bringing things up. Grant. . . .