Sebastian
2018-Mar-25 10:09 UTC
[Pkg-xen-devel] Bug#894013: xen-utils-common: issue with iptables antispoofing rules in xen4.8 generated by vif-bridge and vif-common.sh
Package: xen-utils-common Version: 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 Severity: important Tags: patch security -- System Information: Debian Release: 9.4 APT prefers stable APT policy: (990, 'stable'), (500, 'stable-updates') Architecture: amd64 (x86_64) Kernel: Linux 4.9.0-6-amd64 (SMP w/4 CPU cores) Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8), LANGUAGE=de_DE.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages xen-utils-common depends on: ii lsb-base 9.20161125 ii python 2.7.13-2 ii ucf 3.0036 ii udev 232-25+deb9u2 ii xenstore-utils 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 xen-utils-common recommends no packages. xen-utils-common suggests no packages. -- Configuration Files: /etc/default/xendomains changed [not included] /etc/xen/scripts/vif-bridge changed [not included] /etc/xen/scripts/vif-common.sh changed [not included] /etc/xen/xend-config.sxp changed [not included] /etc/xen/xl.conf changed [not included] -- no debconf information Hi all I've already filed this issue with the Debian user-list and XEN project - they asked me to file it here as well. On XEN project you can find it here: https://lists.xenproject.org/archives/html/xen-users/2018-03/msg00043.html I have issues with the on domU startup automatically generated antispoofing rules by /etc/xen/scripts/vif-bridge and /etc/xen/scripts/vif-common.sh Both are part of the Debian xen-utils-common package (4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 installed on Debian 9.4). A domU test01 has two virtual interfaces - vif-test01-INT and vif-test01-TEST, both are connected to separate bridges named brINT and brTEST. The brINT is just an internal bridge without connectivity to an outside network to just connect all domUs and the dom0. The IP addressfor the vif-test01-INT interface is 192.168.240.68. The automatically generated rules per domU are: 1 ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out vif-test01-INT --physdev-is-bridged 2 ACCEPT udp -- anywhere anywhere PHYSDEV match --physdev-in vif-test01-INT --physdev-is-bridged udp spt:bootpc dpt:bootps 3 ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out vif-test01-INT --physdev-is-bridged 4 ACCEPT all -- 192.168.240.68 anywhere PHYSDEV match --physdev-in vif-test01-INT --physdev-is-bridged 5 ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out vif-test01-TEST --physdev-is-bridged 6 ACCEPT udp -- anywhere anywhere PHYSDEV match --physdev-in vif-test01-TEST --physdev-is-bridged udp spt:bootpc dpt:bootps 7 ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out vif-test01-TEST --physdev-is-bridged 8 ACCEPT all -- test01 anywhere PHYSDEV match --physdev-in vif-test01-TEST --physdev-is-bridged ... 33 REJECT all -- anywhere anywhere reject-with icmp-port-unreachable>From what I see is that the rules 1/3 and 5/7 are doubled.The next issue is that antispoofing rules just don't work. If I change the ip adress of the vif-test01-INT interface to something like 192.168.240.168 IP packets between test01 and other domUs are still forwarded. If I manually change the iptables rules to something like (in this example just for the brINT connected interface): -A FORWARD -m physdev --physdev-is-bridged --physdev-in vif-test01-INT -p all ! -s 192.168.240.68 -j DROP -A FORWARD -m physdev --physdev-is-bridged --physdev-out vif-test01-INT -p all ! -d 192.168.240.68 -j DROP -A FORWARD -m physdev --physdev-is-bridged --physdev-in vif-test01-INT -p all -j ACCEPT -A FORWARD -m physdev --physdev-is-bridged --physdev-out vif-test01-INT -p all -j ACCEPT ... -A FORWARD -j REJECT --reject-with icmp-port-unreachable then antispoofing works and IP packets with IP addresses different then 192.168.240.68 get dropped. Can somebody confirm this is an issue? Or do I just not understand how the antispoofing rules work on a virtual bridge? Is there a way to diable generation of antispoofing rules automatically on domU startup? I could configure a different vif.default.script in xl.conf and write a wrapper script, but it might be easier to just disable it and load iptables rules manually. Why do I think that is a security issue? The antispoofing rules being set up automatically create the false impression of working. At least in my case they don't. If someone connects a domU to the Internet and hosts public services that system is under constant risk. If such a system got hacked the intruder might get access to other internal machines as well by manipulating the domU's IP address. Take for instance NFS authentication based on IP addresses as an example. One could argue here that a proper NFS authentication is needed here and that's completely true, but NFSv4 with Kerberos tickets for domU file services sounds like a nightmare. However dom0 and domU bring a perfect way for antispoofing rules - we know and trust all the interfaces as they are managed by the dom0 and we can apply filter on these. In a physical environment it is not that easy to obtain. Binding IP addresses to MAC addresses doesn't address this issue as MAC addresses easily can be spoofed too. So the equivalent in the physical world would be one-to-one physical connections between hosts and a firewall to bind IP addresses to real physical interfaces. That's a huge benefit of virtualization. Antispoofing much easier to get. Cheers, Sebastian
Hans van Kranenburg
2018-Nov-05 00:17 UTC
[Pkg-xen-devel] Bug#894013: Bug#894013: xen-utils-common: issue with iptables antispoofing rules in xen4.8 generated by vif-bridge and vif-common.sh
Hi Sebastian, On 03/25/2018 12:09 PM, Sebastian wrote:> [...] > > I've already filed this issue with the Debian user-list and XEN project - > they asked me to file it here as well. On XEN project you can find it here: > https://lists.xenproject.org/archives/html/xen-users/2018-03/msg00043.html > > I have issues with the on domU startup automatically generated > antispoofing rules by > > [...] > > The automatically generated rules per domU are: > > 1 ACCEPT all -- anywhere anywhere > PHYSDEV match --physdev-out vif-test01-INT --physdev-is-bridged > 2 ACCEPT udp -- anywhere anywhere > PHYSDEV match --physdev-in vif-test01-INT --physdev-is-bridged udp > spt:bootpc dpt:bootps > 3 ACCEPT all -- anywhere anywhere > PHYSDEV match --physdev-out vif-test01-INT --physdev-is-bridged > 4 ACCEPT all -- 192.168.240.68 anywhere > PHYSDEV match --physdev-in vif-test01-INT --physdev-is-bridged > 5 ACCEPT all -- anywhere anywhere > PHYSDEV match --physdev-out vif-test01-TEST --physdev-is-bridged > 6 ACCEPT udp -- anywhere anywhere > PHYSDEV match --physdev-in vif-test01-TEST --physdev-is-bridged udp > spt:bootpc dpt:bootps > 7 ACCEPT all -- anywhere anywhere > PHYSDEV match --physdev-out vif-test01-TEST --physdev-is-bridged > 8 ACCEPT all -- test01 anywhere > PHYSDEV match --physdev-in vif-test01-TEST --physdev-is-bridged > ... > 33 REJECT all -- anywhere anywhere > reject-with icmp-port-unreachable > > From what I see is that the rules 1/3 and 5/7 are doubled. > > The next issue is that antispoofing rules just don't work. If I change > the ip adress of the vif-test01-INT interface to something like > 192.168.240.168 IP packets between test01 and other domUs are still > forwarded.I actually never understood this whole thing, and I also don't use linux bridge, but openvswitch instead for many years now, so these rules don't do anything at all for me.> If I manually change the iptables rules to something like (in this > example just for the brINT connected interface): > > -A FORWARD -m physdev --physdev-is-bridged --physdev-in vif-test01-INT > -p all ! -s 192.168.240.68 -j DROP > -A FORWARD -m physdev --physdev-is-bridged --physdev-out vif-test01-INT > -p all ! -d 192.168.240.68 -j DROP > -A FORWARD -m physdev --physdev-is-bridged --physdev-in vif-test01-INT > -p all -j ACCEPT > -A FORWARD -m physdev --physdev-is-bridged --physdev-out vif-test01-INT > -p all -j ACCEPT > ... > -A FORWARD -j REJECT --reject-with icmp-port-unreachable > > then antispoofing works and IP packets with IP addresses different then > 192.168.240.68 get dropped.This also requires putting IP addresses in the guest config file, which I never do. So, I end up having these kinds of rules in the dom0 iptables: -A FORWARD -m physdev --physdev-out vif8.0 --physdev-is-bridged -j ACCEPT -A FORWARD -m physdev --physdev-in vif8.0 --physdev-is-bridged -j ACCEPT> Can somebody confirm this is an issue? Or do I just not understand how > the antispoofing rules work on a virtual bridge?The behaviour is an upstream thing... I think it got introduced here: commit 67e0d840e74f7a40087a0a4436ce6ecdd3811044 Author: mjw at wray-m-3.hpl.hp.com <mjw at wray-m-3.hpl.hp.com> Date: Fri Jul 9 10:38:33 2004 +0000 bitkeeper revision 1.1053.1.1 (40ee75a9YghVZFFolzFjyJngpxAbKg) Move to new model of network and vif control using shell scripts. ... And then it was moved around several times.> Is there a way to diable generation of antispoofing rules automatically > on domU startup? I could configure a different vif.default.script in > xl.conf and write a wrapper script, but it might be easier to just > disable it and load iptables rules manually.Just commenting the handle_iptable here and there will disable it. I think we should do that in our packaging. It's just annoying.> Why do I think that is a security issue? > The antispoofing rules being set up automatically create the false impression > of working. At least in my case they don't. If someone connects a domU to > the Internet and hosts public services that system is under constant risk. > If such a system got hacked the intruder might get access to other internal > machines as well by manipulating the domU's IP address.Yes, that's true. There is no proper working thing out of the box.> Take for instance NFS > authentication based on IP addresses as an example. One could argue here that > a proper NFS authentication is needed here and that's completely true, but > NFSv4 with Kerberos tickets for domU file services sounds like a nightmare. > However dom0 and domU bring a perfect way for antispoofing rules - we > know and trust all the interfaces as they are managed by the dom0 and we > can apply filter on these.Perfect... Maybe for very simple configurations, where you also specify some mac address and ip address in the guest config file. It's not perfect. E.g. the domU might be a router... how are you going to deal with specifying what IP addresses it can send traffic for? (only doing layer 2 in the dom0).> In a physical environment it is not that easy > to obtain. Binding IP addresses to MAC addresses doesn't address this > issue as MAC addresses easily can be spoofed too. So the equivalent in > the physical world would be one-to-one physical connections between > hosts and a firewall to bind IP addresses to real physical interfaces. > That's a huge benefit of virtualization. Antispoofing much easier to get.Anyway, implementing an actual working better alternative is something that has to happen upstream. Which still does mean we can help, but fixing this will not be a debian add-on in the packaging. I must admit I didn't spend a lot of time to understand the physdev rules, but it looks like you found a combination of rules that actually does what you think it has to do. If that's the case we could prepare a patch for upstream to fix it. And, remove the handle_iptable from vif-openvswitch, doh. Thanks, Hans
Sebastian Piecha
2018-Nov-06 13:36 UTC
[Pkg-xen-devel] Bug#894013: xen-utils-common: issue with iptables antispoofing rules in xen4.8 generated by vif-bridge and vif-common.sh
Hi Hans, I actually solved this issue by: 1) I disabled all the original iptables creation rules in the xen scripts. 2) I configure all IP-addresses and vif-names in the domU config files (I want control about the IP address I assign to a domU, if somebody changed the IP address in the domU the communication is broken then) 3) The iptables rules for the my virtualized firewall are loaded directly by iptables. 4) All iptables rules for the domU are created by the xen scripts - I created new iptables creation rules in the xen scripts. The advantage of this approach is that with every new domU the right iptables rules get created and I don't have to mess around with iptables. The drawback is that in case I reload iptables the antispoofing rules are gone - but without them all communication to the domUs is blocked (just not to the firewall), so I know immediately I need to insert these rules again. I had to adjust the iptables rules as the one I originally suggested only secure the first domU. With the second domU the rules get sidelined. To achieve antispoofing there are drop and accept rules needed per domU based on its IP address from the domU config. All drop rules for all domUs have to be evaluated first. The accept rules need to be evaluated after evaluation of the drop rules. It is a bit difficult to achieve that automatically in iptables without inspecting the existing rule base first. I achieved that by a trick - I created a new chain called domU and inserted that into the FORWARD chain (done by iptables rules config). In the new domU chain with the start of every single domU the drop rules get inserted (iptables -I) in the beginning of that chain, and the accept rules get appended (iptables -A). So I get the right order and don't need to inspect the iptables rule set. To get these rules evaluated by iptables a sysctl setting needs to be done: sysctl net.bridge.bridge-nf-call-iptables=1 With these rules should somebody change the IP address of a running domU the antispoofing rules would block any traffic. Only by changing the iptables rules or the IP address in the domU config file a communication would be possible. To do this dom0 access is needed. That is exactly what I want as I expose some domUs to the Internet. Patch for the xen script vim-common.sh: -----------8<------------ --- vif-common.sh 2018-11-02 17:13:30.813238234 +0100 +++ vif-common.sh.new 2018-11-06 12:22:46.179073064 +0100 @@ -124,15 +124,21 @@ { if [ "$command" == "online" -o "$command" == "add" ] then - local c="-I" + local ci="-I" + local ca="-A" else - local c="-D" + local ci="-D" + local ca="-D" fi - iptables "$c" FORWARD -m physdev --physdev-is-bridged --physdev-in "$dev" \ - "$@" -j ACCEPT 2>/dev/null && - iptables "$c" FORWARD -m physdev --physdev-is-bridged --physdev-out "$dev" \ - -j ACCEPT 2>/dev/null + iptables "$ca" domu -m physdev --physdev-is-bridged --physdev-out "$dev" \ + -p all -d "$2" -j ACCEPT 2>/dev/null && + iptables "$ca" domu -m physdev --physdev-is-bridged --physdev-in "$dev" \ + -p all -s "$2" -j ACCEPT 2>/dev/null && + iptables "$ci" domu -m physdev --physdev-is-bridged --physdev-out "$dev" \ + -p all ! -d "$2" -j DROP 2>/dev/null && + iptables "$ci" domu -m physdev --physdev-is-bridged --physdev-in "$dev" \ + -p all ! -s "$2" -j DROP 2>/dev/null if [ \( "$command" == "online" -o "$command" == "add" \) -a $? -ne 0 ] then @@ -169,11 +175,11 @@ frob_iptable -s "$addr" done - # Always allow the domain to talk to a DHCP server. - frob_iptable -p udp --sport 68 --dport 67 else # No IP addresses have been specified, so allow anything. - frob_iptable + # needs revision, currently not covered by iptables rules in frob_iptable, + # therefore commented out + # frob_iptable fi release_lock "iptables" -----------8<------------ And these are the iptable rules look like after some domUs got started (as it is rather long I copy the relevant FORWARD section with the new chain domU - domf is one domU, jump another and fw is the firewall, the interfaces are named vif-domU-bridge): Chain FORWARD (policy ACCEPT) target prot opt source destination domu all -- anywhere anywhere ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in vif-fw-OAM --physdev-is-bridged ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out vif-fw-OAM --physdev-is-bridged ... REJECT all -- anywhere anywhere reject-with icmp-port-unreachable ... Chain domu (1 references) target prot opt source destination DROP all -- !192.168.249.3 0.0.0.0/0 PHYSDEV match --physdev-in vif-jump-FE --physdev-is-bridged DROP all -- 0.0.0.0/0 !192.168.249.3 PHYSDEV match --physdev-out vif-jump-FE --physdev-is-bridged DROP all -- !192.168.240.6 0.0.0.0/0 PHYSDEV match --physdev-in vif-domf-INT --physdev-is-bridged DROP all -- 0.0.0.0/0 !192.168.240.6 PHYSDEV match --physdev-out vif-domf-INT --physdev-is-bridged DROP all -- !192.168.250.6 0.0.0.0/0 PHYSDEV match --physdev-in vif-domf-LAN --physdev-is-bridged DROP all -- 0.0.0.0/0 !192.168.250.6 PHYSDEV match --physdev-out vif-domf-LAN --physdev-is-bridged ACCEPT all -- 0.0.0.0/0 192.168.250.6 PHYSDEV match --physdev-out vif-domf-LAN --physdev-is-bridged ACCEPT all -- 192.168.250.6 0.0.0.0/0 PHYSDEV match --physdev-in vif-domf-LAN --physdev-is-bridged ACCEPT all -- 0.0.0.0/0 192.168.240.6 PHYSDEV match --physdev-out vif-domf-INT --physdev-is-bridged ACCEPT all -- 192.168.240.6 0.0.0.0/0 PHYSDEV match --physdev-in vif-domf-INT --physdev-is-bridged ACCEPT all -- 0.0.0.0/0 192.168.249.3 PHYSDEV match --physdev-out vif-jump-FE --physdev-is-bridged ACCEPT all -- 192.168.249.3 0.0.0.0/0 PHYSDEV match --physdev-in vif-jump-FE --physdev-is-bridged Cheers, Sebastian
Sebastian Piecha
2018-Nov-06 21:30 UTC
[Pkg-xen-devel] Bug#894013: xen-utils-common: issue with iptables antispoofing rules in xen4.8 generated by vif-bridge and vif-common.sh
The patch file has a little bug as I commented out the else branch in the last if section. Now it works again... --- vif-common.sh 2018-11-02 17:13:30.813238234 +0100 +++ vif-common.sh.new 2018-11-06 22:24:45.331845798 +0100 @@ -124,15 +124,30 @@ { if [ "$command" == "online" -o "$command" == "add" ] then - local c="-I" +### local c="-I" + local ci="-I" + local ca="-A" else - local c="-D" +### local c="-D" + local ci="-D" + local ca="-D" fi - iptables "$c" FORWARD -m physdev --physdev-is-bridged --physdev-in "$dev" \ - "$@" -j ACCEPT 2>/dev/null && - iptables "$c" FORWARD -m physdev --physdev-is-bridged --physdev-out "$dev" \ - -j ACCEPT 2>/dev/null +### entfernt, da Antispoofing-Regel nicht funktioniert +### iptables "$c" FORWARD -m physdev --physdev-is-bridged --physdev-in "$dev" \ +### "$@" -j ACCEPT 2>/dev/null && +### entfernt, da Antispoofing-Regel nicht funktioniert +### iptables "$c" FORWARD -m physdev --physdev-is-bridged --physdev-out "$dev" \ +### -j ACCEPT 2>/dev/null +### neue Antispoofing-Regeln + iptables "$ca" domu -m physdev --physdev-is-bridged --physdev-out "$dev" \ + -p all -d "$2" -j ACCEPT 2>/dev/null && + iptables "$ca" domu -m physdev --physdev-is-bridged --physdev-in "$dev" \ + -p all -s "$2" -j ACCEPT 2>/dev/null && + iptables "$ci" domu -m physdev --physdev-is-bridged --physdev-out "$dev" \ + -p all ! -d "$2" -j DROP 2>/dev/null && + iptables "$ci" domu -m physdev --physdev-is-bridged --physdev-in "$dev" \ + -p all ! -s "$2" -j DROP 2>/dev/null if [ \( "$command" == "online" -o "$command" == "add" \) -a $? -ne 0 ] then @@ -170,10 +185,14 @@ done # Always allow the domain to talk to a DHCP server. - frob_iptable -p udp --sport 68 --dport 67 +### entfernt, da DomUs kein DHCP benötigen, IP@ fix +### frob_iptable -p udp --sport 68 --dport 67 else # No IP addresses have been specified, so allow anything. - frob_iptable + # needs revision, currently not covered by iptables rules in frob_iptable, + # therefore commented out + # frob_iptable + true fi release_lock "iptables"
Hans van Kranenburg
2019-Jan-03 22:46 UTC
[Pkg-xen-devel] Bug#894013: xen-utils-common: issue with iptables antispoofing rules in xen4.8 generated by vif-bridge and vif-common.sh
Hi, On 11/6/18 10:30 PM, Sebastian Piecha wrote:> The patch file has a little bug as I commented out the else branch in > the last if section. Now it works again... > > [...]I think we should move this discussion to (or extend to) upstream Xen development. If you want, please do so by starting a topic on the xen-users mailing list, summarize things a bit and refer to the url of the Debian bug. What I'm going to do now is patch out the handle_iptable line in the scripts, because we already know the current behaviour is incorrect and useless. This Debian bug will remain open, and there will most likely not be a proper working replacement before the Buster release. Anyway, thanks for already sharing your thoughts and examples. Hans
Hans van Kranenburg
2019-Jan-03 23:01 UTC
[Pkg-xen-devel] Bug#894013: xen-utils-common: issue with iptables antispoofing rules in xen4.8 generated by vif-bridge and vif-common.sh
On 1/3/19 11:46 PM, Hans van Kranenburg wrote:> Hi, > > On 11/6/18 10:30 PM, Sebastian Piecha wrote: >> The patch file has a little bug as I commented out the else branch in >> the last if section. Now it works again... >> >> [...] > > I think we should move this discussion to (or extend to) upstream Xen > development. If you want, please do so by starting a topic on the > xen-users mailing list, summarize things a bit and refer to the url of > the Debian bug. > > [...]Ahem, I see that already happened, and that is how the story started. :) So, taking it to the next level... Getting this forward is more likely to happen with a proposed working alternative implementation as patch-set to the xen-devel list. There seems not to be much interest in this by default, and only complaining that things don't work often doesn't improve that situation. I still think proper anti-spoofing behaviour out of the box would be a really nice thing. But, as we've seen already, it's not trivial because the dom0 needs to have all kinds of information about what's happening inside the domU it starts. Hans
Hans van Kranenburg
2019-Feb-22 20:00 UTC
[Pkg-xen-devel] Bug#894013: Bug#894013: xen-utils-common: issue with iptables antispoofing rules in xen4.8 generated by vif-bridge and vif-common.sh
tags 894013 + upstream severity 894013 wishlist thanks On 1/4/19 12:01 AM, Hans van Kranenburg wrote:> On 1/3/19 11:46 PM, Hans van Kranenburg wrote: >> Hi, >> >> [...]I'm moving this one to the wishlist department of the bug list, and tagging upstream, because this needs to happen upstream. This doesn't mean I'm trying to sweep it under the carpet. This is one of the more interesting issues, but it is not in a state where we can just "do something" and fix it in the next package upload. Hans
Debian Bug Tracking System
2019-Feb-22 20:03 UTC
[Pkg-xen-devel] Processed: Re: Bug#894013: xen-utils-common: issue with iptables antispoofing rules in xen4.8 generated by vif-bridge and vif-common.sh
Processing commands for control at bugs.debian.org:> tags 894013 + upstreamBug #894013 [xen-utils-common] xen-utils-common: issue with iptables antispoofing rules in xen4.8 generated by vif-bridge and vif-common.sh Added tag(s) upstream.> severity 894013 wishlistBug #894013 [xen-utils-common] xen-utils-common: issue with iptables antispoofing rules in xen4.8 generated by vif-bridge and vif-common.sh Severity set to 'wishlist' from 'important'> thanksStopping processing here. Please contact me if you need assistance. -- 894013: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=894013 Debian Bug Tracking System Contact owner at bugs.debian.org with problems
Hi, Some water has passed under the bridge, and I feel the whole thing should be rewritten as nftables rules instead of iptables rules now. As a bonus, we could combine filters for mac spoofing and ip spoofing in one neat package. -- Aleksi Suhonen / TREX Regional Exchanges Oy `What I need,' shouted Ford, by way of clarifying his previous remarks, `is a strong drink and a peer-group.' -- Douglas Adams, Life the Universe and Everything