Pkg xen devel - Mar 2018 - Bug#894013: xen-utils-common: issue with iptables antispoofing rules in xen4.8 generated by vif-bridge and vif-common.sh

Package: xen-utils-common
Version: 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5
Severity: important
Tags: patch security



-- System Information:
Debian Release: 9.4
  APT prefers stable
  APT policy: (990, 'stable'), (500, 'stable-updates')
Architecture: amd64 (x86_64)

Kernel: Linux 4.9.0-6-amd64 (SMP w/4 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8),
LANGUAGE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages xen-utils-common depends on:
ii  lsb-base        9.20161125
ii  python          2.7.13-2
ii  ucf             3.0036
ii  udev            232-25+deb9u2
ii  xenstore-utils  4.8.3+comet2+shim4.10.0+comet3-1+deb9u5

xen-utils-common recommends no packages.

xen-utils-common suggests no packages.

-- Configuration Files:
/etc/default/xendomains changed [not included]
/etc/xen/scripts/vif-bridge changed [not included]
/etc/xen/scripts/vif-common.sh changed [not included]
/etc/xen/xend-config.sxp changed [not included]
/etc/xen/xl.conf changed [not included]

-- no debconf information


Hi all

I've already filed this issue with the Debian user-list and XEN project -
they asked me to file it here as well. On XEN project you can find it here:
https://lists.xenproject.org/archives/html/xen-users/2018-03/msg00043.html


I have issues with the on domU startup automatically generated
antispoofing rules by

/etc/xen/scripts/vif-bridge and
/etc/xen/scripts/vif-common.sh

Both are part of the Debian xen-utils-common package
(4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 installed on Debian 9.4).

A domU test01 has two virtual interfaces - vif-test01-INT and
vif-test01-TEST, both are connected to separate bridges named brINT and
brTEST. The brINT is just an internal bridge without connectivity to an
outside network to just connect all domUs and the dom0. The IP
addressfor the vif-test01-INT interface is 192.168.240.68.


The automatically generated rules per domU are:

1    ACCEPT     all  --  anywhere             anywhere            
PHYSDEV match --physdev-out vif-test01-INT --physdev-is-bridged
2    ACCEPT     udp  --  anywhere             anywhere            
PHYSDEV match --physdev-in vif-test01-INT --physdev-is-bridged udp
spt:bootpc dpt:bootps
3    ACCEPT     all  --  anywhere             anywhere            
PHYSDEV match --physdev-out vif-test01-INT --physdev-is-bridged
4    ACCEPT     all  --  192.168.240.68       anywhere            
PHYSDEV match --physdev-in vif-test01-INT --physdev-is-bridged
5    ACCEPT     all  --  anywhere             anywhere            
PHYSDEV match --physdev-out vif-test01-TEST --physdev-is-bridged
6    ACCEPT     udp  --  anywhere             anywhere            
PHYSDEV match --physdev-in vif-test01-TEST --physdev-is-bridged udp
spt:bootpc dpt:bootps
7    ACCEPT     all  --  anywhere             anywhere            
PHYSDEV match --physdev-out vif-test01-TEST --physdev-is-bridged
8    ACCEPT     all  --  test01               anywhere            
PHYSDEV match --physdev-in vif-test01-TEST --physdev-is-bridged
...
33   REJECT     all  --  anywhere             anywhere            
reject-with icmp-port-unreachable
>From what I see is that the rules 1/3 and 5/7 are doubled.
The next issue is that antispoofing rules just don't work. If I change
the ip adress of the vif-test01-INT interface to something like
192.168.240.168 IP packets between test01 and other domUs are still
forwarded.


If I manually change the iptables rules to something like (in this
example just for the brINT connected interface):

-A FORWARD -m physdev --physdev-is-bridged --physdev-in vif-test01-INT
-p all ! -s 192.168.240.68 -j DROP
-A FORWARD -m physdev --physdev-is-bridged --physdev-out vif-test01-INT
-p all ! -d 192.168.240.68 -j DROP
-A FORWARD -m physdev --physdev-is-bridged --physdev-in vif-test01-INT
-p all -j ACCEPT
-A FORWARD -m physdev --physdev-is-bridged --physdev-out vif-test01-INT
-p all -j ACCEPT
...
-A FORWARD -j REJECT --reject-with icmp-port-unreachable

then antispoofing works and IP packets with IP addresses different then
192.168.240.68 get dropped.


Can somebody confirm this is an issue? Or do I just not understand how
the antispoofing rules work on a virtual bridge?

Is there a way to diable generation of antispoofing rules automatically
on domU startup? I could configure a different vif.default.script in
xl.conf and write a wrapper script, but it might be easier to just
disable it and load iptables rules manually.

Why do I think that is a security issue?
The antispoofing rules being set up automatically create the false impression
of working. At least in my case they don't. If someone connects a domU to
the Internet and hosts public services that system is under constant risk.
If such a system got hacked the intruder might get access to other internal
machines as well by manipulating the domU's IP address. Take for instance
NFS
authentication based on IP addresses as an example. One could argue here that
a proper NFS authentication is needed here and that's completely true, but
NFSv4 with Kerberos tickets for domU file services sounds like a nightmare.
However dom0 and domU bring a perfect way for antispoofing rules - we
know and trust all the interfaces as they are managed by the dom0 and we
can apply filter on these. In a physical environment it is not that easy
to obtain. Binding IP addresses to MAC addresses doesn't address this
issue as MAC addresses easily can be spoofed too. So the equivalent in
the physical world would be one-to-one physical connections between
hosts and a firewall to bind IP addresses to real physical interfaces.
That's a huge benefit of virtualization. Antispoofing much easier to get.

Cheers,
Sebastian

Hi Sebastian,

On 03/25/2018 12:09 PM, Sebastian wrote:
> [...]
> 
> I've already filed this issue with the Debian user-list and XEN project
-
> they asked me to file it here as well. On XEN project you can find it here:
> https://lists.xenproject.org/archives/html/xen-users/2018-03/msg00043.html
> 
> I have issues with the on domU startup automatically generated
> antispoofing rules by
> 
> [...]
> 
> The automatically generated rules per domU are:
> 
> 1    ACCEPT     all  --  anywhere             anywhere            
> PHYSDEV match --physdev-out vif-test01-INT --physdev-is-bridged
> 2    ACCEPT     udp  --  anywhere             anywhere            
> PHYSDEV match --physdev-in vif-test01-INT --physdev-is-bridged udp
> spt:bootpc dpt:bootps
> 3    ACCEPT     all  --  anywhere             anywhere            
> PHYSDEV match --physdev-out vif-test01-INT --physdev-is-bridged
> 4    ACCEPT     all  --  192.168.240.68       anywhere            
> PHYSDEV match --physdev-in vif-test01-INT --physdev-is-bridged
> 5    ACCEPT     all  --  anywhere             anywhere            
> PHYSDEV match --physdev-out vif-test01-TEST --physdev-is-bridged
> 6    ACCEPT     udp  --  anywhere             anywhere            
> PHYSDEV match --physdev-in vif-test01-TEST --physdev-is-bridged udp
> spt:bootpc dpt:bootps
> 7    ACCEPT     all  --  anywhere             anywhere            
> PHYSDEV match --physdev-out vif-test01-TEST --physdev-is-bridged
> 8    ACCEPT     all  --  test01               anywhere            
> PHYSDEV match --physdev-in vif-test01-TEST --physdev-is-bridged
> ...
> 33   REJECT     all  --  anywhere             anywhere            
> reject-with icmp-port-unreachable
> 
> From what I see is that the rules 1/3 and 5/7 are doubled.
> 
> The next issue is that antispoofing rules just don't work. If I change
> the ip adress of the vif-test01-INT interface to something like
> 192.168.240.168 IP packets between test01 and other domUs are still
> forwarded.
I actually never understood this whole thing, and I also don't use linux
bridge, but openvswitch instead for many years now, so these rules don't
do anything at all for me.
> If I manually change the iptables rules to something like (in this
> example just for the brINT connected interface):
> 
> -A FORWARD -m physdev --physdev-is-bridged --physdev-in vif-test01-INT
> -p all ! -s 192.168.240.68 -j DROP
> -A FORWARD -m physdev --physdev-is-bridged --physdev-out vif-test01-INT
> -p all ! -d 192.168.240.68 -j DROP
> -A FORWARD -m physdev --physdev-is-bridged --physdev-in vif-test01-INT
> -p all -j ACCEPT
> -A FORWARD -m physdev --physdev-is-bridged --physdev-out vif-test01-INT
> -p all -j ACCEPT
> ...
> -A FORWARD -j REJECT --reject-with icmp-port-unreachable
> 
> then antispoofing works and IP packets with IP addresses different then
> 192.168.240.68 get dropped.
This also requires putting IP addresses in the guest config file, which
I never do.

So, I end up having these kinds of rules in the dom0 iptables:

-A FORWARD -m physdev --physdev-out vif8.0 --physdev-is-bridged -j ACCEPT
-A FORWARD -m physdev --physdev-in vif8.0 --physdev-is-bridged -j ACCEPT
> Can somebody confirm this is an issue? Or do I just not understand how
> the antispoofing rules work on a virtual bridge?
The behaviour is an upstream thing... I think it got introduced here:

commit 67e0d840e74f7a40087a0a4436ce6ecdd3811044
Author: mjw at wray-m-3.hpl.hp.com <mjw at wray-m-3.hpl.hp.com>
Date:   Fri Jul 9 10:38:33 2004 +0000

bitkeeper revision 1.1053.1.1 (40ee75a9YghVZFFolzFjyJngpxAbKg)

Move to new model of network and vif control using shell scripts.

...

And then it was moved around several times.
> Is there a way to diable generation of antispoofing rules automatically
> on domU startup? I could configure a different vif.default.script in
> xl.conf and write a wrapper script, but it might be easier to just
> disable it and load iptables rules manually.
Just commenting the handle_iptable here and there will disable it. I
think we should do that in our packaging. It's just annoying.
> Why do I think that is a security issue?
> The antispoofing rules being set up automatically create the false
impression
> of working. At least in my case they don't. If someone connects a domU
to
> the Internet and hosts public services that system is under constant risk.
> If such a system got hacked the intruder might get access to other internal
> machines as well by manipulating the domU's IP address.
Yes, that's true. There is no proper working thing out of the box.
> Take for instance NFS
> authentication based on IP addresses as an example. One could argue here
that
> a proper NFS authentication is needed here and that's completely true,
but
> NFSv4 with Kerberos tickets for domU file services sounds like a nightmare.
> However dom0 and domU bring a perfect way for antispoofing rules - we
> know and trust all the interfaces as they are managed by the dom0 and we
> can apply filter on these.
Perfect... Maybe for very simple configurations, where you also specify
some mac address and ip address in the guest config file.

It's not perfect. E.g. the domU might be a router... how are you going
to deal with specifying what IP addresses it can send traffic for? (only
doing layer 2 in the dom0).
> In a physical environment it is not that easy
> to obtain. Binding IP addresses to MAC addresses doesn't address this
> issue as MAC addresses easily can be spoofed too. So the equivalent in
> the physical world would be one-to-one physical connections between
> hosts and a firewall to bind IP addresses to real physical interfaces.
> That's a huge benefit of virtualization. Antispoofing much easier to
get.
Anyway, implementing an actual working better alternative is something
that has to happen upstream. Which still does mean we can help, but
fixing this will not be a debian add-on in the packaging.

I must admit I didn't spend a lot of time to understand the physdev
rules, but it looks like you found a combination of rules that actually
does what you think it has to do. If that's the case we could prepare a
patch for upstream to fix it. And, remove the handle_iptable from
vif-openvswitch, doh.

Thanks,
Hans

Hi Hans,

I actually solved this issue by:

1) I disabled all the original iptables creation rules in the xen scripts.
2) I configure all IP-addresses and vif-names in the domU config files
(I want control about the IP address I assign to a domU, if somebody
changed the IP address in the domU the communication is broken then)
3) The iptables rules for the my virtualized firewall are loaded directly by
iptables.
4) All iptables rules for the domU are created by the xen scripts - I
created new iptables creation rules in the xen scripts.

The advantage of this approach is that with every new domU the right
iptables rules get created and I don't have to mess around with
iptables. The drawback is that in case I reload iptables the
antispoofing rules are gone - but without them all communication to the
domUs is blocked (just not to the firewall), so I know immediately I
need to insert these rules again.

I had to adjust the iptables rules as the one I originally suggested
only secure the first domU. With the second domU the rules get
sidelined. To achieve antispoofing there are drop and accept rules
needed per domU based on its IP address from the domU config. All drop
rules for all domUs have to be evaluated first. The accept rules need to
be evaluated after evaluation of the drop rules. It is a bit difficult
to achieve that automatically in iptables without inspecting the
existing rule base first. I achieved that by a trick - I created a new
chain called domU and inserted that into the FORWARD chain (done by
iptables rules config). In the new domU chain with the start of every
single domU the drop rules get inserted (iptables -I) in the beginning
of that chain, and the accept rules get appended (iptables -A). So I get
the right order and don't need to inspect the iptables rule set.

To get these rules evaluated by iptables a sysctl setting needs to be done:
sysctl net.bridge.bridge-nf-call-iptables=1

With these rules should somebody change the IP address of a running domU
the antispoofing rules would block any traffic. Only by changing the
iptables rules or the IP address in the domU config file a communication
would be possible. To do this dom0 access is needed. That is exactly
what I want as I expose some domUs to the Internet.

Patch for the xen script vim-common.sh:
-----------8<------------
--- vif-common.sh	2018-11-02 17:13:30.813238234 +0100
+++ vif-common.sh.new	2018-11-06 12:22:46.179073064 +0100
@@ -124,15 +124,21 @@
 {
   if [ "$command" == "online" -o "$command" ==
"add" ]
   then
-    local c="-I"
+    local ci="-I"
+    local ca="-A"
   else
-    local c="-D"
+    local ci="-D"
+    local ca="-D"
   fi
 
-  iptables "$c" FORWARD -m physdev --physdev-is-bridged --physdev-in
"$dev" \
-    "$@" -j ACCEPT 2>/dev/null &&
-  iptables "$c" FORWARD -m physdev --physdev-is-bridged --physdev-out
"$dev" \
-    -j ACCEPT 2>/dev/null
+  iptables "$ca" domu -m physdev --physdev-is-bridged --physdev-out
"$dev" \
+    -p all -d "$2" -j ACCEPT 2>/dev/null &&
+  iptables "$ca" domu -m physdev --physdev-is-bridged --physdev-in
"$dev" \
+    -p all -s "$2" -j ACCEPT 2>/dev/null &&
+  iptables "$ci" domu -m physdev --physdev-is-bridged --physdev-out
"$dev" \
+    -p all ! -d "$2" -j DROP 2>/dev/null &&
+  iptables "$ci" domu -m physdev --physdev-is-bridged --physdev-in
"$dev" \
+    -p all ! -s "$2" -j DROP 2>/dev/null
 
   if [ \( "$command" == "online" -o "$command" ==
"add" \) -a $? -ne 0 ]
   then
@@ -169,11 +175,11 @@
         frob_iptable -s "$addr"
       done
 
-      # Always allow the domain to talk to a DHCP server.
-      frob_iptable -p udp --sport 68 --dport 67
   else
       # No IP addresses have been specified, so allow anything.
-      frob_iptable
+      # needs revision, currently not covered by iptables rules in
frob_iptable,
+      # therefore commented out
+      # frob_iptable
   fi
 
   release_lock "iptables"
-----------8<------------


And these are the iptable rules look like after some domUs got
started (as it is rather long I copy the relevant FORWARD section with
the new chain domU - domf is one domU, jump another and fw is the
firewall, the interfaces are named vif-domU-bridge):

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
domu       all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             PHYSDEV
match --physdev-in vif-fw-OAM --physdev-is-bridged
ACCEPT     all  --  anywhere             anywhere             PHYSDEV
match --physdev-out vif-fw-OAM --physdev-is-bridged
...
REJECT     all  --  anywhere             anywhere
reject-with icmp-port-unreachable

...

Chain domu (1 references)
target     prot opt source               destination
DROP       all  -- !192.168.249.3        0.0.0.0/0            PHYSDEV
match --physdev-in vif-jump-FE --physdev-is-bridged
DROP       all  --  0.0.0.0/0           !192.168.249.3        PHYSDEV
match --physdev-out vif-jump-FE --physdev-is-bridged
DROP       all  -- !192.168.240.6        0.0.0.0/0            PHYSDEV
match --physdev-in vif-domf-INT --physdev-is-bridged
DROP       all  --  0.0.0.0/0           !192.168.240.6        PHYSDEV
match --physdev-out vif-domf-INT --physdev-is-bridged
DROP       all  -- !192.168.250.6        0.0.0.0/0            PHYSDEV
match --physdev-in vif-domf-LAN --physdev-is-bridged
DROP       all  --  0.0.0.0/0           !192.168.250.6        PHYSDEV
match --physdev-out vif-domf-LAN --physdev-is-bridged
ACCEPT     all  --  0.0.0.0/0            192.168.250.6        PHYSDEV
match --physdev-out vif-domf-LAN --physdev-is-bridged
ACCEPT     all  --  192.168.250.6        0.0.0.0/0            PHYSDEV
match --physdev-in vif-domf-LAN --physdev-is-bridged
ACCEPT     all  --  0.0.0.0/0            192.168.240.6        PHYSDEV
match --physdev-out vif-domf-INT --physdev-is-bridged
ACCEPT     all  --  192.168.240.6        0.0.0.0/0            PHYSDEV
match --physdev-in vif-domf-INT --physdev-is-bridged
ACCEPT     all  --  0.0.0.0/0            192.168.249.3        PHYSDEV
match --physdev-out vif-jump-FE --physdev-is-bridged
ACCEPT     all  --  192.168.249.3        0.0.0.0/0            PHYSDEV
match --physdev-in vif-jump-FE --physdev-is-bridged


Cheers,
Sebastian

The patch file has a little bug as I commented out the else branch in
the last if section. Now it works again...


--- vif-common.sh    2018-11-02 17:13:30.813238234 +0100

+++ vif-common.sh.new    2018-11-06 22:24:45.331845798 +0100
@@ -124,15 +124,30 @@
 {
   if [ "$command" == "online" -o "$command" ==
"add" ]
   then
-    local c="-I"
+###    local c="-I"
+    local ci="-I"
+    local ca="-A"
   else
-    local c="-D"
+###    local c="-D"
+    local ci="-D"
+    local ca="-D"
   fi
 
-  iptables "$c" FORWARD -m physdev --physdev-is-bridged --physdev-in
"$dev" \
-    "$@" -j ACCEPT 2>/dev/null &&
-  iptables "$c" FORWARD -m physdev --physdev-is-bridged --physdev-out
"$dev" \
-    -j ACCEPT 2>/dev/null
+### entfernt, da Antispoofing-Regel nicht funktioniert
+###  iptables "$c" FORWARD -m physdev --physdev-is-bridged
--physdev-in
"$dev" \
+###    "$@" -j ACCEPT 2>/dev/null &&
+### entfernt, da Antispoofing-Regel nicht funktioniert
+###  iptables "$c" FORWARD -m physdev --physdev-is-bridged
--physdev-out "$dev" \
+###    -j ACCEPT 2>/dev/null
+### neue Antispoofing-Regeln
+  iptables "$ca" domu -m physdev --physdev-is-bridged --physdev-out
"$dev" \
+    -p all -d "$2" -j ACCEPT 2>/dev/null &&
+  iptables "$ca" domu -m physdev --physdev-is-bridged --physdev-in
"$dev" \
+    -p all -s "$2" -j ACCEPT 2>/dev/null &&
+  iptables "$ci" domu -m physdev --physdev-is-bridged --physdev-out
"$dev" \
+    -p all ! -d "$2" -j DROP 2>/dev/null &&
+  iptables "$ci" domu -m physdev --physdev-is-bridged --physdev-in
"$dev" \
+    -p all ! -s "$2" -j DROP 2>/dev/null
 
   if [ \( "$command" == "online" -o "$command" ==
"add" \) -a $? -ne 0 ]
   then
@@ -170,10 +185,14 @@
       done
 
       # Always allow the domain to talk to a DHCP server.
-      frob_iptable -p udp --sport 68 --dport 67
+### entfernt, da DomUs kein DHCP benötigen, IP@ fix
+###      frob_iptable -p udp --sport 68 --dport 67
   else
       # No IP addresses have been specified, so allow anything.
-      frob_iptable
+      # needs revision, currently not covered by iptables rules in
frob_iptable,
+      # therefore commented out
+      # frob_iptable
+      true
   fi
 
   release_lock "iptables"

Hi,

On 11/6/18 10:30 PM, Sebastian Piecha wrote:
> The patch file has a little bug as I commented out the else branch in
> the last if section. Now it works again...
> 
> [...]
I think we should move this discussion to (or extend to) upstream Xen
development. If you want, please do so by starting a topic on the
xen-users mailing list, summarize things a bit and refer to the url of
the Debian bug.

What I'm going to do now is patch out the handle_iptable line in the
scripts, because we already know the current behaviour is incorrect and
useless.

This Debian bug will remain open, and there will most likely not be a
proper working replacement before the Buster release.

Anyway, thanks for already sharing your thoughts and examples.

Hans

On 1/3/19 11:46 PM, Hans van Kranenburg wrote:
> Hi,
> 
> On 11/6/18 10:30 PM, Sebastian Piecha wrote:
>> The patch file has a little bug as I commented out the else branch in
>> the last if section. Now it works again...
>>
>> [...]
> 
> I think we should move this discussion to (or extend to) upstream Xen
> development. If you want, please do so by starting a topic on the
> xen-users mailing list, summarize things a bit and refer to the url of
> the Debian bug.
> 
> [...]
Ahem, I see that already happened, and that is how the story started. :)

So, taking it to the next level... Getting this forward is more likely
to happen with a proposed working alternative implementation as
patch-set to the xen-devel list.

There seems not to be much interest in this by default, and only
complaining that things don't work often doesn't improve that situation.

I still think proper anti-spoofing behaviour out of the box would be a
really nice thing. But, as we've seen already, it's not trivial because
the dom0 needs to have all kinds of information about what's happening
inside the domU it starts.

Hans

tags 894013 + upstream
severity 894013 wishlist
thanks

On 1/4/19 12:01 AM, Hans van Kranenburg wrote:
> On 1/3/19 11:46 PM, Hans van Kranenburg wrote:
>> Hi,
>>
>> [...]
I'm moving this one to the wishlist department of the bug list, and
tagging upstream, because this needs to happen upstream. This doesn't
mean I'm trying to sweep it under the carpet. This is one of the more
interesting issues, but it is not in a state where we can just "do
something" and fix it in the next package upload.

Hans

Processing commands for control at bugs.debian.org:
> tags 894013 + upstream
Bug #894013 [xen-utils-common] xen-utils-common: issue with iptables
antispoofing rules in xen4.8 generated by vif-bridge and vif-common.sh
Added tag(s) upstream.
> severity 894013 wishlist
Bug #894013 [xen-utils-common] xen-utils-common: issue with iptables
antispoofing rules in xen4.8 generated by vif-bridge and vif-common.sh
Severity set to 'wishlist' from
'important'
> thanks
Stopping processing here.

Please contact me if you need assistance.
-- 
894013: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=894013
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems

Hi,

Some water has passed under the bridge, and I feel the whole thing 
should be rewritten as nftables rules instead of iptables rules now.

As a bonus, we could combine filters for mac spoofing and ip spoofing in 
one neat package.

-- 
	Aleksi Suhonen / TREX Regional Exchanges Oy

     `What I need,' shouted Ford, by way of clarifying his
     previous remarks, `is a strong drink and a peer-group.'
        -- Douglas Adams, Life the Universe and Everything