netfilter buglog - Sep 2011 - [Bug 751] New: IPv6 bridging bug

http://bugzilla.netfilter.org/show_bug.cgi?id=751

           Summary: IPv6 bridging bug
           Product: iptables
           Version: unspecified
          Platform: x86_64
        OS/Version: Gentoo
            Status: NEW
          Severity: normal
          Priority: P3
         Component: ip6tables
        AssignedTo: netfilter-buglog at lists.netfilter.org
        ReportedBy: david at commroom.net
   Estimated Hours: 0.0


First and foremost, many kind thanks to all of the developers and maintainers
of IPtables and netfilter. Your work is much appreciated and I thank you for
it.

I am really hoping that somebody can shed some light and help me to discover
what I am doing wrong. I am really stumped and cannot seem to figure out why I
am unable to get this working. I do have another slightly older box sitting
here that is running a very similar (similar!) setup from a few kernels back (a
SUSE box, with 2-3.4) and everything works great.

I really want to apologize in advance if this bug is a duplicate, but I have
searched vigorously throughout this bugzilla and using many other sources but I
cannot seem to come up with an answer. Finally, please accept my apologies if
this bug report should be destined for iptables-kernel [or bridging] instead of
this section.

I am trying to write some IPv6 firewall rules for some HVM virtual machines I
am running on under a Xen kernel. I am using ip6tables v.1.4.12.1 compiled from
gentoo e-build iptables-1.4.12.1-r1.ebuild. I am running this on a Xen kernel.
Output for uname -srvmpio is as follows:

Linux 2.6.38-xen #1 SMP Mon Sep 26 09:46:29 PDT 2011 x86_64 Pentium(R)
Dual-Core CPU E6700 @ 3.20GHz GenuineIntel GNU/Linux

What I am doing is creating a bridge, br0, and allowing virtual HVM clients
hosted under a Xen machine to use this bridge. When I mention "HVM",
what I
mean, is, these virtual machines are not paravirtualized. The are fully
emulated x86 hosts. This is achieved using the vif-bridge script that
accompanies the xen-tools (I have emerged the xen-tools-3.4.2-r1.ebuild ebuild
provided by gentoo portage). Each HVM client increments a new tapx.x interface,
and also a corresponding vifx.x interface. I am not using the standard xenbr0
configuration that everybody else is talking about and using. My bridge looks
something like the following:

 (cloud) 
    |
[wired-eth0]-----|
                 |---------[br0 on XEN DOM0 HOST]
[wired-eth1]-----|           |
                             |---[vifx.x/tapx.x]----[eth0-HVM-virtmach.]

wired-eth0 is the only real interface connected to a real wire. eth1 is part of
the bridge, but not connected to anything.

I have configured my ip6tables policies with a default ACCEPT policy, and then
explicitly specified ACCEPT targets for the INPUT, OUTPUT, and FORWARD chains.

I have tried the following:
# ip6tables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all      anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all      anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all      anywhere             anywhere    

Because vifx.x and tapx.x interfaces are being used by the Xen host to add the
HVM hosts to the network bridge, I also tried the following configuration:
# ip6tables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all      anywhere             anywhere             PHYSDEV match
--physdev-is-in

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all      anywhere             anywhere             PHYSDEV match
--physdev-is-bridged

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all      anywhere             anywhere             PHYSDEV match
--physdev-is-out

Finally, I have tried them both combined, this way:
# ip6tables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere             PHYSDEV match
--physdev-is-in
ACCEPT     all      anywhere             anywhere             PHYSDEV match
--physdev-is-out

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere             PHYSDEV match
--physdev-is-bridged

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere             PHYSDEV match
--physdev-is-out
ACCEPT     all      anywhere             anywhere             PHYSDEV match
--physdev-is-in

Finally, since IPv6 has some multicast dependencies, I also fiddled around with
allowing ff00::/8 in my [-d] policy rules, but that didn't make a difference
or
change anything. I also messed around with allowing fe80::/64 as well because
this is part of IPv6; I didn't expect it to work or change anything because
I
had "ACCEPT" targets in my chains anyway and that should have taken
precedence
anyway (ff00::/8 and fe80::/64 or not). But I still tried it.

When I set /proc/sys/net/bridge/bridge-nf-call-ip6tables to 1 (e.g. echo 1 >
/proc/sys/net/bridge/bridge-nf-call-ip6tables) the HVM clients that are
connected to the br0 bridge cannot send or receive any IPv6 at all. No IPv6
router advertisements can be seen by the HVM clients; no IPv6 multicast can be
seen by the HVM clients. The HVM clients cannot send or receive any ICMPv6, not
even to link-local addresses in the same broadcast domain.

Wireshark and TCPdump confirm to me that no IPv6 packets are passing across
neither the tapx.x or vifx.x bridged interfaces for the HVM virtual machines.

The DOM0 machine's IPv6 configuration works perfectly. RADVD advertisements
are
seen and IPv6 stateless autoconfiguration works perfectly. IPv6 layer7
connectivity works perfectly with local destinations and also with the IPv6
global unicast for the DOM0 (the xen host system). The xen host has perfectly
good layer3, layer7 connectivity, even when subject to more stringent policies,
but the HVM guests on that machine do not have any IPv6 communication passing
at all.

The other odd thing to me is, IPv4 works perfectly with this configuration. For
this machine, I have also set "1" on
/proc/sys/net/bridge/bridge-nf-call-iptables (the ipv4 version). It seems to me
that a "1" has to be in both
/proc/sys/net/bridge/bridge-nf-call-iptables in
order to use IPTables with bridging and then
/proc/sys/net/bridge/bridge-nf-call-ip6tables also needs to be set to
"1" in
order to use IPv6, ip6tables with bridging so both of these are set to one. For
reference, these are both set to "1":
# for i in /proc/sys/net/bridge/*; do echo $i && cat $i; done
/proc/sys/net/bridge/bridge-nf-call-arptables
0
/proc/sys/net/bridge/bridge-nf-call-ip6tables
1
/proc/sys/net/bridge/bridge-nf-call-iptables
1
/proc/sys/net/bridge/bridge-nf-filter-pppoe-tagged
0
/proc/sys/net/bridge/bridge-nf-filter-vlan-tagged
0

Also, IPv6 forwarding is not enabled (enabling it didn't seem to make a
difference either - for giggles, I tried it both ways).

The IPv4 configuration for this machine and the VM guests works great! And it
does exactly what I want, perfectly as I expected; I even have some
restrictions on the IPv4 protocols with some of my IPv4 policies and everything
still works beautifully with IPv4. VM guests can reach layer3 and layer7 with
no problems.

But absolutely NO IPv6 packets will cross the same bridge, no matter what I do.
I have also used shorewall6 to configure my ip6tables policies, similarly
allowing the same policy (all all ACCEPT) and I end up with the same result (no
IPv6 crossing the bridge over to the VM's and vice-versa). This really has
me
stumped, because all of these machines should be in the same "broadcast
domain"
and I think that these hosts should be able to communicate using the bridge.

The very instant I echo a "0" to
/proc/sys/net/bridge/bridge-nf-call-ip6tables,
IPv6 communication resumes perfectly with the guests. This is the only way I am
able to get IPv6 communication working for the HVM guests on the bridge; but
this isn't good for me because I would rather set some ip6tables policies
for
the Xen host and for the VM machines that it is hosting for some better
security.

Like I mentioned above, I have another box sitting close by that is a little
older, that has the same configuration (maybe a little older version of
iptables and a little older kernel, say around 2-6.34) and the IPv6 ip6tables
works perfectly. All of the configuration for that machine is the same
(bridge-nf-call-ip6tables=1, same vif-bridge script, xen-3.4.1 instead of
3.4.2, ip6tables policy is installed and working, VM guests subject to policy
through the bridge, RA and SLAAC work great for the clients and so on).

The thing that has me a little nervous is that I have compiled in most of the
netfilter options, the xtable options, and the ebtables options; I am wondering
if perhaps somebody knows of a newer, more recent, netfilter/iptables/xtables
kernel option that perhaps might be suspect with this problem and hampering my
ability to use ip6tables with this configuration. Does anyone know of any
module or something that maybe I compiled in from iptables/netfilter/xtables
that would certainly break this like this? I can post out my kernel
configuration if somebody wants to have a look at it.

If not, is there anything else I should look for specifically when debugging?
If anybody has any insight or ideas, I would really appreciate some input and
feedback. Thank you for your attention, ideas and thinking about this.

Once again, you have my gratitude for IPtables and netfilter projects. Life
wouldn't be the same without these. Many kind thanks to those working on,
and
supporting these projects.


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.

http://bugzilla.netfilter.org/show_bug.cgi?id=751





--- Comment #1 from David Davidson <david at commroom.net>  2011-09-27
09:40:32 ---
Created an attachment (id=367)
 --> (http://bugzilla.netfilter.org/attachment.cgi?id=367)
kernel configuration

This is a copy of the kernel configuration used. I am hoping that somebody can
identify an option in here that would "jump out" and identify the
symptoms for
this problem, or maybe even point out some debugging tips or something I
missed.
Thank you again for your help and thinking about this.


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.

http://bugzilla.netfilter.org/show_bug.cgi?id=751


David Davidson <david at commroom.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |david at commroom.net




--- Comment #2 from David Davidson <david at commroom.net>  2011-09-30
06:34:21 ---
Update:
I did discover something I hadn't noticed before. The DOM0 >CAN< reach
the DOMU
guests and the guests can reach the DOM0, using IPv6, when the ip6tables
firewall is enabled (f.e. /proc/sys/net/bridge/bridge-nf-call-ip6tables = 1).
All of this internal communication still works through the bridge with
/proc/sys/net/bridge/bridge-nf-call-ip6tables = 1. It is still very strange to
me that the real hosts external to the DOM0 bridge cannot reach the VM guests
through the bridge, from the same network broadcast segment. For recap:

if /proc/sys/net/bridge/bridge-nf-call-ip6tables = 1, then:

[xen-guest] can reach [xen-dom0] through br0 using tapx.x/vifx.x. using ipv6.

[xen-dom0] can reach [xen-guest] through br0 using ipv6.

[realhosts-same-broadcast-segment] can reach [xen-dom0] through br0 using
eth0/eth1 using ipv6.

[xen-dom0] can reach [realhost-same-broadcast-segment] through br0 using
eth0/eth1 using ipv6.

[realhosts-same-broadcast-segment] CANNOT reach [xen-guest] (through DOM0's
br0
interface).

[xen-guest] CANNOT reach [realhosts-same-broadcast-segment] (through DOM0's
br0
interface).

For /proc/sys/net/bridge/bridge-nf-call-iptables = 1 (IPv4 version), all of the
above is still true, excluding the last 2 lines - the realhosts CAN reach the
VM's and the VM's CAN reach the realhosts (using IPv4 - everything
works). The
policies are very similar, if not identical for both the IPv6 tables and the
IPv4 tables. Here again, even if I make all of the policy targets
"ACCEPT", the
VM's are still not accessible using IPv6. 

Because communication between the DOM0 and DOMU works, and vice-versa, this
gives me the tendency to believe that the ip6tables code is working, but maybe
perhaps one of the chains isn't correct or maybe the code believes that this
communication should be "routed" or "brouted" even though it
is supposed to be
in the same network segment.

The other odd thing is, the /etc/xen/scripts/vif-bridge script installs 2
iptables rules in the forward chain for IPv4 each time it loads a VM. These 2
rules end up being:

# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             state
RELATED,ESTABLISHED PHYSDEV match --physdev-out vif2.0
ACCEPT     all  --  anywhere             anywhere             PHYSDEV match
--physdev-in vif2.0

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination 

This script does not produce any corresponding rules using ipv6tables. If I add
those same types of rules manually to the ipv6 chains, it still doesn't fix
the
communication issues I described above with ip6tables.

When /etc/xen/scripts/vif-bridge installs those rules for IPv4, the following
log appears in the syslog messages:

xt_physdev: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains
for non-bridged traffic is not supported anymore.

I did find another report in the mailing list that documents that this feature
was removed in kernel 2.6.20 due to a layering violation problem
(http://lists.netfilter.org/pipermail/netfilter/2007-September/069659.html).
Does this mean that this script is defective and should add
--physdev-is-bridged to these 2 rules? I haven't adjusted them because the
IPv4
works great, so I have left the script alone. But this makes me wonder if this
is why the ip6tables communication between the DOMU and real hosts isn't
working.

For the record, this traffic should be "bridged":
# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.000854b12d04       no              eth0
                                                        eth1
                                                        tap2.0
                                                        vif2.0

Again, if I try to add the same rules to the FORWARD chain using the ip6tables,
the result is still the same as above - domu guests still cannot reach
realhosts and realhosts still cannot reach domu guests using IPv6. It
doesn't
matter if I use "--physdev-is-bridged" on the 2 rules or not - the
result is
the same (I have tried it both ways).

The only way to allow the IPv6 reachability between the realhosts and the
VM's
is setting 0 to /proc/sys/net/bridge/bridge-nf-call-ip6tables. Then, everything
works the exact same way that the IPv4 works, but obviously losing the ability
to set stateful rules/restrictions for the IPv6 hosts. As soon as I set this to
0, neighbor discovery kicks off on the VM and the communication immediately
becomes successful.

I hope that describing these items in detail might help to better identify the
symptoms and topology - maybe somebody could explain to me what I have
overlooked, or if I am dealing with a strange bug. Again, I have tested this
with another box I have here running openSUSE, and it works great; it's just
a
little bit older. The configuration is almost identical though. Of course the
gentoo box is a little different because of the "gentoo" way of doing
things
but for the most part, the configuration and topology are just about the same
thing.

Or, perhaps I have a nasty kernel code conflicting with this - I compiled most
everything into this kernel with a lot of code built-in instead of modules; I
still fear that perhaps I have something nasty in there that might be breaking
this. I attached my kernel configuration in the last post just in case anyone
might know of something that would cause this odd behavior.

Many kind thanks again for any consideration of this and thinking about it.


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.

http://bugzilla.netfilter.org/show_bug.cgi?id=751





--- Comment #3 from David Davidson <david at commroom.net>  2011-10-01
23:38:30 ---
UPDATE:

I have discovered something very interesting related to this problem. The
problem goes away if I install an older kernel version; for example, the
openSUSE box I mentioned has a version of 2.6.34.
So I decided to compile from an older source tree on the Gentoo box to see what
happens, for giggles. Since I knew the openSUSE box had 2.6.34, I decided to
use the Gentoo ebuild "xen-sources-2.6.34-r4.ebuild" which is provided
by the
Gentoo Portage tree. Obviously, this accompanies a different set of "gentoo
xen" kernel patches than from the 2.6.38 ebuild.
It resolved the problem. Here's what I did:

1. I edited the /etc/portage/package.keywords and forced the older kernel
version:
=sys-kernel/xen-sources-2.6.34-r4

2. I re-emerged my xen-sources
emerge xen-sources

3. I changed directory to /usr/src/linux-2.6.34-xen-r4/
cd /usr/src/linux-2.6.34-xen-r4/

4. I copied the kernel config from the 2-6.38 setup.
cp ../linux-2.6.38-xen/.config .config

5. Built the kernel.
gmake
gmake modules_install
cp vmlinux /boot
(copy config-kernel-2.6.34-xen-r4, Module.symvers, symvers-kernel-2.6.34-xen-r4
to boot also).

6. Edited my GRUB configuration to boot the older kernel instead.

7. Reboot the box. Viola! It worked.


The problem went away! Notice how I even used the same kernel configuration.
ip6tables and the rules in my chains now work perfectly with
/proc/sys/net/bridge/bridge-nf-call-ip6tables set to "1".


This tells me that one of the following is true:

A. There is a problem with the bridge code using IPv6 in the newer kernel (in
2.6.38-xen).

B. There is a problem in the netfilter/ip6tables support code in the newer
kernel (in 2.6.38-xen).

C. There is a problem with the new xen patch set (xen-patches-2.6.38-2.tar.bz2
or AKA
http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.38-2.tar.bz2).
Perhaps something in the patch set breaks ip6tables or breaks bridging using
ipv6.

D. There is a new kernel module in 2.6.38-xen that breaks ip6tables, or
bridging, if its installed or compiled in (which is what I was worried about in
the first place - this is still a possibility).

Unfortunately I am not a super code person, and so I also haven't a clue
where
this bug should be directed next. I was hoping that the two different patch
sets were numbered the same, but they aren't. Perhaps I will try to analyze
this further to see which patches are included in both patch sets and deduce
which ones are new in the 2.6.38-xen patch set. This might help narrow this
down >if< the problem is in the patch sets (and not A, B, or D, which I
have
mentioned above).

If anybody has some good insight on this, please let me know. Many kind thanks
again for your attention and thinking about this.


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.