Cihangir Akturk
2017-Jan-01 15:37 UTC
[Bridge] Bridge forwards ARP packets but not forwards IP datagrams
Hi,
First of all, I would like to wish you a happy new year.
I am trying to write a toy network stack in userspace. In order
to test my code I am using a tap interface as a virtual ethernet
device. My setup is as follows;
br0 : bridge eth0 (public network interface)
/ \
/ \
/ \
eth1 tap0 (My program reads/writes to/from this interface)
(enslaved)
The above setup is runing in a KVM virtual machine. I implemented
ARP request/reply mechanism and looking at "arp -n" output I can
confirm that it runs as expected.
Now the problem I am experiencing is that br0 bridge forwards ARP
packets but not IP datagrams when they are sent from the host box.
When I say "host box", I mean the machine on which my test VM runs.
It seems that ARP packets are forwarded no matter which host (host
or guest box) they come from. But when it comes to IP datagrams,
they are forwarded to tap0 as expected if I send them from the same
host as tap0 interface (guest VM). If I send them from the host
box (I mean the computer on which my VM runs), IP datagrams reach
br0 bridge, but somehow they are not forwarded to tap0 interface,
I can see this with tcpdump.
I know the bridging functionality is independent of layer 3
protocols, and it should forward packets according to their layer
2 addresses, in this case their mac addresses.
VM mac addresses:
br0: fe:58:34:6b:8c:06
eth0: 52:54:00:99:c7:f2
eth1: 52:54:00:04:c7:f8
tap0: 46:22:c9:8a:16:ba
Host mac address:
virbr2: fe:54:00:04:c7:f8
Also please ignore the length field in the tcpdump output, because
it is possible that I might have sent a different input to netcat.
* When I try to send a UDP datagram from the host machine I get the
following results:
Host box;
[cakturk at portege unet]$ echo "foobar" | nc -u 172.28.128.44 33333
Guest VM, tcpdump listening on tap0
[root at unet-devel vagrant]# tcpdump -i tap0 -nnevv arp or udp
tcpdump: listening on tap0, link-type EN10MB (Ethernet), capture size 262144
bytes
12:32:32.172272 fe:54:00:04:c7:f8 > ff:ff:ff:ff:ff:ff, ethertype ARP
(0x0806), length 42: Ethernet (l
en 6), IPv4 (len 4), Request who-has 172.28.128.44 tell 172.28.128.1, length 28
12:32:32.172446 46:22:c9:8a:16:ba > fe:54:00:04:c7:f8, ethertype ARP
(0x0806), length 42: Ethernet (l
en 6), IPv4 (len 4), Reply 172.28.128.44 is-at 46:22:c9:8a:16:ba, length 28
Guest VM, tcpdump listening on br0
[root at unet-devel vagrant]# tcpdump -i br0 -nnevv arp or udp
tcpdump: listening on br0, link-type EN10MB (Ethernet), capture size 262144
bytes
13:02:46.643782 fe:54:00:04:c7:f8 > ff:ff:ff:ff:ff:ff, ethertype ARP
(0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has
172.28.128.44 tell 172.28.128.1, length 28
13:02:46.643996 46:22:c9:8a:16:ba > fe:54:00:04:c7:f8, ethertype ARP
(0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 172.28.128.44 is-at
46:22:c9:8a:16:ba, length 28
13:02:46.646676 fe:54:00:04:c7:f8 > 46:22:c9:8a:16:ba, ethertype IPv4
(0x0800), length 49: (tos 0x0, ttl 64, id 27172, offset 0, flags [DF], proto UDP
(17), length 35)
172.28.128.1.40327 > 172.28.128.44.33333: [udp sum ok] UDP, length 7
* But sending packets from the host machine to the guest VM I get the
following results:
Host machine;
[cakturk at portege unet]$ echo "foobar" | nc -u 172.28.128.44 33333
Guest VM tcpdump listening on tap0
[root at unet-devel vagrant]# tcpdump -i tap0 -nnevv arp or udp
tcpdump: listening on tap0, link-type EN10MB (Ethernet), capture size 262144
bytes
12:29:58.542681 fe:58:34:6b:8c:06 > ff:ff:ff:ff:ff:ff, ethertype ARP
(0x0806), length 42: Ethernet (l
en 6), IPv4 (len 4), Request who-has 172.28.128.44 tell 172.28.128.14, length 28
12:29:58.542928 46:22:c9:8a:16:ba > fe:58:34:6b:8c:06, ethertype ARP
(0x0806), length 42: Ethernet (l
en 6), IPv4 (len 4), Reply 172.28.128.44 is-at 46:22:c9:8a:16:ba, length 28
12:29:58.544910 fe:58:34:6b:8c:06 > 46:22:c9:8a:16:ba, ethertype IPv4
(0x0800), length 51: (tos 0x0,
ttl 64, id 1397, offset 0, flags [DF], proto UDP (17), length 37)
172.28.128.14.42506 > 172.28.128.44.33333: [udp sum ok] UDP, length 9
3 packets captured
3 packets received by filter
0 packets dropped by kernel
As can be clearly seen, when packets come from the host machine
ARP requests reach tap0, but IP/UDP datagrams are not forwarded to the
tap0 interface.
Can anyone give me any pointers on this?
Some additional information on my test environment:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Routing table, iptables and ebtables information for the guest VM.
-----------------------------------------------------------------
[root at unet-devel vagrant]# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.121.1 0.0.0.0 UG 0 0 0 eth0
0.0.0.0 172.28.128.1 0.0.0.0 UG 0 0 0 br0
172.28.128.0 0.0.0.0 255.255.255.0 U 0 0 0 br0
172.28.128.1 0.0.0.0 255.255.255.255 UH 0 0 0 br0
192.168.121.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
[root at unet-devel vagrant]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
[root at unet-devel vagrant]# ebtables -L
Bridge table: filter
Bridge chain: INPUT, entries: 0, policy: ACCEPT
Bridge chain: FORWARD, entries: 0, policy: ACCEPT
Bridge chain: OUTPUT, entries: 0, policy: ACCEPT
Routing table, iptables and ebtables information for the host machine
---------------------------------------------------------------------
[cakturk at portege unet]$ netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 wlp2s0
172.28.128.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr2
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 wlp2s0
192.168.121.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr1
192.168.124.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
[cakturk at portege unet]$ sudo iptables -L
[sudo] password for cakturk:
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps
ACCEPT all -- anywhere anywhere ctstate
RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere
INPUT_direct all -- anywhere anywhere
INPUT_ZONES_SOURCE all -- anywhere anywhere
INPUT_ZONES all -- anywhere anywhere
DROP all -- anywhere anywhere ctstate INVALID
REJECT all -- anywhere anywhere reject-with
icmp-host-prohibited
Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere 172.28.128.0/24 ctstate
RELATED,ESTABLISHED
ACCEPT all -- 172.28.128.0/24 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with
icmp-port-unreachable
REJECT all -- anywhere anywhere reject-with
icmp-port-unreachable
ACCEPT all -- anywhere 192.168.121.0/24 ctstate
RELATED,ESTABLISHED
ACCEPT all -- 192.168.121.0/24 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with
icmp-port-unreachable
REJECT all -- anywhere anywhere reject-with
icmp-port-unreachable
ACCEPT all -- anywhere 192.168.124.0/24 ctstate
RELATED,ESTABLISHED
ACCEPT all -- 192.168.124.0/24 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with
icmp-port-unreachable
REJECT all -- anywhere anywhere reject-with
icmp-port-unreachable
ACCEPT all -- anywhere anywhere ctstate
RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere
FORWARD_direct all -- anywhere anywhere
FORWARD_IN_ZONES_SOURCE all -- anywhere anywhere
FORWARD_IN_ZONES all -- anywhere anywhere
FORWARD_OUT_ZONES_SOURCE all -- anywhere anywhere
FORWARD_OUT_ZONES all -- anywhere anywhere
DROP all -- anywhere anywhere ctstate INVALID
REJECT all -- anywhere anywhere reject-with
icmp-host-prohibited
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:bootpc
ACCEPT udp -- anywhere anywhere udp dpt:bootpc
ACCEPT udp -- anywhere anywhere udp dpt:bootpc
OUTPUT_direct all -- anywhere anywhere
Chain FORWARD_IN_ZONES (1 references)
target prot opt source destination
FWDI_FedoraWorkstation all -- anywhere anywhere [goto]
FWDI_FedoraWorkstation all -- anywhere anywhere [goto]
Chain FORWARD_IN_ZONES_SOURCE (1 references)
target prot opt source destination
Chain FORWARD_OUT_ZONES (1 references)
target prot opt source destination
FWDO_FedoraWorkstation all -- anywhere anywhere [goto]
FWDO_FedoraWorkstation all -- anywhere anywhere [goto]
Chain FORWARD_OUT_ZONES_SOURCE (1 references)
target prot opt source destination
Chain FORWARD_direct (1 references)
target prot opt source destination
Chain FWDI_FedoraWorkstation (2 references)
target prot opt source destination
FWDI_FedoraWorkstation_log all -- anywhere anywhere
FWDI_FedoraWorkstation_deny all -- anywhere anywhere
FWDI_FedoraWorkstation_allow all -- anywhere anywhere
ACCEPT icmp -- anywhere anywhere
Chain FWDI_FedoraWorkstation_allow (1 references)
target prot opt source destination
Chain FWDI_FedoraWorkstation_deny (1 references)
target prot opt source destination
Chain FWDI_FedoraWorkstation_log (1 references)
target prot opt source destination
Chain FWDO_FedoraWorkstation (2 references)
target prot opt source destination
FWDO_FedoraWorkstation_log all -- anywhere anywhere
FWDO_FedoraWorkstation_deny all -- anywhere anywhere
FWDO_FedoraWorkstation_allow all -- anywhere anywhere
Chain FWDO_FedoraWorkstation_allow (1 references)
target prot opt source destination
Chain FWDO_FedoraWorkstation_deny (1 references)
target prot opt source destination
Chain FWDO_FedoraWorkstation_log (1 references)
target prot opt source destination
Chain INPUT_ZONES (1 references)
target prot opt source destination
IN_FedoraWorkstation all -- anywhere anywhere [goto]
IN_FedoraWorkstation all -- anywhere anywhere [goto]
Chain INPUT_ZONES_SOURCE (1 references)
target prot opt source destination
Chain INPUT_direct (1 references)
target prot opt source destination
Chain IN_FedoraWorkstation (2 references)
target prot opt source destination
IN_FedoraWorkstation_log all -- anywhere anywhere
IN_FedoraWorkstation_deny all -- anywhere anywhere
IN_FedoraWorkstation_allow all -- anywhere anywhere
ACCEPT icmp -- anywhere anywhere
Chain IN_FedoraWorkstation_allow (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere tcp dpt:sunrpc
ctstate NEW
ACCEPT udp -- anywhere anywhere udp dpt:sunrpc
ctstate NEW
ACCEPT tcp -- anywhere anywhere tcp dpt:mountd
ctstate NEW
ACCEPT udp -- anywhere anywhere udp dpt:mountd
ctstate NEW
ACCEPT tcp -- anywhere anywhere tcp dpt:nfs
ctstate NEW
ACCEPT udp -- anywhere 224.0.0.251 udp dpt:mdns
ctstate NEW
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh
ctstate NEW
ACCEPT udp -- anywhere anywhere udp dpt:netbios-ns
ctstate NEW
ACCEPT udp -- anywhere anywhere udp
dpt:netbios-dgm ctstate NEW
ACCEPT udp -- anywhere anywhere udp
dpts:blackjack:65535 ctstate NEW
ACCEPT tcp -- anywhere anywhere tcp
dpts:blackjack:65535 ctstate NEW
Chain IN_FedoraWorkstation_deny (1 references)
target prot opt source destination
Chain IN_FedoraWorkstation_log (1 references)
target prot opt source destination
Chain OUTPUT_direct (1 references)
target prot opt source destination
[cakturk at portege unet]$ sudo ebtables -L
Bridge table: filter
Bridge chain: INPUT, entries: 1, policy: ACCEPT
-j INPUT_direct
Bridge chain: FORWARD, entries: 1, policy: ACCEPT
-j FORWARD_direct
Bridge chain: OUTPUT, entries: 1, policy: ACCEPT
-j OUTPUT_direct
Bridge chain: INPUT_direct, entries: 0, policy: RETURN
Bridge chain: OUTPUT_direct, entries: 0, policy: RETURN
Bridge chain: FORWARD_direct, entries: 0, policy: RETURN
Arp entries in host machine:
[cakturk at portege unet]$ arp -n
Address HWtype HWaddress Flags Mask Iface
192.168.1.24 ether 34:36:3b:ca:ed:74 C
wlp2s0
172.28.128.14 ether fe:58:34:6b:8c:06 C
virbr2
192.168.1.1 ether 24:df:6a:fc:8d:32 C
wlp2s0
172.28.128.44 ether 46:22:c9:8a:16:ba C
virbr2
192.168.121.53 ether 52:54:00:99:c7:f2 C
virbr1
Arp entries in guest VM:
[vagrant at unet-devel ~]$ arp -n
Address HWtype HWaddress Flags Mask Iface
192.168.121.1 ether fe:54:00:99:c7:f2 C eth0
172.28.128.1 ether fe:54:00:04:c7:f8 C br0
172.28.128.44 ether 46:22:c9:8a:16:ba C br0
Cihangir.
Stephen Hemminger
2017-Jan-01 18:22 UTC
[Bridge] Bridge forwards ARP packets but not forwards IP datagrams
On Sun, 1 Jan 2017 18:37:49 +0300 Cihangir Akturk <cakturk at gmail.com> wrote:> Now the problem I am experiencing is that br0 bridge forwards ARP > packets but not IP datagrams when they are sent from the host box. > When I say "host box", I mean the machine on which my test VM runs. > It seems that ARP packets are forwarded no matter which host (host > or guest box) they come from. But when it comes to IP datagrams, > they are forwarded to tap0 as expected if I send them from the same > host as tap0 interface (guest VM). If I send them from the host > box (I mean the computer on which my VM runs), IP datagrams reach > br0 bridge, but somehow they are not forwarded to tap0 interface, > I can see this with tcpdump. > > I know the bridging functionality is independent of layer 3 > protocols, and it should forward packets according to their layer > 2 addresses, in this case their mac addresses.Short answer. It looks like you put an IP address on one of the devices enslaved into the bridge. Because of how the kernel L3 stack works you need to only put IP addresses on the bridge device and not the underlying Ethernet device. The only exception is if you are doing tricky things with brouting in iptables.