Adrian Pascalau
2017-Dec-18 08:05 UTC
[Bridge] linux bridge does not forward arp reply back packets in a vmware vm
On Mon, Dec 18, 2017 at 4:54 AM, Toshiaki Makita <makita.toshiaki at lab.ntt.co.jp> wrote:> On 2017/12/17 5:01, Adrian P wrote: > ... >> Further investigation reveals something strange: when the >> communication starts with an arp request (which happens almost all the >> time), the bridge wrongly assigns the eth0 mac address to port 1, >> instead of port 3. >> >> Flow again: >> >> default gw --- vmware --- [ ens160 bridge tap ] --- eth0 >> >> On my bridge, ens160 is port 1, and the tap interface is port 3. Eth0 >> mac address is fa:16:3e:9a:04:95 >> >> What I have found is that in the forwarding table, the bridge wrongly >> assigns the eth0 mac address to the port 1, which is ens160 interface, >> instead of assigning it to the port 3, which is the tap interface. >> This happens only if the arp table in the cirros VM instance does not >> contain the mac address of the destination I am pinging (default gw in >> this case), so the cirros VM sends an arp request. See below the eth0 >> mac address wrongly assigned in the forwarding table to the port 1: >> >> # brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95 >> 1 fa:16:3e:9a:04:95 no 0.67 >> >> However, if I manually add the mac address of the destination IP I am >> pining into the cirros VM instance arp table, and there is no arp >> request sent, just icmp packets going out, then the bridge correctly >> assigns the eth0 mac address to the port 3, which is the tap >> interface, and everything starts working fine. See below the eth0 mac >> address correctly assigned in the forwarding table to the port 3: >> >> # brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95 >> 3 fa:16:3e:9a:04:95 no 9.26 >> > > It looks like the broadcast arp request is coming back from gw or > vmware. Would you check it?So if I look into the flow below, once more: default gw --- physical switch --- vmware --- [ ens160 bridge tap ] --- eth0 The arp request goes out on the eth0 interface, and enters the bridge on the tap interface. The bridge assigns the eth0 mac address to the tap interface, and sends the arp request out on the ens160 interface. Now some deice on the left side of the bridge (either de vmware, physical switch or the default gw), broadcasts that arp requests back, therefore the same arp request enters back the bridge on the ens160 interface, and the bridge assigns the source mac address of the arp request (which is still the eth0 mac address) to the ens160 port in the forwarding table, which causes the behavior I have noticed... This explains why I see multiple times the arp request when using tcpdump. By why there are three? Maybe two devices broadcasts back the arp request? # tcpdump -n -i ens160 arp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes 01:38:29.008219 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 28 01:38:29.008619 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 46 01:38:29.008732 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 46 01:38:29.009000 ARP, Reply 10.20.21.1 is-at 00:17:08:c4:52:80, length 46 What is strange in the above tcpdump is that the first arp request is 28 bytes in length (I suppose this is the one that enters the bridge on the tap interface), and the 2 others is 46 bytes in length (I suppose these are the ones that enters the bridge on the ens160 interface). If I open the capture in wireshark, I see some padding added in the last two arp requests, which does not appear in the first arp request: Ethernet II, Src: fa:16:3e:9a:04:95 (fa:16:3e:9a:04:95), Dst: Broadcast (ff:ff:ff:ff:ff:ff) Destination: Broadcast (ff:ff:ff:ff:ff:ff) Source: fa:16:3e:9a:04:95 (fa:16:3e:9a:04:95) Type: ARP (0x0806) Padding: 000000000000000000000000000000000000 Any thoughts how I can find out who duplicates the two additional arps?
Adrian Pascalau
2017-Dec-19 19:18 UTC
[Bridge] linux bridge does not forward arp reply back packets in a vmware vm
On Mon, Dec 18, 2017 at 10:05 AM, Adrian Pascalau <adrian27oradea at gmail.com> wrote:> On Mon, Dec 18, 2017 at 4:54 AM, Toshiaki Makita > <makita.toshiaki at lab.ntt.co.jp> wrote: >> On 2017/12/17 5:01, Adrian P wrote: >> ... >>> Further investigation reveals something strange: when the >>> communication starts with an arp request (which happens almost all the >>> time), the bridge wrongly assigns the eth0 mac address to port 1, >>> instead of port 3. >>> >>> Flow again: >>> >>> default gw --- vmware --- [ ens160 bridge tap ] --- eth0 >>> >>> On my bridge, ens160 is port 1, and the tap interface is port 3. Eth0 >>> mac address is fa:16:3e:9a:04:95 >>> >>> What I have found is that in the forwarding table, the bridge wrongly >>> assigns the eth0 mac address to the port 1, which is ens160 interface, >>> instead of assigning it to the port 3, which is the tap interface. >>> This happens only if the arp table in the cirros VM instance does not >>> contain the mac address of the destination I am pinging (default gw in >>> this case), so the cirros VM sends an arp request. See below the eth0 >>> mac address wrongly assigned in the forwarding table to the port 1: >>> >>> # brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95 >>> 1 fa:16:3e:9a:04:95 no 0.67 >>> >>> However, if I manually add the mac address of the destination IP I am >>> pining into the cirros VM instance arp table, and there is no arp >>> request sent, just icmp packets going out, then the bridge correctly >>> assigns the eth0 mac address to the port 3, which is the tap >>> interface, and everything starts working fine. See below the eth0 mac >>> address correctly assigned in the forwarding table to the port 3: >>> >>> # brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95 >>> 3 fa:16:3e:9a:04:95 no 9.26 >>> >> >> It looks like the broadcast arp request is coming back from gw or >> vmware. Would you check it? > > So if I look into the flow below, once more: > > default gw --- physical switch --- vmware --- [ ens160 bridge tap ] --- eth0 > > The arp request goes out on the eth0 interface, and enters the bridge > on the tap interface. The bridge assigns the eth0 mac address to the > tap interface, and sends the arp request out on the ens160 interface. > Now some deice on the left side of the bridge (either de vmware, > physical switch or the default gw), broadcasts that arp requests back, > therefore the same arp request enters back the bridge on the ens160 > interface, and the bridge assigns the source mac address of the arp > request (which is still the eth0 mac address) to the ens160 port in > the forwarding table, which causes the behavior I have noticed... > > This explains why I see multiple times the arp request when using > tcpdump. By why there are three? Maybe two devices broadcasts back the > arp request? > > # tcpdump -n -i ens160 arp > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes > 01:38:29.008219 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 28 > 01:38:29.008619 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 46 > 01:38:29.008732 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 46 > 01:38:29.009000 ARP, Reply 10.20.21.1 is-at 00:17:08:c4:52:80, length 46 > > What is strange in the above tcpdump is that the first arp request is > 28 bytes in length (I suppose this is the one that enters the bridge > on the tap interface), and the 2 others is 46 bytes in length (I > suppose these are the ones that enters the bridge on the ens160 > interface). If I open the capture in wireshark, I see some padding > added in the last two arp requests, which does not appear in the first > arp request: > > Ethernet II, Src: fa:16:3e:9a:04:95 (fa:16:3e:9a:04:95), Dst: > Broadcast (ff:ff:ff:ff:ff:ff) > Destination: Broadcast (ff:ff:ff:ff:ff:ff) > Source: fa:16:3e:9a:04:95 (fa:16:3e:9a:04:95) > Type: ARP (0x0806) > Padding: 000000000000000000000000000000000000 > > Any thoughts how I can find out who duplicates the two additional arps?Is there a way to trace the packet as it passes through the bridge? Something like a debug mode?