Linus Lüssing
2014-Mar-03 21:27 UTC
[Bridge] bridge is not forwaring ICMP6 neighbor solicitation to KVM guest
Hi Jan, On Mon, Mar 03, 2014 at 02:47:15PM -0500, Jan Stancek wrote:> I'm seeing an issue where bridge (sometimes) stops forwarding ICMP6 > neighbor solicitation packets to KVM guest and as result KVM guest doesn't > respond with neighbor advertisement.Hm, okay, that's not supposed to happen.> The reason I think this packet is related is because when I send same exact > packet I'm often hitting same issue - bridge stops forwarding ICMP6 neigh. > solicitation packets to KVM guest.Yes, the MLD query is kicking the multicast snooping into gear. If there's never a query, then snooping is basically disabled (compare: "bridge: disable snooping if there is no querier").> > My current way to reproduce this is: > 0. host B IP / MAC is: 2620:52:0:1040:221:5aff:fe47:931c / 00:21:5a:47:93:1c > guest IP / MAC is: 2620:52:0:1040:5056:ff:fe00:29 / 52:56:00:00:00:29 > 1. host B is sending neigh solicit packets every 5 seconds with KVM guest IP > using ns6 from ipv6toolkit: http://www.si6networks.com/tools/ipv6toolkit/ > with parameters: > --src-address=2620:52:0:1040:221:5aff:fe47:931c --dst-address=ff02::1:ff00:0029 > -t 2620:52:0:1040:5056:ff:fe00:29 --link-src-address=00:21:5a:47:93:1c > --source-lla-opt=00:21:5a:47:93:1c --link-dst-address=33:33:ff:00:00:29 > tcpdump running on guest can see both solicit and advertisement packets > 2. wait ~5 minutes > 3. host B sends Multicast Listener Query packet described above > 4. tcpdump running on guest is no longer seeing any neigh solicit packetsJust to clarify, host B is behind eno1 and vnet0 is directly connected to the interface of the guest, no additional bridge or anything else on top of that, right? Would it be possible for you to upload the tcpdumps from host B (or if you can't tcpdump on host B, then capturing on eno1) and the guest somewhere and saying at which time/packet in the dumps it stops working (probably ~10 seconds after the query). Filtering for ICMPv6 should be sufficient. What I'm curious about is, whether the guest receives the MLD query and responds with an MLD report. I suspect that either the bridge doesn't get an MLD report and therefore is shutting down the according port or there's a bug in parsing the MLD report in the bridge code. Thanks for the detailed report so far! Cheers, Linus -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: Digital signature URL: <http://lists.linuxfoundation.org/pipermail/bridge/attachments/20140303/d6043ba0/attachment-0001.sig>
Vlad Yasevich
2014-Mar-03 21:40 UTC
Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest
On 03/03/2014 04:27 PM, Linus Lüssing wrote:> Hi Jan, > > On Mon, Mar 03, 2014 at 02:47:15PM -0500, Jan Stancek wrote: >> I'm seeing an issue where bridge (sometimes) stops forwarding ICMP6 >> neighbor solicitation packets to KVM guest and as result KVM guest doesn't >> respond with neighbor advertisement. > > Hm, okay, that's not supposed to happen. > >> The reason I think this packet is related is because when I send same exact >> packet I'm often hitting same issue - bridge stops forwarding ICMP6 neigh. >> solicitation packets to KVM guest. > > Yes, the MLD query is kicking the multicast snooping into gear. If > there's never a query, then snooping is basically disabled > (compare: "bridge: disable snooping if there is no querier"). > >> >> My current way to reproduce this is: >> 0. host B IP / MAC is: 2620:52:0:1040:221:5aff:fe47:931c / 00:21:5a:47:93:1c >> guest IP / MAC is: 2620:52:0:1040:5056:ff:fe00:29 / 52:56:00:00:00:29 >> 1. host B is sending neigh solicit packets every 5 seconds with KVM guest IP >> using ns6 from ipv6toolkit: http://www.si6networks.com/tools/ipv6toolkit/ >> with parameters: >> --src-address=2620:52:0:1040:221:5aff:fe47:931c --dst-address=ff02::1:ff00:0029 >> -t 2620:52:0:1040:5056:ff:fe00:29 --link-src-address=00:21:5a:47:93:1c >> --source-lla-opt=00:21:5a:47:93:1c --link-dst-address=33:33:ff:00:00:29 >> tcpdump running on guest can see both solicit and advertisement packets >> 2. wait ~5 minutes >> 3. host B sends Multicast Listener Query packet described above >> 4. tcpdump running on guest is no longer seeing any neigh solicit packets > > Just to clarify, host B is behind eno1 and vnet0 is directly > connected to the interface of the guest, no additional bridge or > anything else on top of that, right? > > Would it be possible for you to upload the tcpdumps from host B > (or if you can't tcpdump on host B, then capturing on eno1) > and the guest somewhere and saying at which time/packet in the dumps > it stops working (probably ~10 seconds after the query). Filtering > for ICMPv6 should be sufficient. > > What I'm curious about is, whether the guest receives > the MLD query and responds with an MLD report. I suspect that > either the bridge doesn't get an MLD report and therefore is > shutting down the according port or there's a bug in parsing the > MLD report in the bridge code. >I did notice a minor issue in the bridge code. The following code: /* Prevent flooding this packet if there is no listener present */ if (!ipv6_addr_is_ll_all_nodes(&ip6h->daddr)) BR_INPUT_SKB_CB(skb)->mrouters_only = 1; if (ip6h->nexthdr != IPPROTO_HOPOPTS || ip6h->payload_len == 0) return 0; will mark most multicast traffic is mrouters_only. The two statement should be probably be reversed. However, that's shouldn't cause the reported problem. -vlad> > Thanks for the detailed report so far! > > Cheers, Linus >
Vlad Yasevich
2014-Mar-03 21:40 UTC
[Bridge] bridge is not forwaring ICMP6 neighbor solicitation to KVM guest
On 03/03/2014 04:27 PM, Linus L?ssing wrote:> Hi Jan, > > On Mon, Mar 03, 2014 at 02:47:15PM -0500, Jan Stancek wrote: >> I'm seeing an issue where bridge (sometimes) stops forwarding ICMP6 >> neighbor solicitation packets to KVM guest and as result KVM guest doesn't >> respond with neighbor advertisement. > > Hm, okay, that's not supposed to happen. > >> The reason I think this packet is related is because when I send same exact >> packet I'm often hitting same issue - bridge stops forwarding ICMP6 neigh. >> solicitation packets to KVM guest. > > Yes, the MLD query is kicking the multicast snooping into gear. If > there's never a query, then snooping is basically disabled > (compare: "bridge: disable snooping if there is no querier"). > >> >> My current way to reproduce this is: >> 0. host B IP / MAC is: 2620:52:0:1040:221:5aff:fe47:931c / 00:21:5a:47:93:1c >> guest IP / MAC is: 2620:52:0:1040:5056:ff:fe00:29 / 52:56:00:00:00:29 >> 1. host B is sending neigh solicit packets every 5 seconds with KVM guest IP >> using ns6 from ipv6toolkit: http://www.si6networks.com/tools/ipv6toolkit/ >> with parameters: >> --src-address=2620:52:0:1040:221:5aff:fe47:931c --dst-address=ff02::1:ff00:0029 >> -t 2620:52:0:1040:5056:ff:fe00:29 --link-src-address=00:21:5a:47:93:1c >> --source-lla-opt=00:21:5a:47:93:1c --link-dst-address=33:33:ff:00:00:29 >> tcpdump running on guest can see both solicit and advertisement packets >> 2. wait ~5 minutes >> 3. host B sends Multicast Listener Query packet described above >> 4. tcpdump running on guest is no longer seeing any neigh solicit packets > > Just to clarify, host B is behind eno1 and vnet0 is directly > connected to the interface of the guest, no additional bridge or > anything else on top of that, right? > > Would it be possible for you to upload the tcpdumps from host B > (or if you can't tcpdump on host B, then capturing on eno1) > and the guest somewhere and saying at which time/packet in the dumps > it stops working (probably ~10 seconds after the query). Filtering > for ICMPv6 should be sufficient. > > What I'm curious about is, whether the guest receives > the MLD query and responds with an MLD report. I suspect that > either the bridge doesn't get an MLD report and therefore is > shutting down the according port or there's a bug in parsing the > MLD report in the bridge code. >I did notice a minor issue in the bridge code. The following code: /* Prevent flooding this packet if there is no listener present */ if (!ipv6_addr_is_ll_all_nodes(&ip6h->daddr)) BR_INPUT_SKB_CB(skb)->mrouters_only = 1; if (ip6h->nexthdr != IPPROTO_HOPOPTS || ip6h->payload_len == 0) return 0; will mark most multicast traffic is mrouters_only. The two statement should be probably be reversed. However, that's shouldn't cause the reported problem. -vlad> > Thanks for the detailed report so far! > > Cheers, Linus >
Jan Stancek
2014-Mar-03 22:45 UTC
Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest
----- Original Message -----> From: "Linus Lüssing" <linus.luessing@web.de> > To: "Jan Stancek" <jstancek@redhat.com> > Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, bridge@lists.linux-foundation.org > Sent: Monday, 3 March, 2014 10:27:59 PM > Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest > > Hi Jan, > > On Mon, Mar 03, 2014 at 02:47:15PM -0500, Jan Stancek wrote: > > I'm seeing an issue where bridge (sometimes) stops forwarding ICMP6 > > neighbor solicitation packets to KVM guest and as result KVM guest doesn't > > respond with neighbor advertisement. > > Hm, okay, that's not supposed to happen. > > > The reason I think this packet is related is because when I send same exact > > packet I'm often hitting same issue - bridge stops forwarding ICMP6 neigh. > > solicitation packets to KVM guest. > > Yes, the MLD query is kicking the multicast snooping into gear. If > there's never a query, then snooping is basically disabled > (compare: "bridge: disable snooping if there is no querier"). > > > > > My current way to reproduce this is: > > 0. host B IP / MAC is: 2620:52:0:1040:221:5aff:fe47:931c / > > 00:21:5a:47:93:1c > > guest IP / MAC is: 2620:52:0:1040:5056:ff:fe00:29 / 52:56:00:00:00:29 > > 1. host B is sending neigh solicit packets every 5 seconds with KVM guest > > IP > > using ns6 from ipv6toolkit: > > http://www.si6networks.com/tools/ipv6toolkit/ > > with parameters: > > --src-address=2620:52:0:1040:221:5aff:fe47:931c > > --dst-address=ff02::1:ff00:0029 > > -t 2620:52:0:1040:5056:ff:fe00:29 --link-src-address=00:21:5a:47:93:1c > > --source-lla-opt=00:21:5a:47:93:1c --link-dst-address=33:33:ff:00:00:29 > > tcpdump running on guest can see both solicit and advertisement packets > > 2. wait ~5 minutes > > 3. host B sends Multicast Listener Query packet described above > > 4. tcpdump running on guest is no longer seeing any neigh solicit packets > > Just to clarify, host B is behind eno1 and vnet0 is directly > connected to the interface of the guest, no additional bridge or > anything else on top of that, right?Yes, host B should be behind eno1 (All hosts are remote to me). There should be only single bridge on host A. Host A has 3 more interfaces but those are all down. # cat /etc/sysconfig/network-scripts/ifcfg-eno1 DEVICE=eno1 ONBOOT=yes BRIDGE=br1 HWADDR=00:23:ae:ed:1a:00 # cat /etc/sysconfig/network-scripts/ifcfg-br1 DEVICE=br1 BOOTPROTO=dhcp ONBOOT=yes TYPE=Bridge DELAY=0 There is also bridge on host B. I assume that doesn't matter but I could set up host B without bridge if needed.> > Would it be possible for you to upload the tcpdumps from host B > (or if you can't tcpdump on host B, then capturing on eno1) > and the guest somewhere and saying at which time/packet in the dumps > it stops working (probably ~10 seconds after the query). Filtering > for ICMPv6 should be sufficient.Here are tcpdumps from hostA, hostB and guest (on hostA): http://jan.stancek.eu/tmp/neigh_solicit_and_bridge_traces1/ I didn't apply any filter, because that multicast query wasn't showing up for some reason when I tried to filter by icmp6. What I did: 1. started tcpdump on all systems 2. send 3 neigh. solicit from hostB manually with couple seconds in between 3. send multicast listener query from hostB manually 4. send 5 neigh. solicit from hostB manually with couple seconds in between hostA.cap tcpdump -i eno1 -w hostA.cap frame 124, 125 -> OK frame 217, 218 -> OK frame 291, 292 -> OK frame 373 -> Multicast Listener Query frame 484 -> no reply? frame 572 -> no reply? frame 665 -> no reply? hostB.cap tcpdump -i br0 -w hostB.cap frame 106, 108 -> OK frame 214, 216 -> OK frame 300, 302 -> OK frame 396 -> Multicast Listener Query frame 523 -> no reply? frame 623 -> no reply? frame 730 -> no reply? guest.cap tcpdump -i eth0 -w guest.cap frame 89, 90 -> OK frame 181, 182 -> OK frame 254, 255 -> OK frame 334 -> Multicast Listener Query no more neigh. solicit packets> > What I'm curious about is, whether the guest receives > the MLD query and responds with an MLD report. I suspect that > either the bridge doesn't get an MLD report and therefore is > shutting down the according port or there's a bug in parsing the > MLD report in the bridge code.I'm no expert in this area, but shouldn't neigh. solicit packets be forwarded to all ports regardless of any/no MLD reports? Regards, Jan> > > Thanks for the detailed report so far! > > Cheers, Linus >
Jan Stancek
2014-Mar-03 22:45 UTC
[Bridge] bridge is not forwaring ICMP6 neighbor solicitation to KVM guest
----- Original Message -----> From: "Linus L?ssing" <linus.luessing at web.de> > To: "Jan Stancek" <jstancek at redhat.com> > Cc: netdev at vger.kernel.org, "Florian Westphal" <fwestpha at redhat.com>, bridge at lists.linux-foundation.org > Sent: Monday, 3 March, 2014 10:27:59 PM > Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest > > Hi Jan, > > On Mon, Mar 03, 2014 at 02:47:15PM -0500, Jan Stancek wrote: > > I'm seeing an issue where bridge (sometimes) stops forwarding ICMP6 > > neighbor solicitation packets to KVM guest and as result KVM guest doesn't > > respond with neighbor advertisement. > > Hm, okay, that's not supposed to happen. > > > The reason I think this packet is related is because when I send same exact > > packet I'm often hitting same issue - bridge stops forwarding ICMP6 neigh. > > solicitation packets to KVM guest. > > Yes, the MLD query is kicking the multicast snooping into gear. If > there's never a query, then snooping is basically disabled > (compare: "bridge: disable snooping if there is no querier"). > > > > > My current way to reproduce this is: > > 0. host B IP / MAC is: 2620:52:0:1040:221:5aff:fe47:931c / > > 00:21:5a:47:93:1c > > guest IP / MAC is: 2620:52:0:1040:5056:ff:fe00:29 / 52:56:00:00:00:29 > > 1. host B is sending neigh solicit packets every 5 seconds with KVM guest > > IP > > using ns6 from ipv6toolkit: > > http://www.si6networks.com/tools/ipv6toolkit/ > > with parameters: > > --src-address=2620:52:0:1040:221:5aff:fe47:931c > > --dst-address=ff02::1:ff00:0029 > > -t 2620:52:0:1040:5056:ff:fe00:29 --link-src-address=00:21:5a:47:93:1c > > --source-lla-opt=00:21:5a:47:93:1c --link-dst-address=33:33:ff:00:00:29 > > tcpdump running on guest can see both solicit and advertisement packets > > 2. wait ~5 minutes > > 3. host B sends Multicast Listener Query packet described above > > 4. tcpdump running on guest is no longer seeing any neigh solicit packets > > Just to clarify, host B is behind eno1 and vnet0 is directly > connected to the interface of the guest, no additional bridge or > anything else on top of that, right?Yes, host B should be behind eno1 (All hosts are remote to me). There should be only single bridge on host A. Host A has 3 more interfaces but those are all down. # cat /etc/sysconfig/network-scripts/ifcfg-eno1 DEVICE=eno1 ONBOOT=yes BRIDGE=br1 HWADDR=00:23:ae:ed:1a:00 # cat /etc/sysconfig/network-scripts/ifcfg-br1 DEVICE=br1 BOOTPROTO=dhcp ONBOOT=yes TYPE=Bridge DELAY=0 There is also bridge on host B. I assume that doesn't matter but I could set up host B without bridge if needed.> > Would it be possible for you to upload the tcpdumps from host B > (or if you can't tcpdump on host B, then capturing on eno1) > and the guest somewhere and saying at which time/packet in the dumps > it stops working (probably ~10 seconds after the query). Filtering > for ICMPv6 should be sufficient.Here are tcpdumps from hostA, hostB and guest (on hostA): http://jan.stancek.eu/tmp/neigh_solicit_and_bridge_traces1/ I didn't apply any filter, because that multicast query wasn't showing up for some reason when I tried to filter by icmp6. What I did: 1. started tcpdump on all systems 2. send 3 neigh. solicit from hostB manually with couple seconds in between 3. send multicast listener query from hostB manually 4. send 5 neigh. solicit from hostB manually with couple seconds in between hostA.cap tcpdump -i eno1 -w hostA.cap frame 124, 125 -> OK frame 217, 218 -> OK frame 291, 292 -> OK frame 373 -> Multicast Listener Query frame 484 -> no reply? frame 572 -> no reply? frame 665 -> no reply? hostB.cap tcpdump -i br0 -w hostB.cap frame 106, 108 -> OK frame 214, 216 -> OK frame 300, 302 -> OK frame 396 -> Multicast Listener Query frame 523 -> no reply? frame 623 -> no reply? frame 730 -> no reply? guest.cap tcpdump -i eth0 -w guest.cap frame 89, 90 -> OK frame 181, 182 -> OK frame 254, 255 -> OK frame 334 -> Multicast Listener Query no more neigh. solicit packets> > What I'm curious about is, whether the guest receives > the MLD query and responds with an MLD report. I suspect that > either the bridge doesn't get an MLD report and therefore is > shutting down the according port or there's a bug in parsing the > MLD report in the bridge code.I'm no expert in this area, but shouldn't neigh. solicit packets be forwarded to all ports regardless of any/no MLD reports? Regards, Jan> > > Thanks for the detailed report so far! > > Cheers, Linus >
Linus Lüssing
2014-Mar-04 00:00 UTC
[Bridge] bridge is not forwaring ICMP6 neighbor solicitation to KVM guest
Hi Jan, On Mon, Mar 03, 2014 at 05:45:49PM -0500, Jan Stancek wrote:> There is also bridge on host B. I assume that doesn't matter > but I could set up host B without bridge if needed.It can matter, but in this case it doesn't :).> > What I'm curious about is, whether the guest receives > > the MLD query and responds with an MLD report. I suspect that > > either the bridge doesn't get an MLD report and therefore is > > shutting down the according port or there's a bug in parsing the > > MLD report in the bridge code. > > I'm no expert in this area, but shouldn't neigh. solicit packets > be forwarded to all ports regardless of any/no MLD reports?That's the beauty of IPv6 Neighbor Discovery using these neat solicited-node multicast addresses :). With IPv4 and ARP requests there's no other way than flooding. But for IPv6 we know in advance behind which bridge port someone interested in the neighbor solicitation message might be (assuming MLD is working, properly), allowing us to save bandwidth. In this case, MLD is not working properly, the main issue is the following: Host B sends broken MLD queries, the source address should be an IPv6 link-local one, not "100:0:600:0:78fb:100::". MLDv2 mandates this (see RFC3810, section 5.1.14.: "Source Addresses for Queries"). Though I couldn't find that requirement for MLDv1, Linux ignores MLDv1 queries with a non-link-local source address, too (see net/ipv6/mcast.c, igmp6_event_query() ). So Linux never sends an MLD report in reply to these broken queries. The second "minor" but in this case fatal issue is, that the bridge code doesn't have this link-local-src check, therefore kicking the snooping into gear even though it shouldn't because we don't have a _working_ querier. I'm going to make a patch for the bridge code adding this sanity check. For the broken query, ok, it's your manually crafted query. But did you see a query with such a bogus source address "in the wild", too? (I'm curious how urgent this sanity check is) Cheers, Linus -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: Digital signature URL: <http://lists.linuxfoundation.org/pipermail/bridge/attachments/20140304/4ca85342/attachment.sig>