Darren Reed
2009-Apr-08 03:55 UTC
[crossbow-discuss] Broadcast ping problem with etherstub - known problem?
# dladm show-link | grep stub
stub0 etherstub 9000 unknown --
vnicstub0 vnic 9000 up stub0
vnicstub1 vnic 9000 up stub0
vnicstub2 vnic 9000 up stub0
# ifconfig vnicstub0
vnicstub0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 9000
index 4
inet 1.1.1.1 netmask ffffff00 broadcast 1.1.1.255
ether 2:8:20:0:3e:29
# ifconfig vnicstub2
vnicstub2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 9000
index 6
inet 1.1.1.3 netmask ffffff00 broadcast 1.1.1.255
ether 2:8:20:9a:cc:f0
# ifconfig vnicstub1
ifconfig: status: SIOCGLIFFLAGS: vnicstub1: no such interface
# ping -s -i 1.1.1.3 1.1.1.255
PING 1.1.1.255: 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0. time=1.535 ms
64 bytes from 1.1.1.3: icmp_seq=0. time=12.149 ms
64 bytes from 1.1.1.3: icmp_seq=0. time=20.863 ms
64 bytes from 1.1.1.1: icmp_seq=0. time=21.275 ms
... there should only be 2 replies...
Darren
Darren Reed
2009-Apr-08 05:33 UTC
[crossbow-discuss] Broadcast ping problem with etherstub - known problem?
On 7/04/09 08:55 PM, Darren Reed wrote:> # dladm show-link | grep stub > stub0 etherstub 9000 unknown -- > vnicstub0 vnic 9000 up stub0 > vnicstub1 vnic 9000 up stub0 > vnicstub2 vnic 9000 up stub0 > > # ifconfig vnicstub0 > vnicstub0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 9000 > index 4 > inet 1.1.1.1 netmask ffffff00 broadcast 1.1.1.255 > ether 2:8:20:0:3e:29 # ifconfig vnicstub2 > vnicstub2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 9000 > index 6 > inet 1.1.1.3 netmask ffffff00 broadcast 1.1.1.255 > ether 2:8:20:9a:cc:f0 # ifconfig vnicstub1 > ifconfig: status: SIOCGLIFFLAGS: vnicstub1: no such interface > > # ping -s -i 1.1.1.3 1.1.1.255 > PING 1.1.1.255: 56 data bytes > 64 bytes from 1.1.1.1: icmp_seq=0. time=1.535 ms > 64 bytes from 1.1.1.3: icmp_seq=0. time=12.149 ms > 64 bytes from 1.1.1.3: icmp_seq=0. time=20.863 ms > 64 bytes from 1.1.1.1: icmp_seq=0. time=21.275 ms > > ... there should only be 2 replies...and a local zone with an exclusive ip instance will add in its two packets worth too... Darren
Nicolas Droux
2009-Apr-08 21:36 UTC
[crossbow-discuss] Broadcast ping problem with etherstub - known problem?
I think what''s happening here is that you have a distribution of the multicast packets done by the virtual switch between the VNICs, and since you have plumbed both VNICs with IP addresses on the same network from the same IP stack, IP will do its own multicast packet distribution between these interfaces. If the VNICs are plumbed by separate stacks, then you would see only the duplication happening at the MAC layer by the virtual switch. Nicolas. On Apr 7, 2009, at 9:55 PM, Darren Reed wrote:> # dladm show-link | grep stub > stub0 etherstub 9000 unknown -- > vnicstub0 vnic 9000 up stub0 > vnicstub1 vnic 9000 up stub0 > vnicstub2 vnic 9000 up stub0 > > # ifconfig vnicstub0 > vnicstub0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu > 9000 index 4 > inet 1.1.1.1 netmask ffffff00 broadcast 1.1.1.255 > ether 2:8:20:0:3e:29 # ifconfig vnicstub2 > vnicstub2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu > 9000 index 6 > inet 1.1.1.3 netmask ffffff00 broadcast 1.1.1.255 > ether 2:8:20:9a:cc:f0 # ifconfig vnicstub1 > ifconfig: status: SIOCGLIFFLAGS: vnicstub1: no such interface > > # ping -s -i 1.1.1.3 1.1.1.255 > PING 1.1.1.255: 56 data bytes > 64 bytes from 1.1.1.1: icmp_seq=0. time=1.535 ms > 64 bytes from 1.1.1.3: icmp_seq=0. time=12.149 ms > 64 bytes from 1.1.1.3: icmp_seq=0. time=20.863 ms > 64 bytes from 1.1.1.1: icmp_seq=0. time=21.275 ms > > ... there should only be 2 replies... > > Darren > > > > _______________________________________________ > crossbow-discuss mailing list > crossbow-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss-- Nicolas Droux - Solaris Kernel Networking - Sun Microsystems, Inc. droux at sun.com - http://blogs.sun.com/droux
Darren Reed
2009-Apr-08 21:46 UTC
[crossbow-discuss] Broadcast ping problem with etherstub - known problem?
I''ve filed 6828070 to track this issue... On 8/04/09 02:36 PM, Nicolas Droux wrote:> I think what''s happening here is that you have a distribution of the > multicast packets done by the virtual switch between the VNICs, and > since you have plumbed both VNICs with IP addresses on the same > network from the same IP stack, IP will do its own multicast packet > distribution between these interfaces. > > If the VNICs are plumbed by separate stacks, then you would see only > the duplication happening at the MAC layer by the virtual switch. > > Nicolas. > > On Apr 7, 2009, at 9:55 PM, Darren Reed wrote: > >> # dladm show-link | grep stub >> stub0 etherstub 9000 unknown -- >> vnicstub0 vnic 9000 up stub0 >> vnicstub1 vnic 9000 up stub0 >> vnicstub2 vnic 9000 up stub0 >> >> # ifconfig vnicstub0 >> vnicstub0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu >> 9000 index 4 >> inet 1.1.1.1 netmask ffffff00 broadcast 1.1.1.255 >> ether 2:8:20:0:3e:29 # ifconfig vnicstub2 >> vnicstub2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu >> 9000 index 6 >> inet 1.1.1.3 netmask ffffff00 broadcast 1.1.1.255 >> ether 2:8:20:9a:cc:f0 # ifconfig vnicstub1 >> ifconfig: status: SIOCGLIFFLAGS: vnicstub1: no such interface >> >> # ping -s -i 1.1.1.3 1.1.1.255 >> PING 1.1.1.255: 56 data bytes >> 64 bytes from 1.1.1.1: icmp_seq=0. time=1.535 ms >> 64 bytes from 1.1.1.3: icmp_seq=0. time=12.149 ms >> 64 bytes from 1.1.1.3: icmp_seq=0. time=20.863 ms >> 64 bytes from 1.1.1.1: icmp_seq=0. time=21.275 ms >> >> ... there should only be 2 replies... >> >> Darren >> >> >> >> _______________________________________________ >> crossbow-discuss mailing list >> crossbow-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >
Nicolas Droux
2009-Apr-09 01:50 UTC
[crossbow-discuss] Broadcast ping problem with etherstub - known problem?
On Apr 8, 2009, at 3:46 PM, Darren Reed wrote:> I''ve filed 6828070 to track this issue...I don''t see a bug here, the virtual switch in MAC is doing its job correctly, and IP is doing its own duplication since you have plumbed two interfaces on the same subnet in the same IP instances. Nicolas.> > > On 8/04/09 02:36 PM, Nicolas Droux wrote: >> I think what''s happening here is that you have a distribution of >> the multicast packets done by the virtual switch between the VNICs, >> and since you have plumbed both VNICs with IP addresses on the same >> network from the same IP stack, IP will do its own multicast packet >> distribution between these interfaces. >> >> If the VNICs are plumbed by separate stacks, then you would see >> only the duplication happening at the MAC layer by the virtual >> switch. >> >> Nicolas. >> >> On Apr 7, 2009, at 9:55 PM, Darren Reed wrote: >> >>> # dladm show-link | grep stub >>> stub0 etherstub 9000 unknown -- >>> vnicstub0 vnic 9000 up stub0 >>> vnicstub1 vnic 9000 up stub0 >>> vnicstub2 vnic 9000 up stub0 >>> >>> # ifconfig vnicstub0 >>> vnicstub0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu >>> 9000 index 4 >>> inet 1.1.1.1 netmask ffffff00 broadcast 1.1.1.255 >>> ether 2:8:20:0:3e:29 # ifconfig vnicstub2 >>> vnicstub2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu >>> 9000 index 6 >>> inet 1.1.1.3 netmask ffffff00 broadcast 1.1.1.255 >>> ether 2:8:20:9a:cc:f0 # ifconfig vnicstub1 >>> ifconfig: status: SIOCGLIFFLAGS: vnicstub1: no such interface >>> >>> # ping -s -i 1.1.1.3 1.1.1.255 >>> PING 1.1.1.255: 56 data bytes >>> 64 bytes from 1.1.1.1: icmp_seq=0. time=1.535 ms >>> 64 bytes from 1.1.1.3: icmp_seq=0. time=12.149 ms >>> 64 bytes from 1.1.1.3: icmp_seq=0. time=20.863 ms >>> 64 bytes from 1.1.1.1: icmp_seq=0. time=21.275 ms >>> >>> ... there should only be 2 replies... >>> >>> Darren >>> >>> >>> >>> _______________________________________________ >>> crossbow-discuss mailing list >>> crossbow-discuss at opensolaris.org >>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >> >-- Nicolas Droux - Solaris Kernel Networking - Sun Microsystems, Inc. droux at sun.com - http://blogs.sun.com/droux
Darren Reed
2009-Apr-09 01:58 UTC
[crossbow-discuss] Broadcast ping problem with etherstub - known problem?
On 8/04/09 06:50 PM, Nicolas Droux wrote:> On Apr 8, 2009, at 3:46 PM, Darren Reed wrote: > >> I''ve filed 6828070 to track this issue... > > I don''t see a bug here, the virtual switch in MAC is doing its job > correctly, and IP is doing its own duplication since you have plumbed > two interfaces on the same subnet in the same IP instances.In this case: ping -s 1.1.1.255 PING 1.1.1.255: 56 data bytes 64 bytes from 1.1.1.1: icmp_seq=0. time=1.712 ms 64 bytes from 1.1.1.2: icmp_seq=0. time=41.451 ms 64 bytes from 1.1.1.3: icmp_seq=0. time=48.621 ms 64 bytes from 1.1.1.3: icmp_seq=0. time=133.777 ms 64 bytes from 1.1.1.1: icmp_seq=0. time=134.181 ms 64 bytes from 1.1.1.2: icmp_seq=0. time=141.408 ms 1.1.1.2 belongs to a zone, test1, that has an exclusive IP instance. Darren
Nicolas Droux
2009-Apr-09 17:24 UTC
[crossbow-discuss] Broadcast ping problem with etherstub - known problem?
On Apr 8, 2009, at 7:58 PM, Darren Reed wrote:> On 8/04/09 06:50 PM, Nicolas Droux wrote: >> On Apr 8, 2009, at 3:46 PM, Darren Reed wrote: >> >>> I''ve filed 6828070 to track this issue... >> >> I don''t see a bug here, the virtual switch in MAC is doing its job >> correctly, and IP is doing its own duplication since you have >> plumbed two interfaces on the same subnet in the same IP instances. > > In this case: > ping -s 1.1.1.255 > PING 1.1.1.255: 56 data bytes > 64 bytes from 1.1.1.1: icmp_seq=0. time=1.712 ms > 64 bytes from 1.1.1.2: icmp_seq=0. time=41.451 ms > 64 bytes from 1.1.1.3: icmp_seq=0. time=48.621 ms > 64 bytes from 1.1.1.3: icmp_seq=0. time=133.777 ms > 64 bytes from 1.1.1.1: icmp_seq=0. time=134.181 ms > 64 bytes from 1.1.1.2: icmp_seq=0. time=141.408 ms > > 1.1.1.2 belongs to a zone, test1, that has an exclusive IP instance.Since you have plumbed the two VNICs on the same subnet in the global zone from which you are sending, IP will send a duplicate of the broadcast ICMP packets through the two ill''s corresponding to these VNICs. Because you created both of these VNICs on the same etherstub, and the VNIC of zone test1 is also created on top of the same etherstub, the zone test1 will receive two ICMP requests. If you snoop on the VNIC from the zone test1, you should see two ICMP requests with the same sequence number coming from the two IP addresses you configured in the global zone. If you had only one VNIC per subnet in your global zone, IP wouldn''t do the duplication and you would see only one reply per ICMP request for the hosts connected to the virtual switch. Nicolas.> > > Darren >-- Nicolas Droux - Solaris Kernel Networking - Sun Microsystems, Inc. droux at sun.com - http://blogs.sun.com/droux
Erik Nordmark
2009-Apr-10 16:45 UTC
[crossbow-discuss] Broadcast ping problem with etherstub - known problem?
Darren Reed wrote:> On 8/04/09 06:50 PM, Nicolas Droux wrote: >> On Apr 8, 2009, at 3:46 PM, Darren Reed wrote: >> >>> I''ve filed 6828070 to track this issue... >> >> I don''t see a bug here, the virtual switch in MAC is doing its job >> correctly, and IP is doing its own duplication since you have plumbed >> two interfaces on the same subnet in the same IP instances. > > In this case: > ping -s 1.1.1.255 > PING 1.1.1.255: 56 data bytes > 64 bytes from 1.1.1.1: icmp_seq=0. time=1.712 ms > 64 bytes from 1.1.1.2: icmp_seq=0. time=41.451 ms > 64 bytes from 1.1.1.3: icmp_seq=0. time=48.621 ms > 64 bytes from 1.1.1.3: icmp_seq=0. time=133.777 ms > 64 bytes from 1.1.1.1: icmp_seq=0. time=134.181 ms > 64 bytes from 1.1.1.2: icmp_seq=0. time=141.408 ms > > 1.1.1.2 belongs to a zone, test1, that has an exclusive IP instance.You''d see the same thing if instead of vnics you plugged N NICs into the same Ethernet switch (and configured them all to be on the same subnet). Solaris does some cute broadcast duplication out multiple ills since Solaris 2.0. It is described bridly in section 2.2 in http://jurassic.eng/~nordmark/ip-datapath-design.pdf (and that section incorrectly says it is only for network broadcast and all-ones, when it in fact appears any time the destination address is a valid broadcast address on more than one IP interface.) Erik
James Carlson
2009-Apr-10 16:58 UTC
[crossbow-discuss] Broadcast ping problem with etherstub - known problem?
Erik Nordmark writes:> Darren Reed wrote: > > 64 bytes from 1.1.1.1: icmp_seq=0. time=1.712 ms > > 64 bytes from 1.1.1.2: icmp_seq=0. time=41.451 ms > > 64 bytes from 1.1.1.3: icmp_seq=0. time=48.621 ms > > 64 bytes from 1.1.1.3: icmp_seq=0. time=133.777 ms > > 64 bytes from 1.1.1.1: icmp_seq=0. time=134.181 ms > > 64 bytes from 1.1.1.2: icmp_seq=0. time=141.408 ms > > > > 1.1.1.2 belongs to a zone, test1, that has an exclusive IP instance. > > You''d see the same thing if instead of vnics you plugged N NICs into the > same Ethernet switch (and configured them all to be on the same subnet).But if you did that, and complained about the behavior, wouldn''t we tell you that you''re doing something wrong and that the N NICs should actually be put into an IPMP group? If the N NICs are in an IPMP group, then we eliminate the duplicates by choosing just one interface as the designated receiver. I think this case is a little different, because we wouldn''t suggest putting all the VNICs into an IPMP group (would we?) and we do expect people to create multiple VNICs (don''t we?). -- James Carlson, Solaris Networking <james.d.carlson at sun.com> Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677
Nicolas Droux
2009-Apr-10 17:35 UTC
[crossbow-discuss] Broadcast ping problem with etherstub - known problem?
On Apr 10, 2009, at 10:58 AM, James Carlson wrote:> Erik Nordmark writes: >> Darren Reed wrote: >>> 64 bytes from 1.1.1.1: icmp_seq=0. time=1.712 ms >>> 64 bytes from 1.1.1.2: icmp_seq=0. time=41.451 ms >>> 64 bytes from 1.1.1.3: icmp_seq=0. time=48.621 ms >>> 64 bytes from 1.1.1.3: icmp_seq=0. time=133.777 ms >>> 64 bytes from 1.1.1.1: icmp_seq=0. time=134.181 ms >>> 64 bytes from 1.1.1.2: icmp_seq=0. time=141.408 ms >>> >>> 1.1.1.2 belongs to a zone, test1, that has an exclusive IP instance. >> >> You''d see the same thing if instead of vnics you plugged N NICs >> into the >> same Ethernet switch (and configured them all to be on the same >> subnet). > > But if you did that, and complained about the behavior, wouldn''t we > tell you that you''re doing something wrong and that the N NICs should > actually be put into an IPMP group? > > If the N NICs are in an IPMP group, then we eliminate the duplicates > by choosing just one interface as the designated receiver. > > I think this case is a little different, because we wouldn''t suggest > putting all the VNICs into an IPMP group (would we?) and we do expect > people to create multiple VNICs (don''t we?).But we wouldn''t expect people to create multiple VNICs on the same etherstub (virtual switch) *and* plumb these VNICs in the same zone *and* configure them on the same subnet, which is the case here. If someone still wants to do that for some reason, they will see the same behavior as if they were using physical NICs connected to the same physical switch. If the multiple VNICs are plumbed from different zones or VMs, then this additional outbound IP duplication of broadcast packets will not occur. Nicolas.> > > -- > James Carlson, Solaris Networking <james.d.carlson at sun.com > > > Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 > 2084 > MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 > 1677 > _______________________________________________ > crossbow-discuss mailing list > crossbow-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss-- Nicolas Droux - Solaris Kernel Networking - Sun Microsystems, Inc. droux at sun.com - http://blogs.sun.com/droux
James Carlson
2009-Apr-10 17:43 UTC
[crossbow-discuss] Broadcast ping problem with etherstub - known problem?
Nicolas Droux writes:> On Apr 10, 2009, at 10:58 AM, James Carlson wrote: > > I think this case is a little different, because we wouldn''t suggest > > putting all the VNICs into an IPMP group (would we?) and we do expect > > people to create multiple VNICs (don''t we?). > > But we wouldn''t expect people to create multiple VNICs on the same > etherstub (virtual switch) *and* plumb these VNICs in the same zone > *and* configure them on the same subnet, which is the case here.That makes sense, except that Darren says he also saw the anomalous behavior when putting VNICs into non-global zones as well.> If the multiple VNICs are plumbed from different zones or VMs, then > this additional outbound IP duplication of broadcast packets will not > occur.That''s not what it seems to say in the CR. -- James Carlson, Solaris Networking <james.d.carlson at sun.com> Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677
Darren Reed
2009-Apr-10 20:49 UTC
[crossbow-discuss] Broadcast ping problem with etherstub - known problem?
On 10/04/09 10:43 AM, James Carlson wrote:> Nicolas Droux writes: >> On Apr 10, 2009, at 10:58 AM, James Carlson wrote: >>> I think this case is a little different, because we wouldn''t suggest >>> putting all the VNICs into an IPMP group (would we?) and we do expect >>> people to create multiple VNICs (don''t we?). >> But we wouldn''t expect people to create multiple VNICs on the same >> etherstub (virtual switch) *and* plumb these VNICs in the same zone >> *and* configure them on the same subnet, which is the case here. > > That makes sense, except that Darren says he also saw the anomalous > behavior when putting VNICs into non-global zones as well.Well, I was only ever expecting "ping" to send out 1 packet, not 1 per interface, so that there were actually *2* broadcast packets sent out (one from each NIC) is what I''d consider anomalous... even if it is ''normal'' for Solaris. Whilst it may kind of make sense for "ping broadcast_address", if I do "ping -i interface broadcast_address" I still get duplication of packets! I''m used to ping being a very simple program that I understand which sends out a single packet... If the application can''t control this...??? Darren -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/crossbow-discuss/attachments/20090410/0d52d548/attachment.html>
Nicolas Droux
2009-Apr-10 21:02 UTC
[crossbow-discuss] Broadcast ping problem with etherstub - known problem?
On Apr 10, 2009, at 11:43 AM, James Carlson wrote:> Nicolas Droux writes: >> On Apr 10, 2009, at 10:58 AM, James Carlson wrote: >>> I think this case is a little different, because we wouldn''t suggest >>> putting all the VNICs into an IPMP group (would we?) and we do >>> expect >>> people to create multiple VNICs (don''t we?). >> >> But we wouldn''t expect people to create multiple VNICs on the same >> etherstub (virtual switch) *and* plumb these VNICs in the same zone >> *and* configure them on the same subnet, which is the case here. > > That makes sense, except that Darren says he also saw the anomalous > behavior when putting VNICs into non-global zones as well.Darren still had the two interfaces plumbed in the global zone in that case as well. When the new zone with its VNIC configured on the same subnet, he could see two replies coming from that zone. This is expected since two broadcast ICMP messages will be sent in this case from the sending zone through its two ills and corresponding VNICs, and the non-global zone will reply to both ICMP requests.>> If the multiple VNICs are plumbed from different zones or VMs, then >> this additional outbound IP duplication of broadcast packets will not >> occur. > > That''s not what it seems to say in the CR.If you mean specifically about the example in Entry 2 of the description, I could not duplicate that particular problem. We also confirmed using Darren''s machine that when pinging from the non-global zone, only one reply was received from each VNIC on the etherstub. According to Darren the dups in Entry 2 are probably due to copy-paste from the wrong window or other mistype in bugster. That entry should be fixed to avoid further confusion. Nicolas.> > > > > -- > James Carlson, Solaris Networking <james.d.carlson at sun.com > > > Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 > 2084 > MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 > 1677-- Nicolas Droux - Solaris Kernel Networking - Sun Microsystems, Inc. droux at sun.com - http://blogs.sun.com/droux