crossbow discuss - Jun 2009 - Duplicate packets when using aggregate datalinks on bge

I opened a bug report earlier today but it doesn''t seem to have been
added to the bugs database.  I''m posting here in case one of the
Crossbow developers might see it and confirm this behavior.

Description
   Duplicate packets are generated whenever an aggregate is introduced
into the network configuration.  We''ve ruled out switch ports and 
physical bge interfaces as problematic.  It only manifests when an 
aggregation datalink exists in the configuration, regardless of whether
it aggregates multiple physical interfaces (e.g. bge0 and bge1) or a
single physical interface (e.g. bge0).

The duplicates are only generated by the OpenSolaris host.  There are no
inbound duplicates.  The duplicates always come from the same physical 
interface.  Tested with ICMP (ping) and TCP (http) connections.

We have numerous servers in production on e1000 interfaces.  This has
not been a problem with the e1000 driver.  This particular server is a
generic x86 system with dual onboard bge.  We are running OpenSolaris
b113.

Frequency
   Always

Regression
   No

Steps to Reproduce
   Create aggr0 consisting of one or more bge interfaces.  Create a
global0 vnic on top of    aggr0.  Plumb global0 and configure it on the
network.  Ping the address for global0.

Expected Result
   No duplicate packets generated by the host.

Actual Result
   Duplicate packets sent by the host.

Workaround
   Do not use aggregate datalinks on bge, or choose different hardware
(e.g. e1000).



-- 
Jason Dixon
OmniTI Computer Consulting, Inc.
jdixon at omniti.com
443.325.1357 x.241

On 06/11/09 20:58, Jason Dixon wrote:
> I opened a bug report earlier today but it doesn''t seem to have
been
> added to the bugs database.  I''m posting here in case one of the
> Crossbow developers might see it and confirm this behavior.
> 
> Description
>    Duplicate packets are generated whenever an aggregate is introduced
> into the network configuration.  We''ve ruled out switch ports and 
> physical bge interfaces as problematic.  It only manifests when an 
> aggregation datalink exists in the configuration, regardless of whether
> it aggregates multiple physical interfaces (e.g. bge0 and bge1) or a
> single physical interface (e.g. bge0).
> 
> The duplicates are only generated by the OpenSolaris host.  There are no
> inbound duplicates.  The duplicates always come from the same physical 
> interface.  Tested with ICMP (ping) and TCP (http) connections.
> 
> We have numerous servers in production on e1000 interfaces.  This has
> not been a problem with the e1000 driver.  This particular server is a
> generic x86 system with dual onboard bge.  We are running OpenSolaris
> b113.
> 
> Frequency
>    Always
> 
> Regression
>    No
> 
> Steps to Reproduce
>    Create aggr0 consisting of one or more bge interfaces.  Create a
> global0 vnic on top of    aggr0.  Plumb global0 and configure it on the
> network.  Ping the address for global0.
can''t speak for the bug status.

so you are saying that if you create a VNIC on top of the aggr you see 
the system send duplicates, and when you plumb and use the aggr directly 
you don''t? and only on bge.

Steffen
> 
> Expected Result
>    No duplicate packets generated by the host.
> 
> Actual Result
>    Duplicate packets sent by the host.
> 
> Workaround
>    Do not use aggregate datalinks on bge, or choose different hardware
> (e.g. e1000).
> 
> 
>

On Thu, Jun 11, 2009 at 10:11:05PM -0400, Steffen Weiberle
wrote:
> On 06/11/09 20:58, Jason Dixon wrote:
>>
>> Description
>>    Duplicate packets are generated whenever an aggregate is introduced
>> into the network configuration.  We''ve ruled out switch ports
and
>> physical bge interfaces as problematic.  It only manifests when an  
>> aggregation datalink exists in the configuration, regardless of whether
>> it aggregates multiple physical interfaces (e.g. bge0 and bge1) or a
>> single physical interface (e.g. bge0).
>>
>> The duplicates are only generated by the OpenSolaris host.  There are
no
>> inbound duplicates.  The duplicates always come from the same physical
>> interface.  Tested with ICMP (ping) and TCP (http) connections.
>>
>> Steps to Reproduce
>>    Create aggr0 consisting of one or more bge interfaces.  Create a
>> global0 vnic on top of    aggr0.  Plumb global0 and configure it on the
>> network.  Ping the address for global0.
>
> can''t speak for the bug status.
>
> so you are saying that if you create a VNIC on top of the aggr you see  
> the system send duplicates, and when you plumb and use the aggr directly  
> you don''t? and only on bge.
Best to give examples:

bge0 in global: no duplicates
bge1 in global: no duplicates
global0 (vnic) on bge0: no duplicates
global0 (vnic) on bge01 no duplicates
vlan2000 (vlan) on bge0: no duplicates
vlan2001 (vlan) on bge1: no duplicates
global0 (vnic) on aggr0 (aggr) on bge0: duplicates
global0 (vnic) on aggr0 (aggr) on bge1: duplicates
global0 (vnic) on aggr0 (aggr) on bge0,bge1: duplicates

I don''t think I attempted to plumb and configure aggr0 directly.  I can
attempt this tomorrow if you feel it''s necessary.

-- 
Jason Dixon
OmniTI Computer Consulting, Inc.
jdixon at omniti.com
443.325.1357 x.241

Jason Dixon wrote:
> On Thu, Jun 11, 2009 at 10:11:05PM -0400, Steffen Weiberle wrote:
>> On 06/11/09 20:58, Jason Dixon wrote:
>>> Description
>>>    Duplicate packets are generated whenever an aggregate is
introduced
>>> into the network configuration.  We''ve ruled out switch
ports and
>>> physical bge interfaces as problematic.  It only manifests when an
>>> aggregation datalink exists in the configuration, regardless of
whether
>>> it aggregates multiple physical interfaces (e.g. bge0 and bge1) or
a
>>> single physical interface (e.g. bge0).
>>>
>>> The duplicates are only generated by the OpenSolaris host.  There
are no
>>> inbound duplicates.  The duplicates always come from the same
physical
>>> interface.  Tested with ICMP (ping) and TCP (http) connections.
>>>
>>> Steps to Reproduce
>>>    Create aggr0 consisting of one or more bge interfaces.  Create a
>>> global0 vnic on top of    aggr0.  Plumb global0 and configure it on
the
>>> network.  Ping the address for global0.
>> can''t speak for the bug status.
>>
>> so you are saying that if you create a VNIC on top of the aggr you see
>> the system send duplicates, and when you plumb and use the aggr
directly
>> you don''t? and only on bge.
> 
> Best to give examples:
> 
> bge0 in global: no duplicates
> bge1 in global: no duplicates
> global0 (vnic) on bge0: no duplicates
> global0 (vnic) on bge01 no duplicates
> vlan2000 (vlan) on bge0: no duplicates
> vlan2001 (vlan) on bge1: no duplicates
> global0 (vnic) on aggr0 (aggr) on bge0: duplicates
> global0 (vnic) on aggr0 (aggr) on bge1: duplicates
> global0 (vnic) on aggr0 (aggr) on bge0,bge1: duplicates
> 
> I don''t think I attempted to plumb and configure aggr0 directly. 
I can
> attempt this tomorrow if you feel it''s necessary.
Yes that would be interesting to know. Also can you confirm that you are 
seeing only duplication of packets sent by the host, not in the inbound 
direction? Also do you see all unicast + multicast packets duplicated?

Nicolas.

-- 
Nicolas Droux - Solaris Kernel Networking - Sun Microsystems, Inc.
nicolas.droux at sun.com - http://blogs.sun.com/droux

On Fri, Jun 12, 2009 at 10:57:26AM -0600, Nicolas Droux
wrote:
> Jason Dixon wrote:
>> On Thu, Jun 11, 2009 at 10:11:05PM -0400, Steffen Weiberle wrote:
>>> On 06/11/09 20:58, Jason Dixon wrote:
>>>> Description
>>>>    Duplicate packets are generated whenever an aggregate is
introduced
>>>> into the network configuration.  We''ve ruled out
switch ports and
>>>> physical bge interfaces as problematic.  It only manifests when
an
>>>> aggregation datalink exists in the configuration, regardless of
>>>> whether
>>>> it aggregates multiple physical interfaces (e.g. bge0 and bge1)
or a
>>>> single physical interface (e.g. bge0).
>>>>
>>>> The duplicates are only generated by the OpenSolaris host. 
There are no
>>>> inbound duplicates.  The duplicates always come from the same 
>>>> physical  interface.  Tested with ICMP (ping) and TCP (http) 
>>>> connections.
>>>>
>>>> Steps to Reproduce
>>>>    Create aggr0 consisting of one or more bge interfaces. 
Create a
>>>> global0 vnic on top of    aggr0.  Plumb global0 and configure
it on the
>>>> network.  Ping the address for global0.
>>> can''t speak for the bug status.
>>>
>>> so you are saying that if you create a VNIC on top of the aggr you 
>>> see  the system send duplicates, and when you plumb and use the
aggr
>>> directly  you don''t? and only on bge.
>>
>> Best to give examples:
>>
>> bge0 in global: no duplicates
>> bge1 in global: no duplicates
>> global0 (vnic) on bge0: no duplicates
>> global0 (vnic) on bge01 no duplicates
>> vlan2000 (vlan) on bge0: no duplicates
>> vlan2001 (vlan) on bge1: no duplicates
>> global0 (vnic) on aggr0 (aggr) on bge0: duplicates
>> global0 (vnic) on aggr0 (aggr) on bge1: duplicates
>> global0 (vnic) on aggr0 (aggr) on bge0,bge1: duplicates
>>
>> I don''t think I attempted to plumb and configure aggr0
directly.  I can
>> attempt this tomorrow if you feel it''s necessary.
>
> Yes that would be interesting to know. Also can you confirm that you are  
> seeing only duplication of packets sent by the host, not in the inbound  
> direction? Also do you see all unicast + multicast packets duplicated?
Ok, I will test with aggr0 directly.  Yes, this is only for outbound
packets.  So far I''ve only tested unicast.  Any suggestions on
generating
the multicast packets?

Thanks,

-- 
Jason Dixon
OmniTI Computer Consulting, Inc.
jdixon at omniti.com
443.325.1357 x.241

Jason Dixon wrote:
> On Fri, Jun 12, 2009 at 10:57:26AM -0600, Nicolas Droux wrote:
>> Jason Dixon wrote:
>>> On Thu, Jun 11, 2009 at 10:11:05PM -0400, Steffen Weiberle wrote:
>>>> On 06/11/09 20:58, Jason Dixon wrote:
>>>>> Description
>>>>>    Duplicate packets are generated whenever an aggregate is
introduced
>>>>> into the network configuration.  We''ve ruled out
switch ports and
>>>>> physical bge interfaces as problematic.  It only manifests
when an
>>>>> aggregation datalink exists in the configuration,
regardless of
>>>>> whether
>>>>> it aggregates multiple physical interfaces (e.g. bge0 and
bge1) or a
>>>>> single physical interface (e.g. bge0).
>>>>>
>>>>> The duplicates are only generated by the OpenSolaris host. 
There are no
>>>>> inbound duplicates.  The duplicates always come from the
same
>>>>> physical  interface.  Tested with ICMP (ping) and TCP
(http)
>>>>> connections.
>>>>>
>>>>> Steps to Reproduce
>>>>>    Create aggr0 consisting of one or more bge interfaces. 
Create a
>>>>> global0 vnic on top of    aggr0.  Plumb global0 and
configure it on the
>>>>> network.  Ping the address for global0.
>>>> can''t speak for the bug status.
>>>>
>>>> so you are saying that if you create a VNIC on top of the aggr
you
>>>> see  the system send duplicates, and when you plumb and use the
aggr
>>>> directly  you don''t? and only on bge.
>>> Best to give examples:
>>>
>>> bge0 in global: no duplicates
>>> bge1 in global: no duplicates
>>> global0 (vnic) on bge0: no duplicates
>>> global0 (vnic) on bge01 no duplicates
>>> vlan2000 (vlan) on bge0: no duplicates
>>> vlan2001 (vlan) on bge1: no duplicates
>>> global0 (vnic) on aggr0 (aggr) on bge0: duplicates
>>> global0 (vnic) on aggr0 (aggr) on bge1: duplicates
>>> global0 (vnic) on aggr0 (aggr) on bge0,bge1: duplicates
>>>
>>> I don''t think I attempted to plumb and configure aggr0
directly.  I can
>>> attempt this tomorrow if you feel it''s necessary.
>> Yes that would be interesting to know. Also can you confirm that you
are
>> seeing only duplication of packets sent by the host, not in the inbound
>> direction? Also do you see all unicast + multicast packets duplicated?
> 
> Ok, I will test with aggr0 directly.  Yes, this is only for outbound
> packets.  So far I''ve only tested unicast.  Any suggestions on
generating
> the multicast packets?
You can do something as simple as using ping(1M) to send ICMP packets to 
  the broadcast address for your interface.

Nicolas.
> 
> Thanks,
> 
-- 
Nicolas Droux - Solaris Kernel Networking - Sun Microsystems, Inc.
nicolas.droux at sun.com - http://blogs.sun.com/droux

On 06/12/09 13:46, Jason Dixon wrote:
> On Fri, Jun 12, 2009 at 10:57:26AM -0600, Nicolas Droux wrote:
>> Jason Dixon wrote:
>>> On Thu, Jun 11, 2009 at 10:11:05PM -0400, Steffen Weiberle wrote:
>>>> On 06/11/09 20:58, Jason Dixon wrote:
>>>>> Description
>>>>>    Duplicate packets are generated whenever an aggregate is
introduced
>>>>> into the network configuration.  We''ve ruled out
switch ports and
>>>>> physical bge interfaces as problematic.  It only manifests
when an
>>>>> aggregation datalink exists in the configuration,
regardless of
>>>>> whether
>>>>> it aggregates multiple physical interfaces (e.g. bge0 and
bge1) or a
>>>>> single physical interface (e.g. bge0).
>>>>>
>>>>> The duplicates are only generated by the OpenSolaris host. 
There are no
>>>>> inbound duplicates.  The duplicates always come from the
same
>>>>> physical  interface.  Tested with ICMP (ping) and TCP
(http)
>>>>> connections.
>>>>>
>>>>> Steps to Reproduce
>>>>>    Create aggr0 consisting of one or more bge interfaces. 
Create a
>>>>> global0 vnic on top of    aggr0.  Plumb global0 and
configure it on the
>>>>> network.  Ping the address for global0.
>>>> can''t speak for the bug status.
>>>>
>>>> so you are saying that if you create a VNIC on top of the aggr
you
>>>> see  the system send duplicates, and when you plumb and use the
aggr
>>>> directly  you don''t? and only on bge.
>>> Best to give examples:
>>>
>>> bge0 in global: no duplicates
>>> bge1 in global: no duplicates
>>> global0 (vnic) on bge0: no duplicates
>>> global0 (vnic) on bge01 no duplicates
>>> vlan2000 (vlan) on bge0: no duplicates
>>> vlan2001 (vlan) on bge1: no duplicates
>>> global0 (vnic) on aggr0 (aggr) on bge0: duplicates
>>> global0 (vnic) on aggr0 (aggr) on bge1: duplicates
>>> global0 (vnic) on aggr0 (aggr) on bge0,bge1: duplicates
I tried duplicating this on a v210, using bge1 and bge2, and was not 
able to do so with 113

pinebarren# dladm create-aggr -l bge1 -l bge2 aggr1
pinebarren# dladm create-vnic -l aggr1 vnic1
pinebarren# ifconfig vnic1 plumb 10.1.14.141/26 up
pinebarren# ping 10.1.14.130
10.1.14.130 is alive


marlton# snoop -d nge0 host 10.1.14.141
Using device /dev/nge0 (promiscuous mode)
mtllab141 -> mtllab130  ICMP Echo request (ID: 3843 Sequence number: 0)
mtllab130 -> mtllab141  ICMP Echo reply (ID: 3843 Sequence number: 0)


Steffen

Note: I used bge1 and bge2 as bge0 is my admin interface an connected to 
a different switch and subnet.

pinebarren# head -1 /etc/release
                  Solaris Express Community Edition snv_113 SPARC
pinebarren# dladm show-phys
LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
bge2         Ethernet             up         1000   full      bge2
bge0         Ethernet             up         1000   full      bge0
bge1         Ethernet             up         1000   full      bge1
bge3         Ethernet             unknown    0      unknown   bge3
pinebarren# dladm show-link
LINK        CLASS    MTU    STATE    OVER
bge2        phys     1500   up       --
bge0        phys     1500   up       --
bge1        phys     1500   up       --
bge3        phys     1500   unknown  --
aggr1       aggr     1500   up       bge1 bge2
vnic1       vnic     1500   up       aggr1
pinebarren# dladm show-aggr
LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER 
FLAGS
aggr1           L4       auto                 off           short 
-----
pinebarren# dladm show-vnic
LINK         OVER         SPEED  MACADDRESS           MACADDRTYPE 
   VID
vnic1        aggr1        1000   2:8:20:c:63:68       random              0

>>>
>>> I don''t think I attempted to plumb and configure aggr0
directly.  I can
>>> attempt this tomorrow if you feel it''s necessary.
>> Yes that would be interesting to know. Also can you confirm that you
are
>> seeing only duplication of packets sent by the host, not in the inbound
>> direction? Also do you see all unicast + multicast packets duplicated?
> 
> Ok, I will test with aggr0 directly.  Yes, this is only for outbound
> packets.  So far I''ve only tested unicast.  Any suggestions on
generating
> the multicast packets?
> 
> Thanks,
>