crossbow discuss - Jun 2010 - Etherstub connected to a bridge not working

Hi All,

I''m attempting to set up an OpenSolaris xVM server (on snv_130), with
two physical network cards connected to two separate physical switches for
redundancy.

In VMWare and Citrix XenServer one would create a virtual switch fabric between
the two NICs to provide failover in the event of a switch/cable failure. In
OpenSolaris the solution appears to be to use a bridge attached to an etherstub.

So I''ve tried giving this a go, but unfortunately traffic never flows
between the etherstub and the bridge. Here is the configuration:

# dladm show-link
LINK        CLASS     MTU    STATE    BRIDGE     OVER
bnx0        phys      1500   up       xenbridge  --
bnx1        phys      1500   up       xenbridge  --
xenswitch0  etherstub 1500   up       xenbridge  --
xenbridge0  bridge    1500   up       --         bnx0 bnx1 xenswitch0
xvm1_0      vnic      1500   up       --         xenswitch0
xvm2_0      vnic      1500   up       --         xenswitch0

# dladm show-bridge
BRIDGE      PROTECT ADDRESS            PRIORITY DESROOT
xenbridge   stp     32768/0:24:e8:76:53:2f 32768 32768/0:1f:26:ef:c:0

# dladm show-vnic
LINK         OVER         SPEED  MACADDRESS        MACADDRTYPE         VID
xvm1_0       xenswitch0   0      0:16:3e:8b:42:ff  fixed               1257
xvm2_0       xenswitch0   0      0:16:3e:5d:9f:74  fixed               1257

# dladm show-phys
LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
bnx0         Ethernet             up         1000   full      bnx0
bnx1         Ethernet             up         1000   full      bnx1

# dladm show-bridge -l xenbridge
LINK        STATE       UPTIME  DESROOT
bnx0        forwarding  1428    32768/0:1f:26:ef:c:0
bnx1        discarding  1428    32768/0:1f:26:ef:c:0
xenswitch0  forwarding  1428    32768/0:1f:26:ef:c:0

I have two xVM guests, which can talk to each other fine over the etherstub,
however tcpdump in the (Linux) guests doesn''t show any traffic from the
bridge, and they can''t see out.

If I snoop the bridge on the host, it shows all the traffic from bnx0, so the
bridge seems to work. However if I snoop the etherstub, it again only shows
traffic between the xVM guests, and nothing from the bridge.

So traffic just isn''t making it between the bridge and the etherstub.

I haven''t been able to find that many examples online however this
guide seems to suggest I''ve configured things correctly:

http://southbrain.com/south/2009/12/dladm-virtual-networks-sun-xvm.html

Does anyone know if this is a known problem/defect? Or am I "doing it
wrong"? Any help/advice would be greatly appreciated, as at present
bridging seems to be broken. I''ve tried this on snv_134 as well and it
didn''t fix the issue.

Kind Regards,

Alasdair
-- 
This message posted from opensolaris.org

(Bcc: rbridges-dev]

On 06/28/10 08:28 AM, Alasdair Lumsden wrote:
[..]
> If I snoop the bridge on the host, it shows all the traffic from bnx0, so
the bridge seems to work. However if I snoop the etherstub, it again only shows
traffic between the xVM guests, and nothing from the bridge.
>
> So traffic just isn''t making it between the bridge and the
etherstub.
>    
This bug is CR 6959566, as a workaround you could try creating VNICs over
the physical links that are part of the bridge (w/o the Etherstub) and 
assigning
them to xVM guests.

Rishi

> This bug is CR 6959566, as a workaround you could try creating VNICs
> over the physical links that are part of the bridge (w/o the Etherstub)
and
> assigning them to xVM guests.
> 
> Rishi
Hi Rishi,

Many thanks for responding, and thanks for the bug ID! Very useful indeed
:)

I''ve tried doing as you suggested, removing the etherstub and attaching
the vnic directly to the bridged physical interface:

# dladm show-link
LINK        CLASS     MTU    STATE    BRIDGE     OVER
bnx0        phys      1500   up       xenbridge  --
bnx1        phys      1500   up       xenbridge  --
xenbridge0  bridge    1500   up       --         bnx0 bnx1
xvm1_0      vnic      1500   up       --         bnx0

# cat vm.xml
*snip*
    <interface type=''bridge''>
      <mac address=''00:16:3e:8b:42:ff''/>
      <source bridge=''bnx0''/>
      <script path=''/usr/lib/xen/scripts/vif-vnic''/>
      <target dev=''vif-1.0''/>
      <vlan id=''1257'' />
    </interface>

Unfortunately no traffic seems to flow from the VM to the outside world,
although tcpdump in the VM does show some external traffic (such as ARP
requests).

If I remove bnx0 from the bridge with "dladm add-bridge -l bnx0
xenbridge", ping packets start flowing again. As soon as I add bnx0 back
to the bridge, ping packets stop.

Any ideas?

Cheers,

Alasdair

On 06/28/10 10:54 AM, Alasdair Lumsden wrote:
[..]
> I''ve tried doing as you suggested, removing the etherstub and
attaching
> the vnic directly to the bridged physical interface:
>
> # dladm show-link
> LINK        CLASS     MTU    STATE    BRIDGE     OVER
> bnx0        phys      1500   up       xenbridge  --
> bnx1        phys      1500   up       xenbridge  --
> xenbridge0  bridge    1500   up       --         bnx0 bnx1
> xvm1_0      vnic      1500   up       --         bnx0
>
> # cat vm.xml
> *snip*
>      <interface type=''bridge''>
>        <mac address=''00:16:3e:8b:42:ff''/>
>        <source bridge=''bnx0''/>
>        <script
path=''/usr/lib/xen/scripts/vif-vnic''/>
>        <target dev=''vif-1.0''/>
>        <vlan id=''1257'' />
>      </interface>
>
> Unfortunately no traffic seems to flow from the VM to the outside world,
> although tcpdump in the VM does show some external traffic (such as ARP
> requests).
>
> If I remove bnx0 from the bridge with "dladm add-bridge -l bnx0
> xenbridge", ping packets start flowing again. As soon as I add bnx0
back
> to the bridge, ping packets stop.
>    
What is the output of ''dladm show-bridge -l xenbridge''
when you notice ping not working? Can you snoop on the
link in forwarding state (if it is not bnx0) to see if packets
are flowing from the vm? Could you also test w/o specifying
the vlan id for the vnic used by the vm?

Rishi

> What is the output of ''dladm show-bridge -l xenbridge''
> when you notice ping not working? Can you snoop on the
> link in forwarding state (if it is not bnx0) to see if packets
> are flowing from the vm? Could you also test w/o specifying
> the vlan id for the vnic used by the vm?
Hi Rishi,

Thanks again for getting back to me.

I tried simplifying the network setup so there were no vlans involved, and
the bridge/vnics work. However this doesn''t solve the particular
failure
mode I''m trying to account for (physical switch failure).

As an example, if I have bnx0 into switch0 and bnx1 into switch1, and
create a bridge across bnx0 and bnx1, and a vnic on bnx0, if switch0 dies,
then the vnic dies too.

This doesn''t provide us with any form of redundancy, which was the
reason
for using bridging in the first place. 

Do you have any ideas/potential workarounds? Or will we have to wait for
this feature to be completed?

Thanks again,

Alasdair

On 06/28/10 01:03 PM, Alasdair Lumsden wrote:
[..]
> I tried simplifying the network setup so there were no vlans involved, and
> the bridge/vnics work. However this doesn''t solve the particular
failure
> mode I''m trying to account for (physical switch failure).
>
> As an example, if I have bnx0 into switch0 and bnx1 into switch1, and
> create a bridge across bnx0 and bnx1, and a vnic on bnx0, if switch0 dies,
> then the vnic dies too.
>    
When switch0 dies, the link bnx0 state should be down causing
the bridge to generate topology change notification packets
and STP re-calculation. You should see packets being forwarded
on bnx1 instead. Do you see forwarding entries being cleared
and packets sent on bnx1 on failure of switch0? You can use
dladm show-bridge -f bridge to see the forwarding entries. What
is the link state when switch0 fails in dladm show-link
output?
> This doesn''t provide us with any form of redundancy, which was the
reason
> for using bridging in the first place.
>
> Do you have any ideas/potential workarounds? Or will we have to wait for
> this feature to be completed?
>
>    
You could post to networking-discuss to discuss other solutions.
For IP failover you could check Solaris IPMP and see if it
works for your case.

Rishi