Xen devel - Jan 2007 - Problem with VLANs on tg3 NICs

I have 3 machines,
    T60 thinkpad with e1000 NIC
    Dell PowerEdge 2950 with bnx2 NIC
    HP DL360 G4P with tg3 NIC
All running xen 3.03 / 2.6.16.29-xen

I have a netbootable image with 2.6.16.29-xenU.  This image when it 
boots creates a vlan eth0.4002, and then tries to communicate on that 
interface to a central server.

On the machines with e1000 an bnx2 NICs, this works fine.  On the 
machine with tg3 NICs, this does not work.

It''s a little bit hard to tell, but I think whats happening is that the
packets are going out tagged ok.  But, on ingress in dom0, the tg3 
driver is stripping the vlan tag:

I''m not sure I trust tcpdump 100% but it seems to support that the arp 
request is going out tagged, but the response is coming back untagged:

09:41:43.126497 00:16:3e:5a:4e:d8 > Broadcast, ethertype 802.1Q 
(0x8100), length 46: vlan 4002, p 0, ethertype ARP, arp who-has 
10.21.1.1 tell 10.21.1.99
        0x0000:  0fa2 0806 0001 0800 0604 0001 0016 3e5a  ..............>Z
        0x0010:  4ed8 0a15 0163 0000 0000 0000 0a15 0101  N....c..........
09:41:43.126591 00:0e:aa:04:00:01 > 00:16:3e:5a:4e:d8, ethertype ARP 
(0x0806), length 60: arp reply 10.21.1.1 is-at 00:0e:aa:04:00:01
        0x0000:  0001 0800 0604 0002 000e aa04 0001 0a15  ................
        0x0010:  0101 0016 3e5a 4ed8 0a15 0163 0000 0000  ....>ZN....c....
        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............

I did a quick experiment using xen 3.0.4_1 and got the same results.

Any ideas?

Evan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> I have 3 machines,
>     T60 thinkpad with e1000 NIC
>     Dell PowerEdge 2950 with bnx2 NIC
>     HP DL360 G4P with tg3 NIC
> All running xen 3.03 / 2.6.16.29-xen
> 
> I have a netbootable image with 2.6.16.29-xenU.  This image when it
> boots creates a vlan eth0.4002, and then tries to communicate on that
> interface to a central server.
Please can you describe the setup in more detail, including the bridge
and NIC setup in dom0. It sounds like you''re trying to pass a VLAN
trunk
into a guest and then use vconfig inside the guest, but I didn''t think
you could do this with the standard linux bridge as it didn''t support
VLAN trunking. 

Ian

 
> On the machines with e1000 an bnx2 NICs, this works fine.  On the
> machine with tg3 NICs, this does not work.
> 
> It''s a little bit hard to tell, but I think whats happening is
that
the
> packets are going out tagged ok.  But, on ingress in dom0, the tg3
> driver is stripping the vlan tag:
> 
> I''m not sure I trust tcpdump 100% but it seems to support that the
arp
> request is going out tagged, but the response is coming back untagged:
> 
> 09:41:43.126497 00:16:3e:5a:4e:d8 > Broadcast, ethertype 802.1Q
> (0x8100), length 46: vlan 4002, p 0, ethertype ARP, arp who-has
> 10.21.1.1 tell 10.21.1.99
>         0x0000:  0fa2 0806 0001 0800 0604 0001 0016 3e5a
..............>Z
>         0x0010:  4ed8 0a15 0163 0000 0000 0000 0a15 0101
N....c..........
> 09:41:43.126591 00:0e:aa:04:00:01 > 00:16:3e:5a:4e:d8, ethertype ARP
> (0x0806), length 60: arp reply 10.21.1.1 is-at 00:0e:aa:04:00:01
>         0x0000:  0001 0800 0604 0002 000e aa04 0001 0a15
................
>         0x0010:  0101 0016 3e5a 4ed8 0a15 0163 0000 0000
....>ZN....c....
>         0x0020:  0000 0000 0000 0000 0000 0000 0000
..............
> 
> I did a quick experiment using xen 3.0.4_1 and got the same results.
> 
> Any ideas?
> 
> Evan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ian Pratt wrote:
>>I have 3 machines,
>>    T60 thinkpad with e1000 NIC
>>    Dell PowerEdge 2950 with bnx2 NIC
>>    HP DL360 G4P with tg3 NIC
>>All running xen 3.03 / 2.6.16.29-xen
>>
>>I have a netbootable image with 2.6.16.29-xenU.  This image when it
>>boots creates a vlan eth0.4002, and then tries to communicate on that
>>interface to a central server.
>>    
>>
>
>Please can you describe the setup in more detail, including the bridge
>and NIC setup in dom0. It sounds like you''re trying to pass a VLAN
trunk
>into a guest and then use vconfig inside the guest, but I didn''t
think
>you could do this with the standard linux bridge as it didn''t
support
>VLAN trunking. 
>
The network setup in dom0 is the default (I''ve just downloaded the 
source tarball and then built and installed it.  The only change I''ve 
made is to enable 8021.q support in the xenU kernel).  Then you are 
correct, in domU I use vconfig to create a vlan interface and ifconfig 
to give it an address and bring it up. On systems whose NICs don''t use 
the tg3 driver, this seems to work fine.  I can tcpdump on the bridge in 
dom0 and see the tagged traffic coming and going.  Here is some info 
from dom0 on one of the working systems.

Successful arp request / reply:

[root@localhost ~]# tcpdump -s 0 -e -v -x -i xenbr0 vlan 4002 and  arp
tcpdump: WARNING: xenbr0: no IPv4 address assigned
tcpdump: listening on xenbr0, link-type EN10MB (Ethernet), capture size 
65535 bytes
13:41:05.560522 00:0e:aa:04:00:0a > 00:0e:aa:04:00:01, ethertype 802.1Q 
(0x8100), length 46: vlan 4002, p 0, ethertype ARP, arp who-has 
10.21.1.1 tell 10.21.1.9
        0x0000:  0fa2 0806 0001 0800 0604 0001 000e aa04  ................
        0x0010:  000a 0a15 0109 0000 0000 0000 0a15 0101  ................
13:41:05.560596 00:0e:aa:04:00:01 > 00:0e:aa:04:00:0a, ethertype 802.1Q 
(0x8100), length 64: vlan 4002, p 0, ethertype ARP, arp reply 10.21.1.1 
is-at 00:0e:aa:04:00:01
        0x0000:  0fa2 0806 0001 0800 0604 0002 000e aa04  ................
        0x0010:  0001 0a15 0101 000e aa04 000a 0a15 0109  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000

Bridge:

[root@localhost ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
xenbr0          8000.feffffffffff       no              peth0
                                                        vif0.0
                                                        vif41.0
                                                        vif42.0
                                                        vif43.0

Interfaces:

[root@localhost ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:18:8B:47:B9:DA
          inet addr:10.20.36.150  Bcast:10.20.36.255  Mask:255.255.255.0
          inet6 addr: fe80::218:8bff:fe47:b9da/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1966533 errors:0 dropped:0 overruns:0 frame:0
          TX packets:98750 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1034752722 (986.8 MiB)  TX bytes:73493849 (70.0 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:208 errors:0 dropped:0 overruns:0 frame:0
          TX packets:208 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:20595 (20.1 KiB)  TX bytes:20595 (20.1 KiB)

peth0     Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:2387336 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1488147 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1704932347 (1.5 GiB)  TX bytes:400348056 (381.8 MiB)
          Interrupt:16 Memory:d6000000-d6011100

vif0.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:98764 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1966544 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:73496741 (70.0 MiB)  TX bytes:1034753448 (986.8 MiB)

vif41.0   Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:74309 errors:0 dropped:0 overruns:0 frame:0
          TX packets:234316 errors:0 dropped:82 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:15679399 (14.9 MiB)  TX bytes:123190189 (117.4 MiB)

vif42.0   Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:74107 errors:0 dropped:0 overruns:0 frame:0
          TX packets:171556 errors:0 dropped:68 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:15630126 (14.9 MiB)  TX bytes:96787536 (92.3 MiB)

vif43.0   Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:74210 errors:0 dropped:0 overruns:0 frame:0
          TX packets:179269 errors:0 dropped:44 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:15624781 (14.9 MiB)  TX bytes:107122338 (102.1 MiB)

xenbr0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:46072 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2601825 (2.4 MiB)  TX bytes:0 (0.0 b)

/Evan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> 	Please can you describe the setup in more detail, including the
> bridge
> 	and NIC setup in dom0. It sounds like you''re trying to pass a
VLAN
> trunk
> 	into a guest and then use vconfig inside the guest, but I didn''t
> think
> 	you could do this with the standard linux bridge as it didn''t
support
> 	VLAN trunking.
> 
> The network setup in dom0 is the default (I''ve just downloaded the
source
> tarball and then built and installed it.  The only change I''ve
made is
to
> enable 8021.q support in the xenU kernel).  Then you are correct, in
domU I
> use vconfig to create a vlan interface and ifconfig to give it an
address
> and bring it up. On systems whose NICs don''t use the tg3 driver,
this
seems
> to work fine.  I can tcpdump on the bridge in dom0 and see the tagged
> traffic coming and going.  Here is some info from dom0 on one of the
> working systems.
I''m still a bit confused as to how the standard linux bridge can work
with VLAN trunks. With a ''real'' Ethernet switch you have to
enable which
VLANs you want to appear on a port. I''m not aware of the Linux bridge
having such functionality -- I thought you had to set up a separate
bridge for each VLAN then mux/demux them at the physical/virtual NICs.

Ian


 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>I''m still a bit confused as to how the standard linux bridge can
work
>with VLAN trunks. With a ''real'' Ethernet switch you have
to enable which
>VLANs you want to appear on a port. I''m not aware of the Linux
bridge
>having such functionality -- I thought you had to set up a separate
>bridge for each VLAN then mux/demux them at the physical/virtual NICs.
>
>  
>
 I''m not expecting it to deliver packets based on vlan membership,
I''m just expecting the normal bridging behavior, but leaving the VLAN
tags intact.  This seems to work fine for the e1000 and bnx2 drivers. 
I''m just speculating but my guess is that the tg3 driver is stripping
the tag before it even hits the bridge.

Evan




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, 18 Jan 2007, Ian Pratt wrote:
> I''m still a bit confused as to how the standard linux bridge can
work
> with VLAN trunks. With a ''real'' Ethernet switch you have
to enable which
> VLANs you want to appear on a port. I''m not aware of the Linux
bridge
> having such functionality -- I thought you had to set up a separate 
> bridge for each VLAN then mux/demux them at the physical/virtual NICs.
You''re thinking of VLAN-aware switches.. if you''re using an
unmanaged
switch that doesn''t have a concept of VLAN''s (ie, the Linux
bridging code,
IIRC), it''ll just pass the packet on, tags and all.

In theory, this should work. I have a feeling that the tg3 driver is 
indeed stripping the VLAN tags, but am not a coder, so I can''t confirm 
that.. :(

------------------------------------------------------------------------
| nate carlson | natecars@natecarlson.com | http://www.natecarlson.com |
|       depriving some poor village of its idiot since 1981            |
------------------------------------------------------------------------

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> On Thu, 18 Jan 2007, Ian Pratt wrote:
> > I''m still a bit confused as to how the standard linux bridge
can
work
> > with VLAN trunks. With a ''real'' Ethernet switch you
have to enable
which
> > VLANs you want to appear on a port. I''m not aware of the
Linux
bridge
> > having such functionality -- I thought you had to set up a separate
> > bridge for each VLAN then mux/demux them at the physical/virtual
NICs.
> 
> You''re thinking of VLAN-aware switches.. if you''re using
an unmanaged
> switch that doesn''t have a concept of VLAN''s (ie, the
Linux bridging
code,
> IIRC), it''ll just pass the packet on, tags and all.
If its broadcasting all VLAN-tagged traffic to all ports that''s
disastrous for performance. You''d be much better off setting up a
separate bridge for each VLAN. Alternatively, it might be worth looking
at The Linux Switching Appliance (LISA) project -- their bridge supports
VLAN trunks.

Ian 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>If its broadcasting all VLAN-tagged traffic to all ports that''s
>disastrous for performance. You''d be much better off setting up a
>separate bridge for each VLAN. Alternatively, it might be worth looking
>at The Linux Switching Appliance (LISA) project -- their bridge supports
>VLAN trunks.
>  
>
I''d have to do a tcpdump to be sure but empirically I haven''t
noticed
any performance problems (and these are nfs booted images).

All I''m expecting of the bridge code is that it ignore the tag and 
deliver the packet by mac address as normal. Because all this works with 
other than the tg3 driver I''m confident the bridge code is doing what I
expect.  The problem is not with the bridge, the problem is with the tg3 
driver. I believe that any network utility on top of the tg3 driver 
would have the same problem, as the tg3 driver is stripping the vlan 
tags before it passes them on.

Evan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> The problem is not with the bridge, the problem is with the tg3
> driver. I believe that any network utility on top of the tg3 driver
> would have the same problem, as the tg3 driver is stripping the vlan
> tags before it passes them on.
Presumably the tg3 driver works fine if you''re just using vconfig to
''terminate'' particular vlan''s in the kernel? 

Looking at the driver, you might want to try modifying tg3_rx to always
call netif_receive_skb rather than tg3_vlan_rx

The other thing you could do is to copy the driver out of 2.6.20 to see
if that helps.

Ian




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ian Pratt wrote:
>> The problem is not with the bridge, the problem is with the tg3
>> driver. I believe that any network utility on top of the tg3 driver
>> would have the same problem, as the tg3 driver is stripping the vlan
>> tags before it passes them on.
> 
> Presumably the tg3 driver works fine if you''re just using vconfig
to
> ''terminate'' particular vlan''s in the kernel? 
> 
> Looking at the driver, you might want to try modifying tg3_rx to always
> call netif_receive_skb rather than tg3_vlan_rx
> 
> The other thing you could do is to copy the driver out of 2.6.20 to see
> if that helps.
> 
> Ian
The proprietary driver has tag stripping options, as well.  I have never
used it, though.

I think the tg3 driver will receive without tag stripping if the
interface is configured with a vlan (and then pass to the VLAN
interface), but always strip otherwise.  I haven''t messed with it in a
long time, so I could be wrong.

-- 
Christopher G. Stach II

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel