I have 3 machines,
T60 thinkpad with e1000 NIC
Dell PowerEdge 2950 with bnx2 NIC
HP DL360 G4P with tg3 NIC
All running xen 3.03 / 2.6.16.29-xen
I have a netbootable image with 2.6.16.29-xenU. This image when it
boots creates a vlan eth0.4002, and then tries to communicate on that
interface to a central server.
On the machines with e1000 an bnx2 NICs, this works fine. On the
machine with tg3 NICs, this does not work.
It''s a little bit hard to tell, but I think whats happening is that the
packets are going out tagged ok. But, on ingress in dom0, the tg3
driver is stripping the vlan tag:
I''m not sure I trust tcpdump 100% but it seems to support that the arp
request is going out tagged, but the response is coming back untagged:
09:41:43.126497 00:16:3e:5a:4e:d8 > Broadcast, ethertype 802.1Q
(0x8100), length 46: vlan 4002, p 0, ethertype ARP, arp who-has
10.21.1.1 tell 10.21.1.99
0x0000: 0fa2 0806 0001 0800 0604 0001 0016 3e5a ..............>Z
0x0010: 4ed8 0a15 0163 0000 0000 0000 0a15 0101 N....c..........
09:41:43.126591 00:0e:aa:04:00:01 > 00:16:3e:5a:4e:d8, ethertype ARP
(0x0806), length 60: arp reply 10.21.1.1 is-at 00:0e:aa:04:00:01
0x0000: 0001 0800 0604 0002 000e aa04 0001 0a15 ................
0x0010: 0101 0016 3e5a 4ed8 0a15 0163 0000 0000 ....>ZN....c....
0x0020: 0000 0000 0000 0000 0000 0000 0000 ..............
I did a quick experiment using xen 3.0.4_1 and got the same results.
Any ideas?
Evan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
> I have 3 machines, > T60 thinkpad with e1000 NIC > Dell PowerEdge 2950 with bnx2 NIC > HP DL360 G4P with tg3 NIC > All running xen 3.03 / 2.6.16.29-xen > > I have a netbootable image with 2.6.16.29-xenU. This image when it > boots creates a vlan eth0.4002, and then tries to communicate on that > interface to a central server.Please can you describe the setup in more detail, including the bridge and NIC setup in dom0. It sounds like you''re trying to pass a VLAN trunk into a guest and then use vconfig inside the guest, but I didn''t think you could do this with the standard linux bridge as it didn''t support VLAN trunking. Ian> On the machines with e1000 an bnx2 NICs, this works fine. On the > machine with tg3 NICs, this does not work. > > It''s a little bit hard to tell, but I think whats happening is thatthe> packets are going out tagged ok. But, on ingress in dom0, the tg3 > driver is stripping the vlan tag: > > I''m not sure I trust tcpdump 100% but it seems to support that the arp > request is going out tagged, but the response is coming back untagged: > > 09:41:43.126497 00:16:3e:5a:4e:d8 > Broadcast, ethertype 802.1Q > (0x8100), length 46: vlan 4002, p 0, ethertype ARP, arp who-has > 10.21.1.1 tell 10.21.1.99 > 0x0000: 0fa2 0806 0001 0800 0604 0001 0016 3e5a..............>Z> 0x0010: 4ed8 0a15 0163 0000 0000 0000 0a15 0101N....c..........> 09:41:43.126591 00:0e:aa:04:00:01 > 00:16:3e:5a:4e:d8, ethertype ARP > (0x0806), length 60: arp reply 10.21.1.1 is-at 00:0e:aa:04:00:01 > 0x0000: 0001 0800 0604 0002 000e aa04 0001 0a15................> 0x0010: 0101 0016 3e5a 4ed8 0a15 0163 0000 0000....>ZN....c....> 0x0020: 0000 0000 0000 0000 0000 0000 0000..............> > I did a quick experiment using xen 3.0.4_1 and got the same results. > > Any ideas? > > Evan > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Pratt wrote:>>I have 3 machines, >> T60 thinkpad with e1000 NIC >> Dell PowerEdge 2950 with bnx2 NIC >> HP DL360 G4P with tg3 NIC >>All running xen 3.03 / 2.6.16.29-xen >> >>I have a netbootable image with 2.6.16.29-xenU. This image when it >>boots creates a vlan eth0.4002, and then tries to communicate on that >>interface to a central server. >> >> > >Please can you describe the setup in more detail, including the bridge >and NIC setup in dom0. It sounds like you''re trying to pass a VLAN trunk >into a guest and then use vconfig inside the guest, but I didn''t think >you could do this with the standard linux bridge as it didn''t support >VLAN trunking. >The network setup in dom0 is the default (I''ve just downloaded the source tarball and then built and installed it. The only change I''ve made is to enable 8021.q support in the xenU kernel). Then you are correct, in domU I use vconfig to create a vlan interface and ifconfig to give it an address and bring it up. On systems whose NICs don''t use the tg3 driver, this seems to work fine. I can tcpdump on the bridge in dom0 and see the tagged traffic coming and going. Here is some info from dom0 on one of the working systems. Successful arp request / reply: [root@localhost ~]# tcpdump -s 0 -e -v -x -i xenbr0 vlan 4002 and arp tcpdump: WARNING: xenbr0: no IPv4 address assigned tcpdump: listening on xenbr0, link-type EN10MB (Ethernet), capture size 65535 bytes 13:41:05.560522 00:0e:aa:04:00:0a > 00:0e:aa:04:00:01, ethertype 802.1Q (0x8100), length 46: vlan 4002, p 0, ethertype ARP, arp who-has 10.21.1.1 tell 10.21.1.9 0x0000: 0fa2 0806 0001 0800 0604 0001 000e aa04 ................ 0x0010: 000a 0a15 0109 0000 0000 0000 0a15 0101 ................ 13:41:05.560596 00:0e:aa:04:00:01 > 00:0e:aa:04:00:0a, ethertype 802.1Q (0x8100), length 64: vlan 4002, p 0, ethertype ARP, arp reply 10.21.1.1 is-at 00:0e:aa:04:00:01 0x0000: 0fa2 0806 0001 0800 0604 0002 000e aa04 ................ 0x0010: 0001 0a15 0101 000e aa04 000a 0a15 0109 ................ 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0030: 0000 Bridge: [root@localhost ~]# brctl show bridge name bridge id STP enabled interfaces xenbr0 8000.feffffffffff no peth0 vif0.0 vif41.0 vif42.0 vif43.0 Interfaces: [root@localhost ~]# ifconfig eth0 Link encap:Ethernet HWaddr 00:18:8B:47:B9:DA inet addr:10.20.36.150 Bcast:10.20.36.255 Mask:255.255.255.0 inet6 addr: fe80::218:8bff:fe47:b9da/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1966533 errors:0 dropped:0 overruns:0 frame:0 TX packets:98750 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1034752722 (986.8 MiB) TX bytes:73493849 (70.0 MiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:208 errors:0 dropped:0 overruns:0 frame:0 TX packets:208 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:20595 (20.1 KiB) TX bytes:20595 (20.1 KiB) peth0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:2387336 errors:0 dropped:0 overruns:0 frame:0 TX packets:1488147 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1704932347 (1.5 GiB) TX bytes:400348056 (381.8 MiB) Interrupt:16 Memory:d6000000-d6011100 vif0.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:98764 errors:0 dropped:0 overruns:0 frame:0 TX packets:1966544 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:73496741 (70.0 MiB) TX bytes:1034753448 (986.8 MiB) vif41.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:74309 errors:0 dropped:0 overruns:0 frame:0 TX packets:234316 errors:0 dropped:82 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:15679399 (14.9 MiB) TX bytes:123190189 (117.4 MiB) vif42.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:74107 errors:0 dropped:0 overruns:0 frame:0 TX packets:171556 errors:0 dropped:68 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:15630126 (14.9 MiB) TX bytes:96787536 (92.3 MiB) vif43.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:74210 errors:0 dropped:0 overruns:0 frame:0 TX packets:179269 errors:0 dropped:44 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:15624781 (14.9 MiB) TX bytes:107122338 (102.1 MiB) xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:46072 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2601825 (2.4 MiB) TX bytes:0 (0.0 b) /Evan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Please can you describe the setup in more detail, including the > bridge > and NIC setup in dom0. It sounds like you''re trying to pass aVLAN> trunk > into a guest and then use vconfig inside the guest, but I didn''t > think > you could do this with the standard linux bridge as it didn''tsupport> VLAN trunking. > > The network setup in dom0 is the default (I''ve just downloaded thesource> tarball and then built and installed it. The only change I''ve made isto> enable 8021.q support in the xenU kernel). Then you are correct, indomU I> use vconfig to create a vlan interface and ifconfig to give it anaddress> and bring it up. On systems whose NICs don''t use the tg3 driver, thisseems> to work fine. I can tcpdump on the bridge in dom0 and see the tagged > traffic coming and going. Here is some info from dom0 on one of the > working systems.I''m still a bit confused as to how the standard linux bridge can work with VLAN trunks. With a ''real'' Ethernet switch you have to enable which VLANs you want to appear on a port. I''m not aware of the Linux bridge having such functionality -- I thought you had to set up a separate bridge for each VLAN then mux/demux them at the physical/virtual NICs. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>I''m still a bit confused as to how the standard linux bridge can work >with VLAN trunks. With a ''real'' Ethernet switch you have to enable which >VLANs you want to appear on a port. I''m not aware of the Linux bridge >having such functionality -- I thought you had to set up a separate >bridge for each VLAN then mux/demux them at the physical/virtual NICs. > > >I''m not expecting it to deliver packets based on vlan membership, I''m just expecting the normal bridging behavior, but leaving the VLAN tags intact. This seems to work fine for the e1000 and bnx2 drivers. I''m just speculating but my guess is that the tg3 driver is stripping the tag before it even hits the bridge. Evan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 18 Jan 2007, Ian Pratt wrote:> I''m still a bit confused as to how the standard linux bridge can work > with VLAN trunks. With a ''real'' Ethernet switch you have to enable which > VLANs you want to appear on a port. I''m not aware of the Linux bridge > having such functionality -- I thought you had to set up a separate > bridge for each VLAN then mux/demux them at the physical/virtual NICs.You''re thinking of VLAN-aware switches.. if you''re using an unmanaged switch that doesn''t have a concept of VLAN''s (ie, the Linux bridging code, IIRC), it''ll just pass the packet on, tags and all. In theory, this should work. I have a feeling that the tg3 driver is indeed stripping the VLAN tags, but am not a coder, so I can''t confirm that.. :( ------------------------------------------------------------------------ | nate carlson | natecars@natecarlson.com | http://www.natecarlson.com | | depriving some poor village of its idiot since 1981 | ------------------------------------------------------------------------ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> On Thu, 18 Jan 2007, Ian Pratt wrote: > > I''m still a bit confused as to how the standard linux bridge canwork> > with VLAN trunks. With a ''real'' Ethernet switch you have to enablewhich> > VLANs you want to appear on a port. I''m not aware of the Linuxbridge> > having such functionality -- I thought you had to set up a separate > > bridge for each VLAN then mux/demux them at the physical/virtualNICs.> > You''re thinking of VLAN-aware switches.. if you''re using an unmanaged > switch that doesn''t have a concept of VLAN''s (ie, the Linux bridgingcode,> IIRC), it''ll just pass the packet on, tags and all.If its broadcasting all VLAN-tagged traffic to all ports that''s disastrous for performance. You''d be much better off setting up a separate bridge for each VLAN. Alternatively, it might be worth looking at The Linux Switching Appliance (LISA) project -- their bridge supports VLAN trunks. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>If its broadcasting all VLAN-tagged traffic to all ports that''s >disastrous for performance. You''d be much better off setting up a >separate bridge for each VLAN. Alternatively, it might be worth looking >at The Linux Switching Appliance (LISA) project -- their bridge supports >VLAN trunks. > >I''d have to do a tcpdump to be sure but empirically I haven''t noticed any performance problems (and these are nfs booted images). All I''m expecting of the bridge code is that it ignore the tag and deliver the packet by mac address as normal. Because all this works with other than the tg3 driver I''m confident the bridge code is doing what I expect. The problem is not with the bridge, the problem is with the tg3 driver. I believe that any network utility on top of the tg3 driver would have the same problem, as the tg3 driver is stripping the vlan tags before it passes them on. Evan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> The problem is not with the bridge, the problem is with the tg3 > driver. I believe that any network utility on top of the tg3 driver > would have the same problem, as the tg3 driver is stripping the vlan > tags before it passes them on.Presumably the tg3 driver works fine if you''re just using vconfig to ''terminate'' particular vlan''s in the kernel? Looking at the driver, you might want to try modifying tg3_rx to always call netif_receive_skb rather than tg3_vlan_rx The other thing you could do is to copy the driver out of 2.6.20 to see if that helps. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christopher G. Stach II
2007-Jan-19 14:17 UTC
Re: [Xen-devel] Problem with VLANs on tg3 NICs
Ian Pratt wrote:>> The problem is not with the bridge, the problem is with the tg3 >> driver. I believe that any network utility on top of the tg3 driver >> would have the same problem, as the tg3 driver is stripping the vlan >> tags before it passes them on. > > Presumably the tg3 driver works fine if you''re just using vconfig to > ''terminate'' particular vlan''s in the kernel? > > Looking at the driver, you might want to try modifying tg3_rx to always > call netif_receive_skb rather than tg3_vlan_rx > > The other thing you could do is to copy the driver out of 2.6.20 to see > if that helps. > > IanThe proprietary driver has tag stripping options, as well. I have never used it, though. I think the tg3 driver will receive without tag stripping if the interface is configured with a vlan (and then pass to the VLAN interface), but always strip otherwise. I haven''t messed with it in a long time, so I could be wrong. -- Christopher G. Stach II _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel