Hi, I am trying to make the VF interface of Intel 82576 dual port card, as default outgoing interface on dom0. Data traffic from VMs (HVM) should go through VF interface on dom0. Issue: Tx traffic (arp request) from VMs goes out through VF on dom0, but Rx traffic (arp reply) is not reaching to VMs. Setup: HP dc7800 VTd machine with Xen 3.4 and 2.6.18-igb as dom0 FC11 2.6.30 kernel as domU-1. Ubuntu 2.6.30 kernel as domU-2. I loaded igbvf-1.0.7 driver(from http://sourceforge.net/projects/e1000/files/)<http://sourceforge.net/projects/e1000/files/>on 2.6.18-igb dom0. Minor change was required, in netdev.c file. Replaced ''alloc_etherdev(X)'' with alloc_etherdev_mq(X , 1) Steps: On dom0 #rmmod igb #modprobe igb max_vfs=1 #modprobe igbvf Above steps create eth1, eth2 as PF and eth3, eth4 as VF interfaces. I added following line to default /etc/xen/xend-config.sxp (network-script ''network-bridge netdev=eth3'') Then I started xend #xend start eth3 bridge gets created with peth3 as one interface in the bridge. In the hmv config files, changed the bridge name to xenbr3. vif = [ ''type=ioemu, bridge=xenbr3'' ] Created two VMs and assigned IP address in same network domain as of VF and PF. Four interfaces tap1.0, vif1.0, tap2.0, vif2.0 get added to the eth3 bridge on dom0. Removed iptable rules ''iptables -F'' from dom0, domU-1, domU-2. Observation: Ping from domU-1 to VF and PF ip address works. Ping between domU-1 and domU-2 works. But, ping from domU to another machine on same network on switch doesn''t work. The arp broadcast request goes out through VF interface. But the arp reply doesn''t reach VF interface, they get routed to PF interface. Why is it so? PF interface(eth1) is not part of the eth3 bridge. If I add eth1 to eth3-bridge, then ping from VMs to other machines on switch works. Is there something I am missing? How to make above experiment to work? Regards, -Satish _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Nov 06, 2009 at 06:35:40AM +0530, Satish Chowdhury wrote:> Hi, > > I am trying to make the VF interface of Intel 82576 dual port card, as > default outgoing interface on dom0. Data traffic from VMs (HVM) should go > through VF interface on dom0.[snip]> Ping from domU-1 to VF and PF ip address works. > Ping between domU-1 and domU-2 works. > But, ping from domU to another machine on same network on switch doesn''t > work. > > The arp broadcast request goes out through VF interface. But the arp reply > doesn''t reach VF interface, they get routed to PF interface. Why is it so? > > PF interface(eth1) is not part of the eth3 bridge. If I add eth1 to > eth3-bridge, then ping from VMs to other machines on switch works. > > > Is there something I am missing? How to make above experiment to work?Hi Staish, I suspect that there is some kind of driver issue. Unfortunately the building that the dc7800 that I use is undergoing power-maintenance (= power-outage) today, so I can''t investigate right now. Out of interest, why are you trying to do this? The PF needs to be present and have a link up in dom0 in order for the VF to exist and have a link in dom0, so why not just use the PF? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sun, Nov 08, 2009 at 09:09:43AM +1100, Simon Horman wrote:> On Fri, Nov 06, 2009 at 06:35:40AM +0530, Satish Chowdhury wrote: > > Hi, > > > > I am trying to make the VF interface of Intel 82576 dual port card, as > > default outgoing interface on dom0. Data traffic from VMs (HVM) should go > > through VF interface on dom0. > > [snip] > > > Ping from domU-1 to VF and PF ip address works. > > Ping between domU-1 and domU-2 works. > > But, ping from domU to another machine on same network on switch doesn''t > > work. > > > > The arp broadcast request goes out through VF interface. But the arp reply > > doesn''t reach VF interface, they get routed to PF interface. Why is it so? > > > > PF interface(eth1) is not part of the eth3 bridge. If I add eth1 to > > eth3-bridge, then ping from VMs to other machines on switch works. > > > > > > Is there something I am missing? How to make above experiment to work? > > Hi Staish, > > I suspect that there is some kind of driver issue. Unfortunately > the building that the dc7800 that I use is undergoing power-maintenance > (= power-outage) today, so I can''t investigate right now.The machine came back up and I seem to be able to reproduce the problem. Unfortunately I still don''t have any firm ideas on how to fix it. Have you tried the same thing with the pv_ops kernel, which is based on a new kernel rather than 2.6.18, which is quite ancient. I''m curious to know if the problem exists when using possibly more up to date drivers. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Simon, Thanks for the reply and reproducing the problem. I didn''t tried with pv_ops kernel yet. I guess the issue is not with the driver. As I understand, the behavior is because of the L2 filtering done by 82576 NIC. The NIC filters frames based on MAC address that are assigned to VFs. In my experiment, the arp reply that reaches the NIC, has mac address of interface on VM(domU). NIC''s filtering based on VF MAC address fail. So, packet is not queued to VF interface. On dom0 a bridge is created, with VF and domUs virtual interfaces (no PF). If we group PF interface also into this bridge, because of bridge learning functionality, the packet get routed to the virtual interface and finally to VM. I am not 100% sure about above understanding of mine. Please, me know if you think i am wrong. Regards, -Satish On Mon, Nov 9, 2009 at 4:41 PM, Simon Horman <horms@verge.net.au> wrote:> On Sun, Nov 08, 2009 at 09:09:43AM +1100, Simon Horman wrote: > > On Fri, Nov 06, 2009 at 06:35:40AM +0530, Satish Chowdhury wrote: > > > Hi, > > > > > > I am trying to make the VF interface of Intel 82576 dual port card, as > > > default outgoing interface on dom0. Data traffic from VMs (HVM) should > go > > > through VF interface on dom0. > > > > [snip] > > > > > Ping from domU-1 to VF and PF ip address works. > > > Ping between domU-1 and domU-2 works. > > > But, ping from domU to another machine on same network on switch > doesn''t > > > work. > > > > > > The arp broadcast request goes out through VF interface. But the arp > reply > > > doesn''t reach VF interface, they get routed to PF interface. Why is it > so? > > > > > > PF interface(eth1) is not part of the eth3 bridge. If I add eth1 to > > > eth3-bridge, then ping from VMs to other machines on switch works. > > > > > > > > > Is there something I am missing? How to make above experiment to work? > > > > Hi Staish, > > > > I suspect that there is some kind of driver issue. Unfortunately > > the building that the dc7800 that I use is undergoing power-maintenance > > (= power-outage) today, so I can''t investigate right now. > > The machine came back up and I seem to be able to reproduce the problem. > Unfortunately I still don''t have any firm ideas on how to fix it. > > Have you tried the same thing with the pv_ops kernel, which is > based on a new kernel rather than 2.6.18, which is quite ancient. > I''m curious to know if the problem exists when using possibly more up > to date drivers. > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Nov 10, 2009 at 01:19:51PM +0530, Satish Chowdhury wrote:> Hi Simon, > > Thanks for the reply and reproducing the problem. > > I didn''t tried with pv_ops kernel yet. > > I guess the issue is not with the driver. As I understand, the behavior is > because of the L2 filtering done by 82576 NIC. > > The NIC filters frames based on MAC address that are assigned to VFs. In my > experiment, the arp reply that reaches the NIC, has mac address of interface > on VM(domU). NIC''s filtering based on VF MAC address fail. So, packet is not > queued to VF interface. On dom0 a bridge is created, with VF and domUs > virtual interfaces (no PF). > If we group PF interface also into this bridge, because of bridge learning > functionality, the packet get routed to the virtual interface and finally to > VM. > > I am not 100% sure about above understanding of mine. Please, me know if you > think i am wrong.I suspect you are right, though I was hoping that its a problem that could be fixed by the driver configuring the card slightly differently. It would probably be good to post your problem report to the e1000-devel and/or netdev lists to get some more eye''s on it. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>-----Original Message----- >From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- >bounces@lists.xensource.com] On Behalf Of Simon Horman >Sent: Tuesday, November 10, 2009 11:38 AM >To: Satish Chowdhury >Cc: xen-devel@lists.xensource.com >Subject: Re: [Xen-devel] VF as default interface on dom0 > >On Tue, Nov 10, 2009 at 01:19:51PM +0530, Satish Chowdhury wrote: >> Hi Simon, >> >> Thanks for the reply and reproducing the problem. >> >> I didn''t tried with pv_ops kernel yet. >> >> I guess the issue is not with the driver. As I understand, the >behavior is >> because of the L2 filtering done by 82576 NIC. >> >> The NIC filters frames based on MAC address that are assigned to VFs. >In my >> experiment, the arp reply that reaches the NIC, has mac address of >interface >> on VM(domU). NIC''s filtering based on VF MAC address fail. So, packet >is not >> queued to VF interface. On dom0 a bridge is created, with VF and >domUs >> virtual interfaces (no PF). >> If we group PF interface also into this bridge, because of bridge >learning >> functionality, the packet get routed to the virtual interface and >finally to >> VM. >> >> I am not 100% sure about above understanding of mine. Please, me know >if you >> think i am wrong. > >I suspect you are right, though I was hoping that its a problem >that could be fixed by the driver configuring the card slightly >differently. > >It would probably be good to post your problem report to the >e1000-devel and/or netdev lists to get some more eye''s on it. > >[Rose, Gregory V] I must have missed some previous email on this subject but from the context here I''m guessing that you''re trying to put the PF driver on a bridge so that you can support some emulated connections in some VMs and some VFs in other VMs? Is that the case? - Greg _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, Nov 16, 2009 at 09:20:06AM -0800, Rose, Gregory V wrote:> >-----Original Message----- > >From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- > >bounces@lists.xensource.com] On Behalf Of Simon Horman > >Sent: Tuesday, November 10, 2009 11:38 AM > >To: Satish Chowdhury > >Cc: xen-devel@lists.xensource.com > >Subject: Re: [Xen-devel] VF as default interface on dom0 > > > >On Tue, Nov 10, 2009 at 01:19:51PM +0530, Satish Chowdhury wrote: > >> Hi Simon, > >> > >> Thanks for the reply and reproducing the problem. > >> > >> I didn''t tried with pv_ops kernel yet. > >> > >> I guess the issue is not with the driver. As I understand, the > >behavior is > >> because of the L2 filtering done by 82576 NIC. > >> > >> The NIC filters frames based on MAC address that are assigned to VFs. > >In my > >> experiment, the arp reply that reaches the NIC, has mac address of > >interface > >> on VM(domU). NIC''s filtering based on VF MAC address fail. So, packet > >is not > >> queued to VF interface. On dom0 a bridge is created, with VF and > >domUs > >> virtual interfaces (no PF). > >> If we group PF interface also into this bridge, because of bridge > >learning > >> functionality, the packet get routed to the virtual interface and > >finally to > >> VM. > >> > >> I am not 100% sure about above understanding of mine. Please, me know > >if you > >> think i am wrong. > > > >I suspect you are right, though I was hoping that its a problem > >that could be fixed by the driver configuring the card slightly > >differently. > > > >It would probably be good to post your problem report to the > >e1000-devel and/or netdev lists to get some more eye''s on it. > > > > > [Rose, Gregory V] > > I must have missed some previous email on this subject but from the context here I''m guessing that you''re trying to put the PF driver on a bridge so that you can support some emulated connections in some VMs and some VFs in other VMs? > > Is that the case?I''m unsure of the motivation but the problem is that putting a VF on the bridge (i.e VF is pethX) doesn''t seem to work correctly as some reply packets end up on the PF instead. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel