Hello, I''m facing a strange issue with network card PCI passthrough on my openwrt test domU. - With network PCI passthrough, DNS lookup failed for some domains (exemple, google.com) but not for other (free.fr my ISP, or my domain jbfavre.org). I can ping an IP address without any problem. - Starting domU as a "normal" (ie without PCI passthrough), no problem. As far as I can say, domU is not the root cause. I really think this is related to PCI passsthrough. This seems to be related to packet length. Did not see anything strange in dom0 logs. Is there any incompatibility between 2.6.32 dom0 kernel with Xen 4.0.1 and 2.6.37 domU kernel ? Regards, JB Dom0: Debian Squeeze # uname -a Linux remus 2.6.32-5-xen-amd64 # dpkg -l|grep hyper ii xen-hypervisor-4.0-amd64 4.0.1-1 DomU: Openwrt backfire # uname -a Linux OpenWrt 2.6.37 # lsmod Module Size Used by Not tainted sky2 35744 0 All Xen related stuff is integrated into the kernel, not compiled as module. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Jan 12, 2011 at 04:38:49PM +0100, Jean Baptiste Favre wrote:> Hello, > I''m facing a strange issue with network card PCI passthrough on my > openwrt test domU. > > - With network PCI passthrough, DNS lookup failed for some domains > (exemple, google.com) but not for other (free.fr my ISP, or my domain > jbfavre.org). I can ping an IP address without any problem.Do you have "both" (so PCI passthrough and the Xen network driver) in the guest? If so, have you tried eliminating the xen network driver to see if it is just a routing issue? What does your routing table look like? Your IP table?> > - Starting domU as a "normal" (ie without PCI passthrough), no problem. > > > As far as I can say, domU is not the root cause. I really think this is > related to PCI passsthrough. This seems to be related to packet length.Then that would imply the MTU is not set right.> > Did not see anything strange in dom0 logs. > > Is there any incompatibility between 2.6.32 dom0 kernel with Xen 4.0.1 > and 2.6.37 domU kernel ?No.> > Regards, > JB > > Dom0: Debian Squeeze > # uname -a > Linux remus 2.6.32-5-xen-amd64 > # dpkg -l|grep hyper > ii xen-hypervisor-4.0-amd64 4.0.1-1 > > DomU: Openwrt backfire > # uname -a > Linux OpenWrt 2.6.37 > # lsmod > Module Size Used by Not tainted > sky2 35744 0 > All Xen related stuff is integrated into the kernel, not compiled as module. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello Konrad, Le 12/01/2011 16:43, Konrad Rzeszutek Wilk a écrit :> On Wed, Jan 12, 2011 at 04:38:49PM +0100, Jean Baptiste Favre wrote: >> Hello, >> I''m facing a strange issue with network card PCI passthrough on my >> openwrt test domU. >> >> - With network PCI passthrough, DNS lookup failed for some domains >> (exemple, google.com) but not for other (free.fr my ISP, or my domain >> jbfavre.org). I can ping an IP address without any problem. > > Do you have "both" (so PCI passthrough and the Xen network driver) > in the guest? If so, have you tried eliminating the xen network driver > to see if it is just a routing issue?Have not tried to eliminate xen network driver. Think I have both drivers. My kernel .config looks like: $ grep XEN build_dir/linux-x86_xen_domu/linux-2.6.37/.config CONFIG_XEN=y # CONFIG_XEN_PRIVILEGED_GUEST is not set CONFIG_XEN_PVHVM=y CONFIG_XEN_MAX_DOMAIN_MEMORY=128 CONFIG_XEN_SAVE_RESTORE=y CONFIG_PCI_XEN=y CONFIG_XEN_PCIDEV_FRONTEND=y CONFIG_XEN_BLKDEV_FRONTEND=y CONFIG_XEN_NETDEV_FRONTEND=y CONFIG_HVC_XEN=y CONFIG_XEN_BALLOON=y CONFIG_XEN_SCRUB_PAGES=y CONFIG_XEN_DEV_EVTCHN=y CONFIG_XENFS=y CONFIG_XEN_COMPAT_XENFS=y CONFIG_XEN_SYS_HYPERVISOR=y # CONFIG_XEN_PLATFORM_PCI is not set CONFIG_SWIOTLB_XEN=y So, I should remove CONFIG_XEN_NETDEV_FRONTEND ?> What does your routing table look like? Your IP table?My routing table is pretty clean, nothing strange here # route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 br-wan 0.0.0.0 10.0.0.1 0.0.0.0 UG 0 0 0 br-wan>> - Starting domU as a "normal" (ie without PCI passthrough), no problem. >> >> >> As far as I can say, domU is not the root cause. I really think this is >> related to PCI passsthrough. This seems to be related to packet length. > > Then that would imply the MTU is not set right.Already checked it: 1500 :)>> Did not see anything strange in dom0 logs. >> >> Is there any incompatibility between 2.6.32 dom0 kernel with Xen 4.0.1 >> and 2.6.37 domU kernel ? > > No.So, will try to remove Xen Network driver and see what happen. Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello again, Just tried without Xen network driver and got the same behaviour. Would say hopefully because removing xen network driver prevent me to use my domU as firewall for dom0 as I intented to do. BTW, have no idea left :-/ Regards, JB Le 12/01/2011 16:53, Jean Baptiste Favre a écrit :> Hello Konrad, > > Le 12/01/2011 16:43, Konrad Rzeszutek Wilk a écrit : >> On Wed, Jan 12, 2011 at 04:38:49PM +0100, Jean Baptiste Favre wrote: >>> Hello, >>> I''m facing a strange issue with network card PCI passthrough on my >>> openwrt test domU. >>> >>> - With network PCI passthrough, DNS lookup failed for some domains >>> (exemple, google.com) but not for other (free.fr my ISP, or my domain >>> jbfavre.org). I can ping an IP address without any problem. >> >> Do you have "both" (so PCI passthrough and the Xen network driver) >> in the guest? If so, have you tried eliminating the xen network driver >> to see if it is just a routing issue? > Have not tried to eliminate xen network driver. Think I have both drivers. > > My kernel .config looks like: > $ grep XEN build_dir/linux-x86_xen_domu/linux-2.6.37/.config > CONFIG_XEN=y > # CONFIG_XEN_PRIVILEGED_GUEST is not set > CONFIG_XEN_PVHVM=y > CONFIG_XEN_MAX_DOMAIN_MEMORY=128 > CONFIG_XEN_SAVE_RESTORE=y > CONFIG_PCI_XEN=y > CONFIG_XEN_PCIDEV_FRONTEND=y > CONFIG_XEN_BLKDEV_FRONTEND=y > CONFIG_XEN_NETDEV_FRONTEND=y > CONFIG_HVC_XEN=y > CONFIG_XEN_BALLOON=y > CONFIG_XEN_SCRUB_PAGES=y > CONFIG_XEN_DEV_EVTCHN=y > CONFIG_XENFS=y > CONFIG_XEN_COMPAT_XENFS=y > CONFIG_XEN_SYS_HYPERVISOR=y > # CONFIG_XEN_PLATFORM_PCI is not set > CONFIG_SWIOTLB_XEN=y > > So, I should remove CONFIG_XEN_NETDEV_FRONTEND ? > >> What does your routing table look like? Your IP table? > My routing table is pretty clean, nothing strange here > # route -n > Kernel IP routing table > Destination Gateway Genmask Flags Metric Ref Use > Iface > 10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 > br-wan > 0.0.0.0 10.0.0.1 0.0.0.0 UG 0 0 0 > br-wan > >>> - Starting domU as a "normal" (ie without PCI passthrough), no problem. >>> >>> >>> As far as I can say, domU is not the root cause. I really think this is >>> related to PCI passsthrough. This seems to be related to packet length. >> >> Then that would imply the MTU is not set right. > Already checked it: 1500 :) > >>> Did not see anything strange in dom0 logs. >>> >>> Is there any incompatibility between 2.6.32 dom0 kernel with Xen 4.0.1 >>> and 2.6.37 domU kernel ? >> >> No. > So, will try to remove Xen Network driver and see what happen. > > Regards, > JB > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Jan 12, 2011 at 04:53:51PM +0100, Jean Baptiste Favre wrote:> Hello Konrad, > > Le 12/01/2011 16:43, Konrad Rzeszutek Wilk a écrit : > > On Wed, Jan 12, 2011 at 04:38:49PM +0100, Jean Baptiste Favre wrote: > >> Hello, > >> I''m facing a strange issue with network card PCI passthrough on my > >> openwrt test domU. > >> > >> - With network PCI passthrough, DNS lookup failed for some domains > >> (exemple, google.com) but not for other (free.fr my ISP, or my domain > >> jbfavre.org). I can ping an IP address without any problem. > > > > Do you have "both" (so PCI passthrough and the Xen network driver) > > in the guest? If so, have you tried eliminating the xen network driver > > to see if it is just a routing issue? > Have not tried to eliminate xen network driver. Think I have both drivers. > > My kernel .config looks like: > $ grep XEN build_dir/linux-x86_xen_domu/linux-2.6.37/.config > CONFIG_XEN=y > # CONFIG_XEN_PRIVILEGED_GUEST is not set > CONFIG_XEN_PVHVM=y > CONFIG_XEN_MAX_DOMAIN_MEMORY=128 > CONFIG_XEN_SAVE_RESTORE=y > CONFIG_PCI_XEN=y > CONFIG_XEN_PCIDEV_FRONTEND=y > CONFIG_XEN_BLKDEV_FRONTEND=y > CONFIG_XEN_NETDEV_FRONTEND=y > CONFIG_HVC_XEN=y > CONFIG_XEN_BALLOON=y > CONFIG_XEN_SCRUB_PAGES=y > CONFIG_XEN_DEV_EVTCHN=y > CONFIG_XENFS=y > CONFIG_XEN_COMPAT_XENFS=y > CONFIG_XEN_SYS_HYPERVISOR=y > # CONFIG_XEN_PLATFORM_PCI is not set > CONFIG_SWIOTLB_XEN=y > > So, I should remove CONFIG_XEN_NETDEV_FRONTEND ?No. You can just do ''ifconfig <x> down'' whatever your Xen netfront NIC is. .. but.> > > What does your routing table look like? Your IP table? > My routing table is pretty clean, nothing strange here > # route -n > Kernel IP routing table > Destination Gateway Genmask Flags Metric Ref Use > Iface > 10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 > br-wan > 0.0.0.0 10.0.0.1 0.0.0.0 UG 0 0 0 > br-wanOk, then the idea that the Xen networking driver and the PCI passthrough send packets over is not the way.. So ignore about the Xen networking part.> > >> - Starting domU as a "normal" (ie without PCI passthrough), no problem. > >> > >> > >> As far as I can say, domU is not the root cause. I really think this is > >> related to PCI passsthrough. This seems to be related to packet length. > > > > Then that would imply the MTU is not set right. > Already checked it: 1500 :)Ok. Next thing, did you try to disable the rx/tx checksumming? If you connect the Ethernet cable for this ''br-wan'' device to another machine (so you could set it up as bridge and just let it pass through packets and sniff the data) what do the packets look like? What happens if the PCI passthrough device is not under the ownership of a bridge? What then? You wouldn''t have any bridge firewall code in? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Le 12/01/2011 17:36, Konrad Rzeszutek Wilk a écrit :> On Wed, Jan 12, 2011 at 04:53:51PM +0100, Jean Baptiste Favre wrote: >> Hello Konrad, >> >> Le 12/01/2011 16:43, Konrad Rzeszutek Wilk a écrit : >>> On Wed, Jan 12, 2011 at 04:38:49PM +0100, Jean Baptiste Favre wrote: >>>> Hello, >>>> I''m facing a strange issue with network card PCI passthrough on my >>>> openwrt test domU. >>>> >>>> - With network PCI passthrough, DNS lookup failed for some domains >>>> (exemple, google.com) but not for other (free.fr my ISP, or my domain >>>> jbfavre.org). I can ping an IP address without any problem. >>> >>> Do you have "both" (so PCI passthrough and the Xen network driver) >>> in the guest? If so, have you tried eliminating the xen network driver >>> to see if it is just a routing issue? >> Have not tried to eliminate xen network driver. Think I have both drivers. >> >> My kernel .config looks like: >> $ grep XEN build_dir/linux-x86_xen_domu/linux-2.6.37/.config >> CONFIG_XEN=y >> # CONFIG_XEN_PRIVILEGED_GUEST is not set >> CONFIG_XEN_PVHVM=y >> CONFIG_XEN_MAX_DOMAIN_MEMORY=128 >> CONFIG_XEN_SAVE_RESTORE=y >> CONFIG_PCI_XEN=y >> CONFIG_XEN_PCIDEV_FRONTEND=y >> CONFIG_XEN_BLKDEV_FRONTEND=y >> CONFIG_XEN_NETDEV_FRONTEND=y >> CONFIG_HVC_XEN=y >> CONFIG_XEN_BALLOON=y >> CONFIG_XEN_SCRUB_PAGES=y >> CONFIG_XEN_DEV_EVTCHN=y >> CONFIG_XENFS=y >> CONFIG_XEN_COMPAT_XENFS=y >> CONFIG_XEN_SYS_HYPERVISOR=y >> # CONFIG_XEN_PLATFORM_PCI is not set >> CONFIG_SWIOTLB_XEN=y >> >> So, I should remove CONFIG_XEN_NETDEV_FRONTEND ? > > No. You can just do ''ifconfig <x> down'' whatever your Xen netfront > NIC is. .. but. >> >>> What does your routing table look like? Your IP table? >> My routing table is pretty clean, nothing strange here >> # route -n >> Kernel IP routing table >> Destination Gateway Genmask Flags Metric Ref Use >> Iface >> 10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 >> br-wan >> 0.0.0.0 10.0.0.1 0.0.0.0 UG 0 0 0 >> br-wan > > Ok, then the idea that the Xen networking driver and the PCI passthrough send > packets over is not the way.. > > So ignore about the Xen networking part.OK.>> >>>> - Starting domU as a "normal" (ie without PCI passthrough), no problem. >>>> >>>> >>>> As far as I can say, domU is not the root cause. I really think this is >>>> related to PCI passsthrough. This seems to be related to packet length. >>> >>> Then that would imply the MTU is not set right. >> Already checked it: 1500 :) > Ok. > > Next thing, did you try to disable the rx/tx checksumming?No. Not sure I even know how to do it, but will have a look on that.> If you connect the Ethernet cable for this ''br-wan'' device to another machine > (so you could set it up as bridge and just let it pass through packets and sniff > the data) what do the packets look like?I can see packets on my gateway (which acts as DNS as well). They looks good. I can see DNS answers as well leaving the gateway, but not reaching my domU. Have only basic switches between my gateway and my domU.> What happens if the PCI passthrough device is not under the ownership of a bridge? > What then? You wouldn''t have any bridge firewall code in?Removed bridge + configure eth: no change. Checked ebtables/iptables (all tables: net filter and mangle) rules: empty, Policy to ACCEPT. Tried to enable/disable ip_forwarding: no change Also disabled IPV6 support (I saw some DNS answers as AAAA but no IPv6 available at home for now): no change Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Jan 12, 2011 at 05:56:45PM +0100, Jean Baptiste Favre wrote:> Le 12/01/2011 17:36, Konrad Rzeszutek Wilk a écrit : > > On Wed, Jan 12, 2011 at 04:53:51PM +0100, Jean Baptiste Favre wrote: > >> Hello Konrad, > >> > >> Le 12/01/2011 16:43, Konrad Rzeszutek Wilk a écrit : > >>> On Wed, Jan 12, 2011 at 04:38:49PM +0100, Jean Baptiste Favre wrote: > >>>> Hello, > >>>> I''m facing a strange issue with network card PCI passthrough on my > >>>> openwrt test domU. > >>>> > >>>> - With network PCI passthrough, DNS lookup failed for some domains > >>>> (exemple, google.com) but not for other (free.fr my ISP, or my domain > >>>> jbfavre.org). I can ping an IP address without any problem. > >>> > >>> Do you have "both" (so PCI passthrough and the Xen network driver) > >>> in the guest? If so, have you tried eliminating the xen network driver > >>> to see if it is just a routing issue? > >> Have not tried to eliminate xen network driver. Think I have both drivers. > >> > >> My kernel .config looks like: > >> $ grep XEN build_dir/linux-x86_xen_domu/linux-2.6.37/.config > >> CONFIG_XEN=y > >> # CONFIG_XEN_PRIVILEGED_GUEST is not set > >> CONFIG_XEN_PVHVM=y > >> CONFIG_XEN_MAX_DOMAIN_MEMORY=128 > >> CONFIG_XEN_SAVE_RESTORE=y > >> CONFIG_PCI_XEN=y > >> CONFIG_XEN_PCIDEV_FRONTEND=y > >> CONFIG_XEN_BLKDEV_FRONTEND=y > >> CONFIG_XEN_NETDEV_FRONTEND=y > >> CONFIG_HVC_XEN=y > >> CONFIG_XEN_BALLOON=y > >> CONFIG_XEN_SCRUB_PAGES=y > >> CONFIG_XEN_DEV_EVTCHN=y > >> CONFIG_XENFS=y > >> CONFIG_XEN_COMPAT_XENFS=y > >> CONFIG_XEN_SYS_HYPERVISOR=y > >> # CONFIG_XEN_PLATFORM_PCI is not set > >> CONFIG_SWIOTLB_XEN=y > >> > >> So, I should remove CONFIG_XEN_NETDEV_FRONTEND ? > > > > No. You can just do ''ifconfig <x> down'' whatever your Xen netfront > > NIC is. .. but. > >> > >>> What does your routing table look like? Your IP table? > >> My routing table is pretty clean, nothing strange here > >> # route -n > >> Kernel IP routing table > >> Destination Gateway Genmask Flags Metric Ref Use > >> Iface > >> 10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 > >> br-wan > >> 0.0.0.0 10.0.0.1 0.0.0.0 UG 0 0 0 > >> br-wan > > > > Ok, then the idea that the Xen networking driver and the PCI passthrough send > > packets over is not the way.. > > > > So ignore about the Xen networking part. > > OK. > > >> > >>>> - Starting domU as a "normal" (ie without PCI passthrough), no problem. > >>>> > >>>> > >>>> As far as I can say, domU is not the root cause. I really think this is > >>>> related to PCI passsthrough. This seems to be related to packet length. > >>> > >>> Then that would imply the MTU is not set right. > >> Already checked it: 1500 :) > > Ok. > > > > Next thing, did you try to disable the rx/tx checksumming? > > No. Not sure I even know how to do it, but will have a look on that. > > > If you connect the Ethernet cable for this ''br-wan'' device to another machine > > (so you could set it up as bridge and just let it pass through packets and sniff > > the data) what do the packets look like? > > I can see packets on my gateway (which acts as DNS as well). They looks > good. I can see DNS answers as well leaving the gateway, but not > reaching my domU. Have only basic switches between my gateway and my domU.Wait a minute. How are your domU''s connected? You did turn on forwarding in your guest with the PCI passthrough right? How is your domain (with the PCI passthrought) connected to the other domains?> > > What happens if the PCI passthrough device is not under the ownership of a bridge? > > What then? You wouldn''t have any bridge firewall code in? > > Removed bridge + configure eth: no change. > Checked ebtables/iptables (all tables: net filter and mangle) rules: > empty, Policy to ACCEPT. > Tried to enable/disable ip_forwarding: no change > Also disabled IPV6 support (I saw some DNS answers as AAAA but no IPv6 > available at home for now): no change > > Regards, > JB > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Le 12/01/2011 18:26, Konrad Rzeszutek Wilk a écrit :> On Wed, Jan 12, 2011 at 05:56:45PM +0100, Jean Baptiste Favre wrote: >> Le 12/01/2011 17:36, Konrad Rzeszutek Wilk a écrit : >>> On Wed, Jan 12, 2011 at 04:53:51PM +0100, Jean Baptiste Favre wrote: >>>> Hello Konrad, >>>> >>>> Le 12/01/2011 16:43, Konrad Rzeszutek Wilk a écrit : >>>>> On Wed, Jan 12, 2011 at 04:38:49PM +0100, Jean Baptiste Favre wrote: >>>>>> Hello, >>>>>> I''m facing a strange issue with network card PCI passthrough on my >>>>>> openwrt test domU. >>>>>> >>>>>> - With network PCI passthrough, DNS lookup failed for some domains >>>>>> (exemple, google.com) but not for other (free.fr my ISP, or my domain >>>>>> jbfavre.org). I can ping an IP address without any problem. >>>>> >>>>> Do you have "both" (so PCI passthrough and the Xen network driver) >>>>> in the guest? If so, have you tried eliminating the xen network driver >>>>> to see if it is just a routing issue? >>>> Have not tried to eliminate xen network driver. Think I have both drivers. >>>> >>>> My kernel .config looks like: >>>> $ grep XEN build_dir/linux-x86_xen_domu/linux-2.6.37/.config >>>> CONFIG_XEN=y >>>> # CONFIG_XEN_PRIVILEGED_GUEST is not set >>>> CONFIG_XEN_PVHVM=y >>>> CONFIG_XEN_MAX_DOMAIN_MEMORY=128 >>>> CONFIG_XEN_SAVE_RESTORE=y >>>> CONFIG_PCI_XEN=y >>>> CONFIG_XEN_PCIDEV_FRONTEND=y >>>> CONFIG_XEN_BLKDEV_FRONTEND=y >>>> CONFIG_XEN_NETDEV_FRONTEND=y >>>> CONFIG_HVC_XEN=y >>>> CONFIG_XEN_BALLOON=y >>>> CONFIG_XEN_SCRUB_PAGES=y >>>> CONFIG_XEN_DEV_EVTCHN=y >>>> CONFIG_XENFS=y >>>> CONFIG_XEN_COMPAT_XENFS=y >>>> CONFIG_XEN_SYS_HYPERVISOR=y >>>> # CONFIG_XEN_PLATFORM_PCI is not set >>>> CONFIG_SWIOTLB_XEN=y >>>> >>>> So, I should remove CONFIG_XEN_NETDEV_FRONTEND ? >>> >>> No. You can just do ''ifconfig <x> down'' whatever your Xen netfront >>> NIC is. .. but. >>>> >>>>> What does your routing table look like? Your IP table? >>>> My routing table is pretty clean, nothing strange here >>>> # route -n >>>> Kernel IP routing table >>>> Destination Gateway Genmask Flags Metric Ref Use >>>> Iface >>>> 10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 >>>> br-wan >>>> 0.0.0.0 10.0.0.1 0.0.0.0 UG 0 0 0 >>>> br-wan >>> >>> Ok, then the idea that the Xen networking driver and the PCI passthrough send >>> packets over is not the way.. >>> >>> So ignore about the Xen networking part. >> >> OK. >> >>>> >>>>>> - Starting domU as a "normal" (ie without PCI passthrough), no problem. >>>>>> >>>>>> >>>>>> As far as I can say, domU is not the root cause. I really think this is >>>>>> related to PCI passsthrough. This seems to be related to packet length. >>>>> >>>>> Then that would imply the MTU is not set right. >>>> Already checked it: 1500 :) >>> Ok. >>> >>> Next thing, did you try to disable the rx/tx checksumming? >> >> No. Not sure I even know how to do it, but will have a look on that. >> >>> If you connect the Ethernet cable for this ''br-wan'' device to another machine >>> (so you could set it up as bridge and just let it pass through packets and sniff >>> the data) what do the packets look like? >> >> I can see packets on my gateway (which acts as DNS as well). They looks >> good. I can see DNS answers as well leaving the gateway, but not >> reaching my domU. Have only basic switches between my gateway and my domU. > > Wait a minute. How are your domU''s connected? You did turn on > forwarding in your guest with the PCI passthrough right?My OpenWRT domU with PCI passthrough is directly connected to my LAN through network card> How is your domain (with the PCI passthrought) connected to the > other domains?For the moment, other domU are not connected in any way to my OpenWRT domU. I want first to make sur PCI passthrough is working before trying more complex architecture. My OpenWRT domU network setup is: dom0 domU LAN vif ----- eth0 eth1 ----- SW ----- GW ----- NET |__ br __| All tests are done from OpenWRT domU for now (but I guess this won''t work either from dom0). I also tried following setup: domU LAN eth1 ----- SW ----- GW ----- NET br __| That means no vif from dom0. But with PCI passthrough, I should reach network directly, right ? Finally, I think that my setup shall not be responsible for the fact that ping works, small DNS answers are received but not bigger ones (limit seems to be around 100 bytes).>>> What happens if the PCI passthrough device is not under the ownership of a bridge? >>> What then? You wouldn''t have any bridge firewall code in? >> >> Removed bridge + configure eth: no change. >> Checked ebtables/iptables (all tables: net filter and mangle) rules: >> empty, Policy to ACCEPT. >> Tried to enable/disable ip_forwarding: no change >> Also disabled IPV6 support (I saw some DNS answers as AAAA but no IPv6 >> available at home for now): no change >> >> Regards, >> JB_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > Finally, I think that my setup shall not be responsible for the fact > that ping works, small DNS answers are received but not bigger onesAah, somehow I missed that fact.> (limit seems to be around 100 bytes).What does wireshark or tcpdump tell you when you send the larger packets? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Le 12/01/2011 19:32, Konrad Rzeszutek Wilk a écrit :>> >> Finally, I think that my setup shall not be responsible for the fact >> that ping works, small DNS answers are received but not bigger ones > > Aah, somehow I missed that fact.Seems yes :)>> (limit seems to be around 100 bytes). > > What does wireshark or tcpdump tell you when you send the larger packets?That''s one of my problem: nothing special :( I can only run tcpdump on my gateway (no SSH available on my OpenWRT domU and only xm console access is available). I see DNS requests comming in, I see DNS answers going out. ARP requests are fine. As an example: dig free.fr works dig jbfavre.org works dig amazon.fr works dig amazon.com works (quite long, but works) dig google.fr fails dig google.com fails>From another machine in the same LAN, all requests work fine.Thought this was due to some ARP restriction setted up by default in OpenWRT, but in this case, I can not explain why some DNS lookup do work and some don''t. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Jan 12, 2011 at 09:07:30PM +0100, Jean Baptiste Favre wrote:> Le 12/01/2011 19:32, Konrad Rzeszutek Wilk a écrit : > >> > >> Finally, I think that my setup shall not be responsible for the fact > >> that ping works, small DNS answers are received but not bigger ones > > > > Aah, somehow I missed that fact. > > Seems yes :) > > >> (limit seems to be around 100 bytes). > > > > What does wireshark or tcpdump tell you when you send the larger packets? > > That''s one of my problem: nothing special :( > > I can only run tcpdump on my gateway (no SSH available on my OpenWRT > domU and only xm console access is available).You can install (or just compile it statically) tcpdump on your OpenWRT, or just hook it up to a beffier guest..> I see DNS requests comming in, I see DNS answers going out. > ARP requests are fine. > > As an example: > dig free.fr works > dig jbfavre.org works > dig amazon.fr works > dig amazon.com works (quite long, but works) > dig google.fr fails > dig google.com failsFails as in response or fails halfway?> > >From another machine in the same LAN, all requests work fine. > > Thought this was due to some ARP restriction setted up by default in > OpenWRT, but in this case, I can not explain why some DNS lookup do work > and some don''t. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Strange, Just noticed some message I missed before: When starting domain: [175004.836398] pciback 0000:04:00.0: device has been assigned to another domain! Over-writting the ownership, but beware. and (XEN) traps.c:2302:d44 Domain attempted WRMSR 00000000c0010004 from 0000c827:4e390408 to 00000000:0000abcd. The first one is very strange as I''m absolutely sure device is NOT assigned to any domain (the only domU I have on this dom0 is my OpenWRT). The second one... well, I don''t know what it''s talking about :-/ Regards, JB Le 12/01/2011 21:07, Jean Baptiste Favre a écrit :> Le 12/01/2011 19:32, Konrad Rzeszutek Wilk a écrit : >>> >>> Finally, I think that my setup shall not be responsible for the fact >>> that ping works, small DNS answers are received but not bigger ones >> >> Aah, somehow I missed that fact. > > Seems yes :) > >>> (limit seems to be around 100 bytes). >> >> What does wireshark or tcpdump tell you when you send the larger packets? > > That''s one of my problem: nothing special :( > > I can only run tcpdump on my gateway (no SSH available on my OpenWRT > domU and only xm console access is available). > I see DNS requests comming in, I see DNS answers going out. > ARP requests are fine. > > As an example: > dig free.fr works > dig jbfavre.org works > dig amazon.fr works > dig amazon.com works (quite long, but works) > dig google.fr fails > dig google.com fails > >>From another machine in the same LAN, all requests work fine. > > Thought this was due to some ARP restriction setted up by default in > OpenWRT, but in this case, I can not explain why some DNS lookup do work > and some don''t. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Le 12/01/2011 22:40, Konrad Rzeszutek Wilk a écrit :> On Wed, Jan 12, 2011 at 09:07:30PM +0100, Jean Baptiste Favre wrote: >> Le 12/01/2011 19:32, Konrad Rzeszutek Wilk a écrit : >>>> >>>> Finally, I think that my setup shall not be responsible for the fact >>>> that ping works, small DNS answers are received but not bigger ones >>> >>> Aah, somehow I missed that fact. >> >> Seems yes :) >> >>>> (limit seems to be around 100 bytes). >>> >>> What does wireshark or tcpdump tell you when you send the larger packets? >> >> That''s one of my problem: nothing special :( >> >> I can only run tcpdump on my gateway (no SSH available on my OpenWRT >> domU and only xm console access is available). > > You can install (or just compile it statically) tcpdump on your OpenWRT, > or just hook it up to a beffier guest..tcpdump is available on my OpenWRT domU, but I can not perform tcpdump and DNS lookup with only console access and AFAIK screen is not available on OpenWRT>> I see DNS requests comming in, I see DNS answers going out. >> ARP requests are fine. >> >> As an example: >> dig free.fr works >> dig jbfavre.org works >> dig amazon.fr works >> dig amazon.com works (quite long, but works) >> dig google.fr fails >> dig google.com fails > > Fails as in response or fails halfway?Fails as in: # dig google.com ; <<>> DiG 9.7.2-P3 <<>> google.com ;; global options: +cmd ;; connection timed out; no servers could be reached>> >> >From another machine in the same LAN, all requests work fine. >> >> Thought this was due to some ARP restriction setted up by default in >> OpenWRT, but in this case, I can not explain why some DNS lookup do work >> and some don''t._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ok, my bad. Screen IS available on OpenWRT. I''ve added it to my disk image and now have a tcpdump capture of network traffic of my DomU Have the same network capture from my GW so I ''ll be able to investigate deeper. Let me know if you have specific points you want me to look at. Regards, JB Le 12/01/2011 22:46, Jean Baptiste Favre a écrit :> Le 12/01/2011 22:40, Konrad Rzeszutek Wilk a écrit : >> On Wed, Jan 12, 2011 at 09:07:30PM +0100, Jean Baptiste Favre wrote: >>> Le 12/01/2011 19:32, Konrad Rzeszutek Wilk a écrit : >>>>> >>>>> Finally, I think that my setup shall not be responsible for the fact >>>>> that ping works, small DNS answers are received but not bigger ones >>>> >>>> Aah, somehow I missed that fact. >>> >>> Seems yes :) >>> >>>>> (limit seems to be around 100 bytes). >>>> >>>> What does wireshark or tcpdump tell you when you send the larger packets? >>> >>> That''s one of my problem: nothing special :( >>> >>> I can only run tcpdump on my gateway (no SSH available on my OpenWRT >>> domU and only xm console access is available). >> >> You can install (or just compile it statically) tcpdump on your OpenWRT, >> or just hook it up to a beffier guest.. > > tcpdump is available on my OpenWRT domU, but I can not perform tcpdump > and DNS lookup with only console access and AFAIK screen is not > available on OpenWRT > >>> I see DNS requests comming in, I see DNS answers going out. >>> ARP requests are fine. >>> >>> As an example: >>> dig free.fr works >>> dig jbfavre.org works >>> dig amazon.fr works >>> dig amazon.com works (quite long, but works) >>> dig google.fr fails >>> dig google.com fails >> >> Fails as in response or fails halfway? > > Fails as in: > # dig google.com > > ; <<>> DiG 9.7.2-P3 <<>> google.com > ;; global options: +cmd > ;; connection timed out; no servers could be reached > >>> >>> >From another machine in the same LAN, all requests work fine. >>> >>> Thought this was due to some ARP restriction setted up by default in >>> OpenWRT, but in this case, I can not explain why some DNS lookup do work >>> and some don''t. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Have had a look on both tcpdump captures. 1. From the OpenWRT domU: nothing special. Lot of STP traffic due to my network architecture. Don''t thing this could be the cause of my problem since I''m always able to reproduce the failure whatever could be the order of DNS requests. Don''t see "big" DNS answers coming in, see multiple requests before timeout. 2. From my gateway: nothing special here too. Same STP traffic (have checked GW network setup as well). See each DNS requests my domU sent and can see as well answers, which are not received by domU. No strange ARP traffic. I wanted to change the network, thinking about a bogus driver (never know). My Dom0 has crashed during reboot and I''ve no way to remotely reboot it (no remote power control and it seems that Magic SysRQ are not transfered through my Putty/screen connection). Will reboot it when at home and continue investigation with another network card and maybe another operating system to check if bad behaviour is still there. Regards, JB Le 12/01/2011 23:18, Jean Baptiste Favre a écrit :> Ok, my bad. Screen IS available on OpenWRT. > I''ve added it to my disk image and now have a tcpdump capture of network > traffic of my DomU > > Have the same network capture from my GW so I ''ll be able to investigate > deeper. > > Let me know if you have specific points you want me to look at. > > Regards, > JB > > > Le 12/01/2011 22:46, Jean Baptiste Favre a écrit : >> Le 12/01/2011 22:40, Konrad Rzeszutek Wilk a écrit : >>> On Wed, Jan 12, 2011 at 09:07:30PM +0100, Jean Baptiste Favre wrote: >>>> Le 12/01/2011 19:32, Konrad Rzeszutek Wilk a écrit : >>>>>> >>>>>> Finally, I think that my setup shall not be responsible for the fact >>>>>> that ping works, small DNS answers are received but not bigger ones >>>>> >>>>> Aah, somehow I missed that fact. >>>> >>>> Seems yes :) >>>> >>>>>> (limit seems to be around 100 bytes). >>>>> >>>>> What does wireshark or tcpdump tell you when you send the larger packets? >>>> >>>> That''s one of my problem: nothing special :( >>>> >>>> I can only run tcpdump on my gateway (no SSH available on my OpenWRT >>>> domU and only xm console access is available). >>> >>> You can install (or just compile it statically) tcpdump on your OpenWRT, >>> or just hook it up to a beffier guest.. >> >> tcpdump is available on my OpenWRT domU, but I can not perform tcpdump >> and DNS lookup with only console access and AFAIK screen is not >> available on OpenWRT >> >>>> I see DNS requests comming in, I see DNS answers going out. >>>> ARP requests are fine. >>>> >>>> As an example: >>>> dig free.fr works >>>> dig jbfavre.org works >>>> dig amazon.fr works >>>> dig amazon.com works (quite long, but works) >>>> dig google.fr fails >>>> dig google.com fails >>> >>> Fails as in response or fails halfway? >> >> Fails as in: >> # dig google.com >> >> ; <<>> DiG 9.7.2-P3 <<>> google.com >> ;; global options: +cmd >> ;; connection timed out; no servers could be reached >> >>>> >>>> >From another machine in the same LAN, all requests work fine. >>>> >>>> Thought this was due to some ARP restriction setted up by default in >>>> OpenWRT, but in this case, I can not explain why some DNS lookup do work >>>> and some don''t._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, My dom0 is back and I performed some more tests. I told in my first mail that ping works. Indeed it works, but not always: # ping -c2 10.0.0.1 PING 10.0.0.1 (10.0.0.1): 56 data bytes 64 bytes from 10.0.0.1: seq=0 ttl=64 time=0.846 ms 64 bytes from 10.0.0.1: seq=1 ttl=64 time=0.824 ms --- 10.0.0.1 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.824/0.835/0.846 ms # ping -c2 -s60 10.0.0.1 PING 10.0.0.1 (10.0.0.1): 60 data bytes 68 bytes from 10.0.0.1: seq=0 ttl=64 time=0.819 ms 68 bytes from 10.0.0.1: seq=1 ttl=64 time=0.807 ms --- 10.0.0.1 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.807/0.813/0.819 ms Increasing packet size is ok until this one: # ping -c2 -s85 10.0.0.1 PING 10.0.0.1 (10.0.0.1): 85 data bytes 93 bytes from 10.0.0.1: seq=0 ttl=64 time=0.823 ms 93 bytes from 10.0.0.1: seq=1 ttl=64 time=0.816 ms --- 10.0.0.1 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.816/0.819/0.823 ms root@OpenWrt:/# ping -c2 -s86 10.0.0.1 PING 10.0.0.1 (10.0.0.1): 86 data bytes --- 10.0.0.1 ping statistics --- 2 packets transmitted, 0 packets received, 100% packet loss As you see, packet size seems to be limited in a way. From another machine on the same LAN I can do ping -s1500 without any problem. So I think I hit a bug. Either it''s Xen related, or Debian (through debian kernel version). Now the question is: how can I determine which part is responsible ? Regards, JB Le 13/01/2011 12:28, Jean Baptiste Favre a écrit :> Have had a look on both tcpdump captures. > 1. From the OpenWRT domU: nothing special. Lot of STP traffic due to my > network architecture. Don''t thing this could be the cause of my problem > since I''m always able to reproduce the failure whatever could be the > order of DNS requests. Don''t see "big" DNS answers coming in, see > multiple requests before timeout. > > 2. From my gateway: nothing special here too. Same STP traffic (have > checked GW network setup as well). See each DNS requests my domU sent > and can see as well answers, which are not received by domU. No strange > ARP traffic. > > I wanted to change the network, thinking about a bogus driver (never > know). My Dom0 has crashed during reboot and I''ve no way to remotely > reboot it (no remote power control and it seems that Magic SysRQ are not > transfered through my Putty/screen connection). > > Will reboot it when at home and continue investigation with another > network card and maybe another operating system to check if bad > behaviour is still there. > > Regards, > JB_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Jan 13, 2011 at 08:18:33PM +0100, Jean Baptiste Favre wrote:> Hello, > My dom0 is back and I performed some more tests. > > I told in my first mail that ping works. Indeed it works, but not always: > # ping -c2 10.0.0.1 > PING 10.0.0.1 (10.0.0.1): 56 data bytes > 64 bytes from 10.0.0.1: seq=0 ttl=64 time=0.846 ms > 64 bytes from 10.0.0.1: seq=1 ttl=64 time=0.824 ms > > --- 10.0.0.1 ping statistics --- > 2 packets transmitted, 2 packets received, 0% packet loss > round-trip min/avg/max = 0.824/0.835/0.846 ms > # ping -c2 -s60 10.0.0.1 > PING 10.0.0.1 (10.0.0.1): 60 data bytes > 68 bytes from 10.0.0.1: seq=0 ttl=64 time=0.819 ms > 68 bytes from 10.0.0.1: seq=1 ttl=64 time=0.807 ms > > --- 10.0.0.1 ping statistics --- > 2 packets transmitted, 2 packets received, 0% packet loss > round-trip min/avg/max = 0.807/0.813/0.819 ms > > Increasing packet size is ok until this one: > # ping -c2 -s85 10.0.0.1 > PING 10.0.0.1 (10.0.0.1): 85 data bytes > 93 bytes from 10.0.0.1: seq=0 ttl=64 time=0.823 ms > 93 bytes from 10.0.0.1: seq=1 ttl=64 time=0.816 ms > > --- 10.0.0.1 ping statistics --- > 2 packets transmitted, 2 packets received, 0% packet loss > round-trip min/avg/max = 0.816/0.819/0.823 ms > root@OpenWrt:/# ping -c2 -s86 10.0.0.1 > PING 10.0.0.1 (10.0.0.1): 86 data bytes > > --- 10.0.0.1 ping statistics --- > 2 packets transmitted, 0 packets received, 100% packet loss > > As you see, packet size seems to be limited in a way. From another > machine on the same LAN I can do ping -s1500 without any problem.One thing that I just thought (which I keep on forgetting to do). You did set ''iommu=soft'' in your Linux guest, right?> > So I think I hit a bug. Either it''s Xen related, or Debian (through > debian kernel version). Now the question is: how can I determine which > part is responsible ?What does tcpdump tell you when you try to send it at -s86? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Le 13/01/2011 21:19, Konrad Rzeszutek Wilk a écrit :> On Thu, Jan 13, 2011 at 08:18:33PM +0100, Jean Baptiste Favre wrote: >> Hello, >> My dom0 is back and I performed some more tests. >> >> I told in my first mail that ping works. Indeed it works, but not always: >> # ping -c2 10.0.0.1 >> PING 10.0.0.1 (10.0.0.1): 56 data bytes >> 64 bytes from 10.0.0.1: seq=0 ttl=64 time=0.846 ms >> 64 bytes from 10.0.0.1: seq=1 ttl=64 time=0.824 ms >> >> --- 10.0.0.1 ping statistics --- >> 2 packets transmitted, 2 packets received, 0% packet loss >> round-trip min/avg/max = 0.824/0.835/0.846 ms >> # ping -c2 -s60 10.0.0.1 >> PING 10.0.0.1 (10.0.0.1): 60 data bytes >> 68 bytes from 10.0.0.1: seq=0 ttl=64 time=0.819 ms >> 68 bytes from 10.0.0.1: seq=1 ttl=64 time=0.807 ms >> >> --- 10.0.0.1 ping statistics --- >> 2 packets transmitted, 2 packets received, 0% packet loss >> round-trip min/avg/max = 0.807/0.813/0.819 ms >> >> Increasing packet size is ok until this one: >> # ping -c2 -s85 10.0.0.1 >> PING 10.0.0.1 (10.0.0.1): 85 data bytes >> 93 bytes from 10.0.0.1: seq=0 ttl=64 time=0.823 ms >> 93 bytes from 10.0.0.1: seq=1 ttl=64 time=0.816 ms >> >> --- 10.0.0.1 ping statistics --- >> 2 packets transmitted, 2 packets received, 0% packet loss >> round-trip min/avg/max = 0.816/0.819/0.823 ms >> root@OpenWrt:/# ping -c2 -s86 10.0.0.1 >> PING 10.0.0.1 (10.0.0.1): 86 data bytes >> >> --- 10.0.0.1 ping statistics --- >> 2 packets transmitted, 0 packets received, 100% packet loss >> >> As you see, packet size seems to be limited in a way. From another >> machine on the same LAN I can do ping -s1500 without any problem. > > One thing that I just thought (which I keep on forgetting to do). > You did set ''iommu=soft'' in your Linux guest, right?Tought I told it in my previou smails. Sorry to missed it: On my dom0: $ cat /proc/cmdline placeholder root=/dev/mapper/system-root ro console=hvc0 earlyprintk=xen nomodeset xen-pciback.permissive xen-pciback.hide=(04:00.0) pci=resource_alignment=04:00.0 quiet Xen hypervisor options are: # xm dmesg (XEN) Xen version 4.0.1 (Debian 4.0.1-1) (waldi@debian.org) (gcc version 4.4.5 20100824 (prerelease) (Debian 4.4.4-11) ) Fri Sep 3 15:38:12 UTC 2010 (XEN) Bootloader: GRUB 1.98+20100804-11 (XEN) Command line: placeholder dom0_mem=256M dom0_max_vcpus=1 dom0_vcpus_pin loglvl=all guest_loglvl=all com1=115200,8n1 console=com1 OpenWRT domU config file is: $ cat /etc/xen/auto/openwrt.cfg kernel = ''/home/domU/wrt/openwrt-x86-xen_domu-vmlinuz'' root = ''/dev/xvda2 rw'' memory = ''256'' vcpus = ''1'' cpus = ''1'' localtime = 0 serial = ''pty'' disk = [''file:/home/domU/wrt/openwrt-x86-xen_domu-combined-ext4.img,xvda,w''] #vif = [ ''bridge=br-wan, mac=00:16:3E:01:00:64, vifname=wrt-100.eth1'' ] on_poweroff = ''destroy'' on_reboot = ''restart'' on_crash = ''restart'' extra = "iommu=soft swiotlb=force console=hvc0 xencons=tty" pci = [ ''04:00.0'' ] name = ''openwrt'' hostname = ''openwrt.clichy.jbfavre.org''>> So I think I hit a bug. Either it''s Xen related, or Debian (through >> debian kernel version). Now the question is: how can I determine which >> part is responsible ? > > What does tcpdump tell you when you try to send it at -s86?I can see echo requests coming in on my gateway, replies going back but replies are never received on the domU. Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Tought I told it in my previou smails. Sorry to missed it:You probably did and I missed it too. Good that you have those options, now: .. snip ..> > What does tcpdump tell you when you try to send it at -s86? > > I can see echo requests coming in on my gateway, replies going back but > replies are never received on the domU.What is the NIC you are using? Is it a Intel one? What does /proc/interrupts look like? Can you send me the full domU output? What does the Xen hypervisor mapping look like (xm debug-keys q, xm debug-keys Q, xm debug-keys i)? What is the motherboard you have? Did you look up to see if there are any errate for the motherboard or the NIC you are using? Do you have the latest firmware for the NIC and the latest BIOS for your motherboard? Is the OpenWRT kernel you are using one that you built yourself or do they package it for you? Can you try using a vaniall built one (2.6.37 vanilla is perfect). Is this a 64-bit kernel or 32-bit? Can you try using a 64-bit one (you should be able to run a 64-bit kernel alongside 32-bit userspace). _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, Le 14/01/2011 15:53, Konrad Rzeszutek Wilk a écrit :>> Tought I told it in my previou smails. Sorry to missed it: > > You probably did and I missed it too. Good that you have those options, now: > > .. snip .. >>> What does tcpdump tell you when you try to send it at -s86? >> >> I can see echo requests coming in on my gateway, replies going back but >> replies are never received on the domU. > > What is the NIC you are using? Is it a Intel one? What does /proc/interrupts look like?The NIC I use is "Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller", not Intel, with sky2 driver. # cat /proc/interrupts CPU0 55: 1393 xen-pirq-pcifront-msi sky2@pci:0000:00:00.0 246: 202 xen-dyn-event blkif 247: 103 xen-dyn-event hvc_console 248: 603 xen-dyn-event pcifront 249: 259 xen-dyn-event xenbus 250: 0 xen-percpu-ipi callfuncsingle0 251: 0 xen-percpu-virq debug0 252: 0 xen-percpu-ipi callfunc0 253: 0 xen-percpu-ipi resched0 254: 0 xen-percpu-ipi spinlock0 255: 246058 xen-percpu-virq timer0 NMI: 0 Non-maskable interrupts LOC: 0 Local timer interrupts SPU: 0 Spurious interrupts PMI: 0 Performance monitoring interrupts IWI: 0 IRQ work interrupts RES: 0 Rescheduling interrupts CAL: 0 Function call interrupts TLB: 0 TLB shootdowns TRM: 0 Thermal event interrupts THR: 0 Threshold APIC interrupts MCE: 0 Machine check exceptions MCP: 0 Machine check polls ERR: 0 MIS: 0> Can you send me the full domU output? What does the Xen hypervisor mapping look likeOutput are attached to this mail.> (xm debug-keys q, xm debug-keys Q, xm debug-keys i)? What is the motherboard you have?dmidecode ... Handle 0x0002, DMI type 2, 15 bytes Base Board Information Manufacturer: ASUSTeK Computer INC. Product Name: M3N18L T-M3N8200 Version: Rev x.xx Serial Number: MS1C85B07000633 Asset Tag: To Be Filled By O.E.M. Features: Board is a hosting board Board is replaceable Location In Chassis: To Be Filled By O.E.M. Chassis Handle: 0x0003 Type: Motherboard Contained Object Handles: 0 ...> Did you look up to see if there are any errate for the motherboard or the NIC you are using?No, since everything was working fine (before I decided to test PCI passthrough).> Do you have the latest firmware for the NIC and the latest BIOS for your motherboard?Not sure. Will check> Is the OpenWRT kernel you are using one that you built yourself or do they package it > for you? Can you try using a vaniall built one (2.6.37 vanilla is perfect).It is packaged by openWRT Builroot. Just have to set the kernel version I want to use. Will try a vanilla kernel as soon as possible.> Is this a 64-bit kernel or 32-bit? Can you try using a 64-bit one (you should be > able to run a 64-bit kernel alongside 32-bit userspace).OK, will try it as well. Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, Has made little tests this week-end. - Can not use full vanilla kernel with OpenWRT. Must apply some patch to make OpenWRT boot and don''t had enough time to figure out which one exactly. Will work on it this week. - Have tested a Debian PV 64bits domU with 2.6.37 kernel from Debian experimental repository. Network is fully working. Will try a 32bits domU this week So, at this point, either because of 64bits, either because of "better" kernel, network with PCI passthrough is working. Still have to try 32bits kernel, as well as filtering OpenWRT patches. Regards, JB Le 15/01/2011 00:29, Jean Baptiste Favre a écrit :> Hello, > > Le 14/01/2011 15:53, Konrad Rzeszutek Wilk a écrit : >>> Tought I told it in my previou smails. Sorry to missed it: >> >> You probably did and I missed it too. Good that you have those options, now: >> >> .. snip .. >>>> What does tcpdump tell you when you try to send it at -s86? >>> >>> I can see echo requests coming in on my gateway, replies going back but >>> replies are never received on the domU. >> >> What is the NIC you are using? Is it a Intel one? What does /proc/interrupts look like? > > The NIC I use is "Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit > Ethernet Controller", not Intel, with sky2 driver. > > # cat /proc/interrupts > CPU0 > 55: 1393 xen-pirq-pcifront-msi sky2@pci:0000:00:00.0 > 246: 202 xen-dyn-event blkif > 247: 103 xen-dyn-event hvc_console > 248: 603 xen-dyn-event pcifront > 249: 259 xen-dyn-event xenbus > 250: 0 xen-percpu-ipi callfuncsingle0 > 251: 0 xen-percpu-virq debug0 > 252: 0 xen-percpu-ipi callfunc0 > 253: 0 xen-percpu-ipi resched0 > 254: 0 xen-percpu-ipi spinlock0 > 255: 246058 xen-percpu-virq timer0 > NMI: 0 Non-maskable interrupts > LOC: 0 Local timer interrupts > SPU: 0 Spurious interrupts > PMI: 0 Performance monitoring interrupts > IWI: 0 IRQ work interrupts > RES: 0 Rescheduling interrupts > CAL: 0 Function call interrupts > TLB: 0 TLB shootdowns > TRM: 0 Thermal event interrupts > THR: 0 Threshold APIC interrupts > MCE: 0 Machine check exceptions > MCP: 0 Machine check polls > ERR: 0 > MIS: 0 > > >> Can you send me the full domU output? What does the Xen hypervisor mapping look like > Output are attached to this mail. > >> (xm debug-keys q, xm debug-keys Q, xm debug-keys i)? What is the motherboard you have? > dmidecode > ... > Handle 0x0002, DMI type 2, 15 bytes > Base Board Information > Manufacturer: ASUSTeK Computer INC. > Product Name: M3N18L T-M3N8200 > Version: Rev x.xx > Serial Number: MS1C85B07000633 > Asset Tag: To Be Filled By O.E.M. > Features: > Board is a hosting board > Board is replaceable > Location In Chassis: To Be Filled By O.E.M. > Chassis Handle: 0x0003 > Type: Motherboard > Contained Object Handles: 0 > ... > >> Did you look up to see if there are any errate for the motherboard or the NIC you are using? > No, since everything was working fine (before I decided to test PCI > passthrough). > >> Do you have the latest firmware for the NIC and the latest BIOS for your motherboard? > Not sure. Will check > >> Is the OpenWRT kernel you are using one that you built yourself or do they package it >> for you? Can you try using a vaniall built one (2.6.37 vanilla is perfect). > It is packaged by openWRT Builroot. Just have to set the kernel version > I want to use. > Will try a vanilla kernel as soon as possible. > >> Is this a 64-bit kernel or 32-bit? Can you try using a 64-bit one (you should be >> able to run a 64-bit kernel alongside 32-bit userspace). > OK, will try it as well. > > Regards, > JB_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello Konrad, Have tested with Debian 32bits domU. Once domU is started, log in via xm console and exec "ping 10.0.0.1" and "ping -s86 10.0.0.1" Here are the results: - 32bits kernel + 32bits Debian Squeeze => ping -s86 FAILS - 64bits kernel + 32bits Debian Squeeze => WORKS - 64bits kernel + 64bits Debian Squeeze => WORKS Used kernels are: 32bit: linux-image-2.6.37-trunk-686-bigmem 64bit: linux-image-2.6.37-trunk-amd64 Will now try to compile openwrt kernel as 64bits and update you. Regards, JB Le 17/01/2011 09:59, Jean Baptiste Favre a écrit :> Hello, > Has made little tests this week-end. > - Can not use full vanilla kernel with OpenWRT. Must apply some patch to > make OpenWRT boot and don''t had enough time to figure out which one > exactly. Will work on it this week. > - Have tested a Debian PV 64bits domU with 2.6.37 kernel from Debian > experimental repository. Network is fully working. Will try a 32bits > domU this week > > So, at this point, either because of 64bits, either because of "better" > kernel, network with PCI passthrough is working. > > Still have to try 32bits kernel, as well as filtering OpenWRT patches. > > Regards, > JB > > > Le 15/01/2011 00:29, Jean Baptiste Favre a écrit : >> Hello, >> >> Le 14/01/2011 15:53, Konrad Rzeszutek Wilk a écrit : >>>> Tought I told it in my previou smails. Sorry to missed it: >>> >>> You probably did and I missed it too. Good that you have those options, now: >>> >>> .. snip .. >>>>> What does tcpdump tell you when you try to send it at -s86? >>>> >>>> I can see echo requests coming in on my gateway, replies going back but >>>> replies are never received on the domU. >>> >>> What is the NIC you are using? Is it a Intel one? What does /proc/interrupts look like? >> >> The NIC I use is "Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit >> Ethernet Controller", not Intel, with sky2 driver. >> >> # cat /proc/interrupts >> CPU0 >> 55: 1393 xen-pirq-pcifront-msi sky2@pci:0000:00:00.0 >> 246: 202 xen-dyn-event blkif >> 247: 103 xen-dyn-event hvc_console >> 248: 603 xen-dyn-event pcifront >> 249: 259 xen-dyn-event xenbus >> 250: 0 xen-percpu-ipi callfuncsingle0 >> 251: 0 xen-percpu-virq debug0 >> 252: 0 xen-percpu-ipi callfunc0 >> 253: 0 xen-percpu-ipi resched0 >> 254: 0 xen-percpu-ipi spinlock0 >> 255: 246058 xen-percpu-virq timer0 >> NMI: 0 Non-maskable interrupts >> LOC: 0 Local timer interrupts >> SPU: 0 Spurious interrupts >> PMI: 0 Performance monitoring interrupts >> IWI: 0 IRQ work interrupts >> RES: 0 Rescheduling interrupts >> CAL: 0 Function call interrupts >> TLB: 0 TLB shootdowns >> TRM: 0 Thermal event interrupts >> THR: 0 Threshold APIC interrupts >> MCE: 0 Machine check exceptions >> MCP: 0 Machine check polls >> ERR: 0 >> MIS: 0 >> >> >>> Can you send me the full domU output? What does the Xen hypervisor mapping look like >> Output are attached to this mail. >> >>> (xm debug-keys q, xm debug-keys Q, xm debug-keys i)? What is the motherboard you have? >> dmidecode >> ... >> Handle 0x0002, DMI type 2, 15 bytes >> Base Board Information >> Manufacturer: ASUSTeK Computer INC. >> Product Name: M3N18L T-M3N8200 >> Version: Rev x.xx >> Serial Number: MS1C85B07000633 >> Asset Tag: To Be Filled By O.E.M. >> Features: >> Board is a hosting board >> Board is replaceable >> Location In Chassis: To Be Filled By O.E.M. >> Chassis Handle: 0x0003 >> Type: Motherboard >> Contained Object Handles: 0 >> ... >> >>> Did you look up to see if there are any errate for the motherboard or the NIC you are using? >> No, since everything was working fine (before I decided to test PCI >> passthrough). >> >>> Do you have the latest firmware for the NIC and the latest BIOS for your motherboard? >> Not sure. Will check >> >>> Is the OpenWRT kernel you are using one that you built yourself or do they package it >>> for you? Can you try using a vaniall built one (2.6.37 vanilla is perfect). >> It is packaged by openWRT Builroot. Just have to set the kernel version >> I want to use. >> Will try a vanilla kernel as soon as possible. >> >>> Is this a 64-bit kernel or 32-bit? Can you try using a 64-bit one (you should be >>> able to run a 64-bit kernel alongside 32-bit userspace). >> OK, will try it as well. >> >> Regards, >> JB > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, Last investigations show that I''ve the latest BIOS version for my motherboard. Do you need more tests, if yes which ones ? This problem annoys me because I did not manage to get OpenWRT working with 64bits kernel. Regards, JB Le 17/01/2011 14:58, Jean Baptiste Favre a écrit :> Hello Konrad, > Have tested with Debian 32bits domU. Once domU is started, log in via xm > console and exec "ping 10.0.0.1" and "ping -s86 10.0.0.1" > Here are the results: > > - 32bits kernel + 32bits Debian Squeeze => ping -s86 FAILS > - 64bits kernel + 32bits Debian Squeeze => WORKS > - 64bits kernel + 64bits Debian Squeeze => WORKS > > Used kernels are: > 32bit: linux-image-2.6.37-trunk-686-bigmem > 64bit: linux-image-2.6.37-trunk-amd64 > > Will now try to compile openwrt kernel as 64bits and update you. > > Regards, > JB > > Le 17/01/2011 09:59, Jean Baptiste Favre a écrit : >> Hello, >> Has made little tests this week-end. >> - Can not use full vanilla kernel with OpenWRT. Must apply some patch to >> make OpenWRT boot and don''t had enough time to figure out which one >> exactly. Will work on it this week. >> - Have tested a Debian PV 64bits domU with 2.6.37 kernel from Debian >> experimental repository. Network is fully working. Will try a 32bits >> domU this week >> >> So, at this point, either because of 64bits, either because of "better" >> kernel, network with PCI passthrough is working. >> >> Still have to try 32bits kernel, as well as filtering OpenWRT patches. >> >> Regards, >> JB >> >> >> Le 15/01/2011 00:29, Jean Baptiste Favre a écrit : >>> Hello, >>> >>> Le 14/01/2011 15:53, Konrad Rzeszutek Wilk a écrit : >>>>> Tought I told it in my previou smails. Sorry to missed it: >>>> >>>> You probably did and I missed it too. Good that you have those options, now: >>>> >>>> .. snip .. >>>>>> What does tcpdump tell you when you try to send it at -s86? >>>>> >>>>> I can see echo requests coming in on my gateway, replies going back but >>>>> replies are never received on the domU. >>>> >>>> What is the NIC you are using? Is it a Intel one? What does /proc/interrupts look like? >>> >>> The NIC I use is "Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit >>> Ethernet Controller", not Intel, with sky2 driver. >>> >>> # cat /proc/interrupts >>> CPU0 >>> 55: 1393 xen-pirq-pcifront-msi sky2@pci:0000:00:00.0 >>> 246: 202 xen-dyn-event blkif >>> 247: 103 xen-dyn-event hvc_console >>> 248: 603 xen-dyn-event pcifront >>> 249: 259 xen-dyn-event xenbus >>> 250: 0 xen-percpu-ipi callfuncsingle0 >>> 251: 0 xen-percpu-virq debug0 >>> 252: 0 xen-percpu-ipi callfunc0 >>> 253: 0 xen-percpu-ipi resched0 >>> 254: 0 xen-percpu-ipi spinlock0 >>> 255: 246058 xen-percpu-virq timer0 >>> NMI: 0 Non-maskable interrupts >>> LOC: 0 Local timer interrupts >>> SPU: 0 Spurious interrupts >>> PMI: 0 Performance monitoring interrupts >>> IWI: 0 IRQ work interrupts >>> RES: 0 Rescheduling interrupts >>> CAL: 0 Function call interrupts >>> TLB: 0 TLB shootdowns >>> TRM: 0 Thermal event interrupts >>> THR: 0 Threshold APIC interrupts >>> MCE: 0 Machine check exceptions >>> MCP: 0 Machine check polls >>> ERR: 0 >>> MIS: 0 >>> >>> >>>> Can you send me the full domU output? What does the Xen hypervisor mapping look like >>> Output are attached to this mail. >>> >>>> (xm debug-keys q, xm debug-keys Q, xm debug-keys i)? What is the motherboard you have? >>> dmidecode >>> ... >>> Handle 0x0002, DMI type 2, 15 bytes >>> Base Board Information >>> Manufacturer: ASUSTeK Computer INC. >>> Product Name: M3N18L T-M3N8200 >>> Version: Rev x.xx >>> Serial Number: MS1C85B07000633 >>> Asset Tag: To Be Filled By O.E.M. >>> Features: >>> Board is a hosting board >>> Board is replaceable >>> Location In Chassis: To Be Filled By O.E.M. >>> Chassis Handle: 0x0003 >>> Type: Motherboard >>> Contained Object Handles: 0 >>> ... >>> >>>> Did you look up to see if there are any errate for the motherboard or the NIC you are using? >>> No, since everything was working fine (before I decided to test PCI >>> passthrough). >>> >>>> Do you have the latest firmware for the NIC and the latest BIOS for your motherboard? >>> Not sure. Will check >>> >>>> Is the OpenWRT kernel you are using one that you built yourself or do they package it >>>> for you? Can you try using a vaniall built one (2.6.37 vanilla is perfect). >>> It is packaged by openWRT Builroot. Just have to set the kernel version >>> I want to use. >>> Will try a vanilla kernel as soon as possible. >>> >>>> Is this a 64-bit kernel or 32-bit? Can you try using a 64-bit one (you should be >>>> able to run a 64-bit kernel alongside 32-bit userspace). >>> OK, will try it as well. >>> >>> Regards, >>> JB_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sat, Jan 22, 2011 at 11:22:59AM +0100, Jean Baptiste Favre wrote:> Hello, > Last investigations show that I''ve the latest BIOS version for my > motherboard. > Do you need more tests, if yes which ones ?I tried it on my 2.6.32.27 (32-bit and 64-bit) and I am not seeing the failures you have. These are the devices I passed in: 00:00.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02) 00:00.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02) 00:00.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02) 00:00.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02) 00:01.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02) 00:02.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07) 00:02.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07) 00:03.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06) Granted the kernel I am using a jeremy/xen/stable-2.6.32.x so not sure how divergent from Debian or Debian Squeeze it is. How are you launching your guest? Do you something like this: kernel="/mnt/lab/2.6.32.27/vmlinuz" ramdisk="/mnt/lab/2.6.32.27/initramfs.cpio.gz" extra="console=hvc0 debug test=crashme iommu=soft" memory=1024 maxmem=2048 vcpus=4 on_crash="preserve" pci= ["00:1d.0","00:1d.1","00:1d.2","00:1d.7","0a:00.1","0000:06:01.1","0000:06:01.0", "09:00.0"] vif = [ ''mac=00:0f:4b:00:00:68, bridge=switch'' ] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello Konrad, Le 27/01/2011 21:27, Konrad Rzeszutek Wilk a écrit :> On Sat, Jan 22, 2011 at 11:22:59AM +0100, Jean Baptiste Favre wrote: >> Hello, >> Last investigations show that I''ve the latest BIOS version for my >> motherboard. >> Do you need more tests, if yes which ones ? > > I tried it on my 2.6.32.27 (32-bit and 64-bit) and I am not seeing > the failures you have. These are the devices I passed in: > > 00:00.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02) > 00:00.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02) > 00:00.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02) > 00:00.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02) > 00:01.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02) > 00:02.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07) > 00:02.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07) > 00:03.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06) > > Granted the kernel I am using a jeremy/xen/stable-2.6.32.x so not sure > how divergent from Debian or Debian Squeeze it is. > > How are you launching your guest? Do you something like this: > > kernel="/mnt/lab/2.6.32.27/vmlinuz" > ramdisk="/mnt/lab/2.6.32.27/initramfs.cpio.gz" > extra="console=hvc0 debug test=crashme iommu=soft" > memory=1024 > maxmem=2048 > vcpus=4 > on_crash="preserve" > pci= ["00:1d.0","00:1d.1","00:1d.2","00:1d.7","0a:00.1","0000:06:01.1","0000:06:01.0", "09:00.0"] > vif = [ ''mac=00:0f:4b:00:00:68, bridge=switch'' ]Here is my domU config file: #################### kernel = ''/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'' ramdisk = ''/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'' builder = ''linux'' memory = ''256'' memory = ''512'' vcpus = ''1'' cpus = ''2'' localtime = 0 serial = ''pty'' disk = [ ''drbd:xps-106,xvda,w'' ] on_poweroff = ''destroy'' on_reboot = ''restart'' on_crash = ''restart'' extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force console=hvc0 xencons=tty" pci = [ ''04:00.0'' ] name = ''xps-106'' hostname = ''xps-106.clichy.jbfavre.org'' #################### As I privately told you, I made the tests you suggested and the result is that problem occurs with 256mb of memory, but is solved with 512mb or more. Basically, what I find surprising is that with 256mb of memory, the max size for incoming packets to be blocked is 128 bytes. Makes me think about an unsigned integer or something like that, but I don''t have enough kernel knowledge to be more precise. Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, I made some more tests today, still with 2.6.37 32bits kernel from Debian experimental, with various memory allocation value. For each test, I make ping on my gateway with various packet size: ping -s15 10.0.0.1 ping -s85 10.0.0.1 ping -s86 10.0.0.1 ping -s150 10.0.0.1 Results bellow: - less than 256mb: works - between 256 and 512mb: ping greater than 85 bytes does not work - more than 512mb: works I''m lost... Regards, JB Le 27/01/2011 22:47, Jean Baptiste Favre a écrit :> Hello Konrad, > > Le 27/01/2011 21:27, Konrad Rzeszutek Wilk a écrit : >> On Sat, Jan 22, 2011 at 11:22:59AM +0100, Jean Baptiste Favre wrote: >>> Hello, >>> Last investigations show that I''ve the latest BIOS version for my >>> motherboard. >>> Do you need more tests, if yes which ones ? >> >> I tried it on my 2.6.32.27 (32-bit and 64-bit) and I am not seeing >> the failures you have. These are the devices I passed in: >> >> 00:00.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02) >> 00:00.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02) >> 00:00.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02) >> 00:00.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02) >> 00:01.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02) >> 00:02.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07) >> 00:02.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07) >> 00:03.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06) >> >> Granted the kernel I am using a jeremy/xen/stable-2.6.32.x so not sure >> how divergent from Debian or Debian Squeeze it is. >> >> How are you launching your guest? Do you something like this: >> >> kernel="/mnt/lab/2.6.32.27/vmlinuz" >> ramdisk="/mnt/lab/2.6.32.27/initramfs.cpio.gz" >> extra="console=hvc0 debug test=crashme iommu=soft" >> memory=1024 >> maxmem=2048 >> vcpus=4 >> on_crash="preserve" >> pci= ["00:1d.0","00:1d.1","00:1d.2","00:1d.7","0a:00.1","0000:06:01.1","0000:06:01.0", "09:00.0"] >> vif = [ ''mac=00:0f:4b:00:00:68, bridge=switch'' ] > > Here is my domU config file: > #################### > kernel = ''/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'' > ramdisk = ''/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'' > builder = ''linux'' > > memory = ''256'' > memory = ''512'' > vcpus = ''1'' > cpus = ''2'' > localtime = 0 > serial = ''pty'' > > disk = [ ''drbd:xps-106,xvda,w'' ] > > on_poweroff = ''destroy'' > on_reboot = ''restart'' > on_crash = ''restart'' > > extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force > console=hvc0 xencons=tty" > > pci = [ ''04:00.0'' ] > > name = ''xps-106'' > hostname = ''xps-106.clichy.jbfavre.org'' > #################### > > As I privately told you, I made the tests you suggested and the result > is that problem occurs with 256mb of memory, but is solved with 512mb or > more. > > Basically, what I find surprising is that with 256mb of memory, the max > size for incoming packets to be blocked is 128 bytes. > Makes me think about an unsigned integer or something like that, but I > don''t have enough kernel knowledge to be more precise. > > Regards, > JB > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 2011-01-28 at 15:47 +0000, Jean Baptiste Favre wrote:> Hello, > I made some more tests today, still with 2.6.37 32bits kernel from > Debian experimental, with various memory allocation value. > > For each test, I make ping on my gateway with various packet size: > ping -s15 10.0.0.1 > ping -s85 10.0.0.1 > ping -s86 10.0.0.1 > ping -s150 10.0.0.1 > > Results bellow: > > - less than 256mb: works > - between 256 and 512mb: ping greater than 85 bytes does not work > - more than 512mb: works > > I''m lost...Me too, this really is the most inexplicable set of symptoms... Does it work correctly with any other guest kernel, e.g. the xen/stable-2.6.32.x branch from xen.git or maybe one of the old-style Xen kernels? The network device in use is one of the Intel NICs below? Any luck just passing through that one device without all the others? Previously you mentioned using a Marvell NIC, so I guess the failure is independent of the NIC type? Your userspace is still OpenWRT in these most recent tests? Is that some sort of busybox based thing? Can you try with e.g. a regular Debian guest userspace to rule out any funnyness from that end? If you restrict dom0 to >256MB but <512MB (using dom0_mem= on hypervisor command line) does the NIC work correctly in non-passedthrough form? Similarly does the kernel running native with mem= cause the failure? Bit of a long shot but are you able to try a 4.0.2-rc hypervisor+tools and/or a 4.1.0-rc setup (not branched yet so still in xen-unstable.hg)? Is the 10.0.0.1 address you are testing against a VM on the same host or some sort of external entity? Ian.> Regards, > JB > > > Le 27/01/2011 22:47, Jean Baptiste Favre a écrit : > > Hello Konrad, > > > > Le 27/01/2011 21:27, Konrad Rzeszutek Wilk a écrit : > >> On Sat, Jan 22, 2011 at 11:22:59AM +0100, Jean Baptiste Favre wrote: > >>> Hello, > >>> Last investigations show that I''ve the latest BIOS version for my > >>> motherboard. > >>> Do you need more tests, if yes which ones ? > >> > >> I tried it on my 2.6.32.27 (32-bit and 64-bit) and I am not seeing > >> the failures you have. These are the devices I passed in: > >> > >> 00:00.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02) > >> 00:00.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02) > >> 00:00.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02) > >> 00:00.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02) > >> 00:01.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02) > >> 00:02.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07) > >> 00:02.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07) > >> 00:03.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06) > >> > >> Granted the kernel I am using a jeremy/xen/stable-2.6.32.x so not sure > >> how divergent from Debian or Debian Squeeze it is. > >> > >> How are you launching your guest? Do you something like this: > >> > >> kernel="/mnt/lab/2.6.32.27/vmlinuz" > >> ramdisk="/mnt/lab/2.6.32.27/initramfs.cpio.gz" > >> extra="console=hvc0 debug test=crashme iommu=soft" > >> memory=1024 > >> maxmem=2048 > >> vcpus=4 > >> on_crash="preserve" > >> pci= ["00:1d.0","00:1d.1","00:1d.2","00:1d.7","0a:00.1","0000:06:01.1","0000:06:01.0", "09:00.0"] > >> vif = [ ''mac=00:0f:4b:00:00:68, bridge=switch'' ] > > > > Here is my domU config file: > > #################### > > kernel = ''/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'' > > ramdisk = ''/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'' > > builder = ''linux'' > > > > memory = ''256'' > > memory = ''512'' > > vcpus = ''1'' > > cpus = ''2'' > > localtime = 0 > > serial = ''pty'' > > > > disk = [ ''drbd:xps-106,xvda,w'' ] > > > > on_poweroff = ''destroy'' > > on_reboot = ''restart'' > > on_crash = ''restart'' > > > > extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force > > console=hvc0 xencons=tty" > > > > pci = [ ''04:00.0'' ] > > > > name = ''xps-106'' > > hostname = ''xps-106.clichy.jbfavre.org'' > > #################### > > > > As I privately told you, I made the tests you suggested and the result > > is that problem occurs with 256mb of memory, but is solved with 512mb or > > more. > > > > Basically, what I find surprising is that with 256mb of memory, the max > > size for incoming packets to be blocked is 128 bytes. > > Makes me think about an unsigned integer or something like that, but I > > don''t have enough kernel knowledge to be more precise. > > > > Regards, > > JB > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello Ian, Le 01/02/2011 12:34, Ian Campbell a écrit :> On Fri, 2011-01-28 at 15:47 +0000, Jean Baptiste Favre wrote: > > Hello, > > I made some more tests today, still with 2.6.37 32bits kernel from > > Debian experimental, with various memory allocation value. > > > > For each test, I make ping on my gateway with various packet size: > > ping -s15 10.0.0.1 > > ping -s85 10.0.0.1 > > ping -s86 10.0.0.1 > > ping -s150 10.0.0.1 > > > > Results bellow: > > > > - less than 256mb: works > > - between 256 and 512mb: ping greater than 85 bytes does not work > > - more than 512mb: works > > > > I''m lost... > > Me too, this really is the most inexplicable set of symptoms... > > Does it work correctly with any other guest kernel, e.g. the > xen/stable-2.6.32.x branch from xen.git or maybe one of the old-style > Xen kernels?I''m compiling 2.6.32 kernel from Jeremy''s GIT repos to check that.> The network device in use is one of the Intel NICs below? Any luck just > passing through that one device without all the others? > > Previously you mentioned using a Marvell NIC, so I guess the failure is > independent of the NIC type?I made all tests with Marvell NIC (driver sky2). I don''t know if the same behaviour occurs with another NIC type. I''ll try with an Intel one (I''ve a dual port Intel NIC, so I could passthrough only one port)> Your userspace is still OpenWRT in these most recent tests? Is that some > sort of busybox based thing? Can you try with e.g. a regular Debian > guest userspace to rule out any funnyness from that end?I made tests with both OpenWRT and Debian Squeeze. Had problems to compile OpenWRT kernel in 64bits :) I tested Debian Squeeze with 2.6.37 kernel from experimental (because of Xen PCI Frontend integration. Not sure it has been backported into 2.6.32). I will test with Jeremy''s 2.6.32 kernel (see above). As a conclusion, last results are from Debian Squeeze with 2.6.37 and that''s why I wrote on debian-kernel maillist :)> If you restrict dom0 to >256MB but <512MB (using dom0_mem= on hypervisor > command line) does the NIC work correctly in non-passedthrough form?My Xen hypervisor commandline is as follow: placeholder dom0_mem=256M dom0_max_vcpus=1 dom0_vcpus_pin loglvl=all guest_loglvl=all com1=115200,8n1 console=com1 Everything works great without passthrough, but my dom0 is 64bits which may explain that (I do have this strange behaviour only with 32bits kernels). I did not tried changing dom0_mem param.> Similarly does the kernel running native with mem= cause the failure?Not sure I understand what you mean here. BTW, I''m preparing a set of automatic tests with different memory values. That will be: * loop for each mem value - set memory in domU configfile - starting domU with 128Mb memory - rc.local will ping my gateway with different packet size, store result in file - halt domU * end of loop * Check results :-/> Bit of a long shot but are you able to try a 4.0.2-rc hypervisor+tools > and/or a 4.1.0-rc setup (not branched yet so still in xen-unstable.hg)?I can eventually try it, but after my looong test list :)> Is the 10.0.0.1 address you are testing against a VM on the same host or > some sort of external entity?It''s my gateway (for the history, WRT54GL with OpenWRT). Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, 2011-02-01 at 12:17 +0000, Jean Baptiste Favre wrote:> > The network device in use is one of the Intel NICs below? Any luck just > > passing through that one device without all the others? > > > > Previously you mentioned using a Marvell NIC, so I guess the failure is > > independent of the NIC type? > I made all tests with Marvell NIC (driver sky2). > I don''t know if the same behaviour occurs with another NIC type. I''ll > try with an Intel one (I''ve a dual port Intel NIC, so I could > passthrough only one port)I was confused -- the quoted list of passthrough devices including Intel NICs I saw was from Konrad not you. It would still be interesting to confirm if the issue is specific to the particular NIC or not.> > Your userspace is still OpenWRT in these most recent tests? Is that some > > sort of busybox based thing? Can you try with e.g. a regular Debian > > guest userspace to rule out any funnyness from that end? > I made tests with both OpenWRT and Debian Squeeze. > Had problems to compile OpenWRT kernel in 64bits :) > I tested Debian Squeeze with 2.6.37 kernel from experimental (because of > Xen PCI Frontend integration. Not sure it has been backported into 2.6.32).The PCI frontend is in the xen.git xen/stable-2.6.32.x branch but not in mainline 2.6.32.y stable branches.> > If you restrict dom0 to >256MB but <512MB (using dom0_mem= on hypervisor > > command line) does the NIC work correctly in non-passedthrough form? > My Xen hypervisor commandline is as follow: > placeholder dom0_mem=256M dom0_max_vcpus=1 dom0_vcpus_pin loglvl=all > guest_loglvl=all com1=115200,8n1 console=com1 > > Everything works great without passthrough, but my dom0 is 64bits which > may explain that (I do have this strange behaviour only with 32bits > kernels).I don''t suppose you could try installing a 32 bit OS in dom0? (e.g. as a skanky hack you might fit something into your existing swap partition ;-))> I did not tried changing dom0_mem param. > > > Similarly does the kernel running native with mem= cause the failure? > Not sure I understand what you mean here.I meant to run the system as a native (non-Xen) system and use the kernels mem= command line parameter to restrict the system to the problematic amounts of memory. Really just trying to verify if something is up specifically due to Xen or not. Probably needs a 32 bit user/kernel to be a useful experiment. What do "ifconfig" or "ethtool -S" show for the device after some failures. Do any of the numbers increment inline with the dropped frames? Perhaps interestingly 85 bytes, plus the ICMP, IP and Ethernet header sizes is something around 128 bytes (depending on options etc). This is pure numerology but I notice that sky2 has a copybreak parameter ("Receive copy threshold") which defaults to 128. I think it would be worth trying 64 and 256. Are you able to see any traffic with tcpdump, either in guest and/or from another host (may require switch configuration to allow it to see the traffic). e.g. do you see the ICMP Echo Request but not the Echo Response or do you see nothing at all etc. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, Le 01/02/2011 14:20, Ian Campbell a écrit :> On Tue, 2011-02-01 at 12:17 +0000, Jean Baptiste Favre wrote: > >> The network device in use is one of the Intel NICs below? Any luck just > >> passing through that one device without all the others? > >> > >> Previously you mentioned using a Marvell NIC, so I guess the failure is > >> independent of the NIC type? > > I made all tests with Marvell NIC (driver sky2). > > I don''t know if the same behaviour occurs with another NIC type. I''ll > > try with an Intel one (I''ve a dual port Intel NIC, so I could > > passthrough only one port) > > I was confused -- the quoted list of passthrough devices including Intel > NICs I saw was from Konrad not you. It would still be interesting to > confirm if the issue is specific to the particular NIC or not. > > >> Your userspace is still OpenWRT in these most recent tests? Is that > some > >> sort of busybox based thing? Can you try with e.g. a regular Debian > >> guest userspace to rule out any funnyness from that end? > > I made tests with both OpenWRT and Debian Squeeze. > > Had problems to compile OpenWRT kernel in 64bits :) > > I tested Debian Squeeze with 2.6.37 kernel from experimental (because of > > Xen PCI Frontend integration. Not sure it has been backported into > 2.6.32). > > The PCI frontend is in the xen.git xen/stable-2.6.32.x branch but not in > mainline 2.6.32.y stable branches.Yes, that''s why I tried with 2.6.37 from experimental, after Pasi''s answer to one of my question on the maillist one month ago or so.> >> If you restrict dom0 to >256MB but <512MB (using dom0_mem= on > hypervisor > >> command line) does the NIC work correctly in non-passedthrough form? > > My Xen hypervisor commandline is as follow: > > placeholder dom0_mem=256M dom0_max_vcpus=1 dom0_vcpus_pin loglvl=all > > guest_loglvl=all com1=115200,8n1 console=com1 > > > > Everything works great without passthrough, but my dom0 is 64bits which > > may explain that (I do have this strange behaviour only with 32bits > > kernels). > > I don''t suppose you could try installing a 32 bit OS in dom0? (e.g. as a > skanky hack you might fit something into your existing swap > partition ;-))I could try, but that would means breaking my test platform. Let''s say I prefer complete other tests before :)> > I did not tried changing dom0_mem param. > > > >> Similarly does the kernel running native with mem= cause the failure? > > Not sure I understand what you mean here. > > I meant to run the system as a native (non-Xen) system and use the > kernels mem= command line parameter to restrict the system to the > problematic amounts of memory. Really just trying to verify if something > is up specifically due to Xen or not. Probably needs a 32 bit > user/kernel to be a useful experiment.Ah ok. But then, running 32bits kernel with 64 bits OS will work ? (I didn''t know 64bits kernel worked with 32bits OS before Konrad told me)> What do "ifconfig" or "ethtool -S" show for the device after some > failures. Do any of the numbers increment inline with the dropped > frames? > > Perhaps interestingly 85 bytes, plus the ICMP, IP and Ethernet header > sizes is something around 128 bytes (depending on options etc). This is > pure numerology but I notice that sky2 has a copybreak parameter > ("Receive copy threshold") which defaults to 128. I think it would be > worth trying 64 and 256.That''s exactly the problem I faced with 256mb memory for my domU. Outgoing packets reached my gateway (tcpdump saw them on it, as well as replies) but incoming packets greater than 128bits were blocked somewhere between Xen hypervisor and domU user space (where I was listening traffic with tcpdump). I didn''t notice the copybreak from sky2. Where can I change it ? It could be exactly what we''re looking for from the beginning, couldn''t it ? How can I change it ?> Are you able to see any traffic with tcpdump, either in guest and/or > from another host (may require switch configuration to allow it to see > the traffic). e.g. do you see the ICMP Echo Request but not the Echo > Response or do you see nothing at all etc.tcpdump tests have been done from my gateway and from domU. On the GW: can see all incoming packets (whatever could be the size) and send all replies. On the DomU: can see all outgoing packets but only incoming ones shorter than 128bytes (global size, means "ping -s85" and less) Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, 2011-02-01 at 14:12 +0000, Jean Baptiste Favre wrote:> Le 01/02/2011 14:20, Ian Campbell a écrit :> > >> If you restrict dom0 to >256MB but <512MB (using dom0_mem= on > > hypervisor > > >> command line) does the NIC work correctly in non-passedthrough form? > > > My Xen hypervisor commandline is as follow: > > > placeholder dom0_mem=256M dom0_max_vcpus=1 dom0_vcpus_pin loglvl=all > > > guest_loglvl=all com1=115200,8n1 console=com1 > > > > > > Everything works great without passthrough, but my dom0 is 64bits which > > > may explain that (I do have this strange behaviour only with 32bits > > > kernels). > > > > I don''t suppose you could try installing a 32 bit OS in dom0? (e.g. as a > > skanky hack you might fit something into your existing swap > > partition ;-)) > I could try, but that would means breaking my test platform. Let''s say I > prefer complete other tests before :)Sure, no problem.> > > I did not tried changing dom0_mem param. > > > > > >> Similarly does the kernel running native with mem= cause the failure? > > > Not sure I understand what you mean here. > > > > I meant to run the system as a native (non-Xen) system and use the > > kernels mem= command line parameter to restrict the system to the > > problematic amounts of memory. Really just trying to verify if something > > is up specifically due to Xen or not. Probably needs a 32 bit > > user/kernel to be a useful experiment. > Ah ok. But then, running 32bits kernel with 64 bits OS will work ? (I > didn''t know 64bits kernel worked with 32bits OS before Konrad told me)32 user on 64 kernel works, 64 user on 32 kernel does not so this will tie in with the 32 bit test above too.> > What do "ifconfig" or "ethtool -S" show for the device after some > > failures. Do any of the numbers increment inline with the dropped > > frames? > > > > Perhaps interestingly 85 bytes, plus the ICMP, IP and Ethernet header > > sizes is something around 128 bytes (depending on options etc). This is > > pure numerology but I notice that sky2 has a copybreak parameter > > ("Receive copy threshold") which defaults to 128. I think it would be > > worth trying 64 and 256. > That''s exactly the problem I faced with 256mb memory for my domU. > Outgoing packets reached my gateway (tcpdump saw them on it, as well as > replies) but incoming packets greater than 128bits were blocked > somewhere between Xen hypervisor and domU user space (where I was > listening traffic with tcpdump). > > I didn''t notice the copybreak from sky2. Where can I change it ? It > could be exactly what we''re looking for from the beginning, couldn''t it ? > How can I change it ?Assuming the driver is modular: "modprobe sky2 copybreak=<N>" Depending on your distro there will be somewhere in /etc you can add this. e.g. on Debian you can create a file in /etc/modprobe.d/ containing "option sky copybreak=<N>" other distros use /etc/modprobe.conf etc.> > Are you able to see any traffic with tcpdump, either in guest and/or > > from another host (may require switch configuration to allow it to see > > the traffic). e.g. do you see the ICMP Echo Request but not the Echo > > Response or do you see nothing at all etc. > tcpdump tests have been done from my gateway and from domU. > On the GW: can see all incoming packets (whatever could be the size) and > send all replies. > On the DomU: can see all outgoing packets but only incoming ones shorter > than 128bytes (global size, means "ping -s85" and less)OK, so it is the sky2 receive path which is at fault, that''s very useful info. It corresponds with the copybreak theory too. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Le 01/02/2011 15:18, Ian Campbell a écrit :> On Tue, 2011-02-01 at 14:12 +0000, Jean Baptiste Favre wrote: > > > Le 01/02/2011 14:20, Ian Campbell a écrit : > > >>>> If you restrict dom0 to >256MB but <512MB (using dom0_mem= on > >> hypervisor > >>>> command line) does the NIC work correctly in non-passedthrough form? > >>> My Xen hypervisor commandline is as follow: > >>> placeholder dom0_mem=256M dom0_max_vcpus=1 dom0_vcpus_pin loglvl=all > >>> guest_loglvl=all com1=115200,8n1 console=com1 > >>> > >>> Everything works great without passthrough, but my dom0 is 64bits > which > >>> may explain that (I do have this strange behaviour only with 32bits > >>> kernels). > >> > >> I don''t suppose you could try installing a 32 bit OS in dom0? (e.g. > as a > >> skanky hack you might fit something into your existing swap > >> partition ;-)) > > I could try, but that would means breaking my test platform. Let''s say I > > prefer complete other tests before :) > > Sure, no problem. > > >>> I did not tried changing dom0_mem param. > >>> > >>>> Similarly does the kernel running native with mem= cause the failure? > >>> Not sure I understand what you mean here. > >> > >> I meant to run the system as a native (non-Xen) system and use the > >> kernels mem= command line parameter to restrict the system to the > >> problematic amounts of memory. Really just trying to verify if > something > >> is up specifically due to Xen or not. Probably needs a 32 bit > >> user/kernel to be a useful experiment. > > Ah ok. But then, running 32bits kernel with 64 bits OS will work ? (I > > didn''t know 64bits kernel worked with 32bits OS before Konrad told me) > > 32 user on 64 kernel works, 64 user on 32 kernel does not so this will > tie in with the 32 bit test above too. > > >> What do "ifconfig" or "ethtool -S" show for the device after some > >> failures. Do any of the numbers increment inline with the dropped > >> frames? > >> > >> Perhaps interestingly 85 bytes, plus the ICMP, IP and Ethernet header > >> sizes is something around 128 bytes (depending on options etc). This is > >> pure numerology but I notice that sky2 has a copybreak parameter > >> ("Receive copy threshold") which defaults to 128. I think it would be > >> worth trying 64 and 256. > > That''s exactly the problem I faced with 256mb memory for my domU. > > Outgoing packets reached my gateway (tcpdump saw them on it, as well as > > replies) but incoming packets greater than 128bits were blocked > > somewhere between Xen hypervisor and domU user space (where I was > > listening traffic with tcpdump). > > > > I didn''t notice the copybreak from sky2. Where can I change it ? It > > could be exactly what we''re looking for from the beginning, couldn''t > it ? > > How can I change it ? > > Assuming the driver is modular: > "modprobe sky2 copybreak=<N>" > > Depending on your distro there will be somewhere in /etc you can add > this. e.g. on Debian you can create a file in /etc/modprobe.d/ > containing "option sky copybreak=<N>" other distros > use /etc/modprobe.conf etc.OK I see but it doesn''t seems to have any effect. I tried "option sky copybreak=0" to get all packet copied with no change. But I have to say that I''m a bit confused: as I run a PV domU, kernel and initrd are provided by dom0. So basically, I had no module related binaries installed. After installation, I tried to remove module and reload it with different configuration without changes. Is there any way to provide this sort of option in kernel commandline so that it ''ll be taken into account even in initrd ?> >> Are you able to see any traffic with tcpdump, either in guest and/or > >> from another host (may require switch configuration to allow it to see > >> the traffic). e.g. do you see the ICMP Echo Request but not the Echo > >> Response or do you see nothing at all etc. > > tcpdump tests have been done from my gateway and from domU. > > On the GW: can see all incoming packets (whatever could be the size) and > > send all replies. > > On the DomU: can see all outgoing packets but only incoming ones shorter > > than 128bytes (global size, means "ping -s85" and less) > > OK, so it is the sky2 receive path which is at fault, that''s very useful > info. It corresponds with the copybreak theory too.And that makes me happy, because I began to desperate :) JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, I think I''ll have to read another time "man modprobe" :) See bellow, I''ve good news Le 01/02/2011 16:14, Jean Baptiste Favre a écrit :> Le 01/02/2011 15:18, Ian Campbell a écrit : > > On Tue, 2011-02-01 at 14:12 +0000, Jean Baptiste Favre wrote: > > > >> Le 01/02/2011 14:20, Ian Campbell a écrit : > > > >>>>> If you restrict dom0 to >256MB but <512MB (using dom0_mem= on > >>> hypervisor > >>>>> command line) does the NIC work correctly in non-passedthrough form? > >>>> My Xen hypervisor commandline is as follow: > >>>> placeholder dom0_mem=256M dom0_max_vcpus=1 dom0_vcpus_pin loglvl=all > >>>> guest_loglvl=all com1=115200,8n1 console=com1 > >>>> > >>>> Everything works great without passthrough, but my dom0 is 64bits > > which > >>>> may explain that (I do have this strange behaviour only with 32bits > >>>> kernels). > >>> > >>> I don''t suppose you could try installing a 32 bit OS in dom0? (e.g. > > as a > >>> skanky hack you might fit something into your existing swap > >>> partition ;-)) > >> I could try, but that would means breaking my test platform. Let''s > say I > >> prefer complete other tests before :) > > > > Sure, no problem. > > > >>>> I did not tried changing dom0_mem param. > >>>> > >>>>> Similarly does the kernel running native with mem= cause the > failure? > >>>> Not sure I understand what you mean here. > >>> > >>> I meant to run the system as a native (non-Xen) system and use the > >>> kernels mem= command line parameter to restrict the system to the > >>> problematic amounts of memory. Really just trying to verify if > > something > >>> is up specifically due to Xen or not. Probably needs a 32 bit > >>> user/kernel to be a useful experiment. > >> Ah ok. But then, running 32bits kernel with 64 bits OS will work ? (I > >> didn''t know 64bits kernel worked with 32bits OS before Konrad told me) > > > > 32 user on 64 kernel works, 64 user on 32 kernel does not so this will > > tie in with the 32 bit test above too. > > > >>> What do "ifconfig" or "ethtool -S" show for the device after some > >>> failures. Do any of the numbers increment inline with the dropped > >>> frames? > >>> > >>> Perhaps interestingly 85 bytes, plus the ICMP, IP and Ethernet header > >>> sizes is something around 128 bytes (depending on options etc). > This is > >>> pure numerology but I notice that sky2 has a copybreak parameter > >>> ("Receive copy threshold") which defaults to 128. I think it would be > >>> worth trying 64 and 256. > >> That''s exactly the problem I faced with 256mb memory for my domU. > >> Outgoing packets reached my gateway (tcpdump saw them on it, as well as > >> replies) but incoming packets greater than 128bits were blocked > >> somewhere between Xen hypervisor and domU user space (where I was > >> listening traffic with tcpdump). > >> > >> I didn''t notice the copybreak from sky2. Where can I change it ? It > >> could be exactly what we''re looking for from the beginning, couldn''t > > it ? > >> How can I change it ? > > > > Assuming the driver is modular: > > "modprobe sky2 copybreak=<N>" > > > > Depending on your distro there will be somewhere in /etc you can add > > this. e.g. on Debian you can create a file in /etc/modprobe.d/ > > containing "option sky copybreak=<N>" other distros > > use /etc/modprobe.conf etc. > OK I see but it doesn''t seems to have any effect. > I tried "option sky copybreak=0" to get all packet copied with no change. > > But I have to say that I''m a bit confused: as I run a PV domU, kernel > and initrd are provided by dom0. > So basically, I had no module related binaries installed. After > installation, I tried to remove module and reload it with different > configuration without changes. > Is there any way to provide this sort of option in kernel commandline so > that it ''ll be taken into account even in initrd ? > > >>> Are you able to see any traffic with tcpdump, either in guest and/or > >>> from another host (may require switch configuration to allow it to see > >>> the traffic). e.g. do you see the ICMP Echo Request but not the Echo > >>> Response or do you see nothing at all etc. > >> tcpdump tests have been done from my gateway and from domU. > >> On the GW: can see all incoming packets (whatever could be the > size) and > >> send all replies. > >> On the DomU: can see all outgoing packets but only incoming ones > shorter > >> than 128bytes (global size, means "ping -s85" and less) > > > > OK, so it is the sky2 receive path which is at fault, that''s very useful > > info. It corresponds with the copybreak theory too.OK, just found it: after domU boot: - log in - ping -s 86 10.0.0.1 (fails) - rmmod sky2 - modprobe sky2 copybreak=0 (no packet copied) - ping -s 86 10.0.0.1 (works) So it''s clearly related to that option. Now the question: what am I supposed to do ? It seems that adding this options in /etc/modprobe.d/sky2.conf does not work, neither using /etc/modules Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, 2011-02-01 at 15:14 +0000, Jean Baptiste Favre wrote:> Le 01/02/2011 15:18, Ian Campbell a écrit :> > Assuming the driver is modular: > > "modprobe sky2 copybreak=<N>" > > > > Depending on your distro there will be somewhere in /etc you can add > > this. e.g. on Debian you can create a file in /etc/modprobe.d/ > > containing "option sky copybreak=<N>" other distros > > use /etc/modprobe.conf etc. > OK I see but it doesn''t seems to have any effect. > I tried "option sky copybreak=0" to get all packet copied with no change.The driver is called sky2 not sky so this won''t have done anything. I typo''d it above, sorry.> But I have to say that I''m a bit confused: as I run a PV domU, kernel > and initrd are provided by dom0. > So basically, I had no module related binaries installed. After > installation, I tried to remove module and reload it with different > configuration without changes. > Is there any way to provide this sort of option in kernel commandline so > that it ''ll be taken into account even in initrd ?It depends on your distro and/or initrd tool, I don''t know a generic answer. If the driver is statically compiled and not modular at all you can do <module>.<param>=<value> on the kernel command line, e.g. "sky2.copybreak=256". There doesn''t appear to be any way to find out what copybreak the driver actually uses, short of hacking something into the driver itself. e.g. diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c index 7d85a38..786b8c6 100644 --- a/drivers/net/sky2.c +++ b/drivers/net/sky2.c @@ -87,7 +87,7 @@ module_param(debug, int, 0); MODULE_PARM_DESC(debug, "Debug level (0=none,...,16=all)"); static int copybreak __read_mostly = 128; -module_param(copybreak, int, 0); +module_param(copybreak, int, 0444); MODULE_PARM_DESC(copybreak, "Receive copy threshold"); static int disable_msi = 0; Allow you to see the current active value in /sys/module/sky2/parameters/copybreak. If you are going to do that you might as well just change the constant above though ;-) Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Le 01/02/2011 16:38, Ian Campbell a écrit :> On Tue, 2011-02-01 at 15:14 +0000, Jean Baptiste Favre wrote: > > Le 01/02/2011 15:18, Ian Campbell a écrit : > > >> Assuming the driver is modular: > >> "modprobe sky2 copybreak=<N>" > >> > >> Depending on your distro there will be somewhere in /etc you can add > >> this. e.g. on Debian you can create a file in /etc/modprobe.d/ > >> containing "option sky copybreak=<N>" other distros > >> use /etc/modprobe.conf etc. > > OK I see but it doesn''t seems to have any effect. > > I tried "option sky copybreak=0" to get all packet copied with no > change. > > The driver is called sky2 not sky so this won''t have done anything. I > typo''d it above, sorry.My bad, I fixed your typo during tests, but I just copied it while writing the mail. So basically, using: # cat /etc/modprobe.d/sky2.conf option sky2 copybreak=0 does not help. I have to remove module and load it with right option. BTW it''s not a real issue for me, I can add it in init script :)> > But I have to say that I''m a bit confused: as I run a PV domU, kernel > > and initrd are provided by dom0. > > So basically, I had no module related binaries installed. After > > installation, I tried to remove module and reload it with different > > configuration without changes. > > Is there any way to provide this sort of option in kernel commandline so > > that it ''ll be taken into account even in initrd ? > > It depends on your distro and/or initrd tool, I don''t know a generic > answer.Both are Debian ones. I''ve tried to compile kernel from Jeremy''s git repo, but face some problems: it''s been a long time since my last kernel compilation and I did not find all configuration options I need :-/> If the driver is statically compiled and not modular at all you can do > <module>.<param>=<value> on the kernel command line, e.g. > "sky2.copybreak=256".I may use this way because, as a firewall, I prefer using static kernels :)> There doesn''t appear to be any way to find out what copybreak the driver > actually uses, short of hacking something into the driver itself. e.g. > > diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c > index 7d85a38..786b8c6 100644 > --- a/drivers/net/sky2.c > +++ b/drivers/net/sky2.c > @@ -87,7 +87,7 @@ module_param(debug, int, 0); > MODULE_PARM_DESC(debug, "Debug level (0=none,...,16=all)"); > > static int copybreak __read_mostly = 128; > -module_param(copybreak, int, 0); > +module_param(copybreak, int, 0444); > MODULE_PARM_DESC(copybreak, "Receive copy threshold"); > > static int disable_msi = 0; > > Allow you to see the current active value > in /sys/module/sky2/parameters/copybreak. If you are going to do that > you might as well just change the constant above though ;-)Will try this as well Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, 2011-02-01 at 15:38 +0000, Jean Baptiste Favre wrote:> OK, just found it: > after domU boot: > - log in > - ping -s 86 10.0.0.1 (fails) > - rmmod sky2 > - modprobe sky2 copybreak=0 (no packet copied) > - ping -s 86 10.0.0.1 (works) > > So it''s clearly related to that option.Awesome!> Now the question: what am I supposed to do ?I think the next step is to try and reproduce on native 32 bit, with RAM artificially limited via the mem= kernel command in option, this will let us determine if this is a generic issue or is somehow Xen specific. The main difference caused by the copybreak option is that for larger frames (i.e. always when copybreak==0) it hits a code path which uses pci_map_single and pci_map_page to access the received data. When len < copybreak it takes a path which uses pci_dma_sync_single_for_cpu, so it seems like the later path is broken somehow. If the issue does turn out to be related to Xen then I think that points to the swiotlb code. I assume you are not seeing "rx mapping error" in your domU dmesg? Did you post a full guest console log at some point? Comparing the logs for the 256MB, 398MB and 512MB guest RAM case might be useful. Konrad, is there some way to force swiotlb use even for native to make the native test cases more relevant? Or is there some other direction we should try first?> It seems that adding this options in /etc/modprobe.d/sky2.conf does not > work, neither using /etc/modulesThat will be a userspace issue in your distro, perhaps with openwrt they are not expected to work that way. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Feb 01, 2011 at 04:23:09PM +0000, Ian Campbell wrote:> On Tue, 2011-02-01 at 15:38 +0000, Jean Baptiste Favre wrote: > > OK, just found it: > > after domU boot: > > - log in > > - ping -s 86 10.0.0.1 (fails) > > - rmmod sky2 > > - modprobe sky2 copybreak=0 (no packet copied) > > - ping -s 86 10.0.0.1 (works) > > > > So it''s clearly related to that option. > > Awesome! > > > Now the question: what am I supposed to do ? > > I think the next step is to try and reproduce on native 32 bit, with RAM > artificially limited via the mem= kernel command in option, this will > let us determine if this is a generic issue or is somehow Xen specific. > > The main difference caused by the copybreak option is that for larger > frames (i.e. always when copybreak==0) it hits a code path which uses > pci_map_single and pci_map_page to access the received data. When len < > copybreak it takes a path which uses pci_dma_sync_single_for_cpu, so it > seems like the later path is broken somehow. If the issue does turn outIt could also have gotten the direction reverted (the 3c5XX code had it wrong at some point so..). Might make sense to compile the kernel with CONFIG_DMA_API_DEBUG which is good at detecting these issues.> to be related to Xen then I think that points to the swiotlb code.> > I assume you are not seeing "rx mapping error" in your domU dmesg? Did > you post a full guest console log at some point? Comparing the logs for > the 256MB, 398MB and 512MB guest RAM case might be useful. > > Konrad, is there some way to force swiotlb use even for native to make > the native test cases more relevant? Or is there some other direction weswiotlb=force will do it.> should try first?<shakes his head> I think we have a good lead.> > > It seems that adding this options in /etc/modprobe.d/sky2.conf does not > > work, neither using /etc/modulesYou could just pass as Linux kernel parameter ''sky2.copybreak=0''. But I am not sure if that stays if the ''sky2'' driver is compiled as module.> > That will be a userspace issue in your distro, perhaps with openwrt they > are not expected to work that way. > > Ian. >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Le 01/02/2011 17:23, Ian Campbell a écrit :> On Tue, 2011-02-01 at 15:38 +0000, Jean Baptiste Favre wrote: >> OK, just found it: >> after domU boot: >> - log in >> - ping -s 86 10.0.0.1 (fails) >> - rmmod sky2 >> - modprobe sky2 copybreak=0 (no packet copied) >> - ping -s 86 10.0.0.1 (works) >> >> So it''s clearly related to that option. > > Awesome! > >> Now the question: what am I supposed to do ? > > I think the next step is to try and reproduce on native 32 bit, with RAM > artificially limited via the mem= kernel command in option, this will > let us determine if this is a generic issue or is somehow Xen specific.OK, so I''ll install a 32bits Debian Squeeze with 2.6.37 32bits kernel to check it.> The main difference caused by the copybreak option is that for larger > frames (i.e. always when copybreak==0) it hits a code path which uses > pci_map_single and pci_map_page to access the received data. When len < > copybreak it takes a path which uses pci_dma_sync_single_for_cpu, so it > seems like the later path is broken somehow. If the issue does turn out > to be related to Xen then I think that points to the swiotlb code. > > I assume you are not seeing "rx mapping error" in your domU dmesg? Did > you post a full guest console log at some point? Comparing the logs for > the 256MB, 398MB and 512MB guest RAM case might be useful.No sure I''ve ever posted that logs. But I can redo my tests :)> Konrad, is there some way to force swiotlb use even for native to make > the native test cases more relevant? Or is there some other direction we > should try first? > >> It seems that adding this options in /etc/modprobe.d/sky2.conf does not >> work, neither using /etc/modules > > That will be a userspace issue in your distro, perhaps with openwrt they > are not expected to work that way.All tests were done with Debian. As the module is included in initramfs and not on domU, since I''ve no kernel installed in the domU, module is loaded before system FS, so /etc/modprobe.d/sky2.conf is not taken into account. I think that the best solution here is to compile kernel with static sky2 driver inclusion and use kernel commandline to pass sky2 copybreak option. Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Le 01/02/2011 20:37, Konrad Rzeszutek Wilk a écrit :> On Tue, Feb 01, 2011 at 04:23:09PM +0000, Ian Campbell wrote: >> On Tue, 2011-02-01 at 15:38 +0000, Jean Baptiste Favre wrote: >>> OK, just found it: >>> after domU boot: >>> - log in >>> - ping -s 86 10.0.0.1 (fails) >>> - rmmod sky2 >>> - modprobe sky2 copybreak=0 (no packet copied) >>> - ping -s 86 10.0.0.1 (works) >>> >>> So it''s clearly related to that option. >> >> Awesome! >> >>> Now the question: what am I supposed to do ? >> >> I think the next step is to try and reproduce on native 32 bit, with RAM >> artificially limited via the mem= kernel command in option, this will >> let us determine if this is a generic issue or is somehow Xen specific. >> >> The main difference caused by the copybreak option is that for larger >> frames (i.e. always when copybreak==0) it hits a code path which uses >> pci_map_single and pci_map_page to access the received data. When len < >> copybreak it takes a path which uses pci_dma_sync_single_for_cpu, so it >> seems like the later path is broken somehow. If the issue does turn out > > It could also have gotten the direction reverted (the 3c5XX code had it > wrong at some point so..). Might make sense to compile the kernel with > CONFIG_DMA_API_DEBUG which is good at detecting these issues. > >> to be related to Xen then I think that points to the swiotlb code. > >> >> I assume you are not seeing "rx mapping error" in your domU dmesg? Did >> you post a full guest console log at some point? Comparing the logs for >> the 256MB, 398MB and 512MB guest RAM case might be useful. >> >> Konrad, is there some way to force swiotlb use even for native to make >> the native test cases more relevant? Or is there some other direction we > > swiotlb=force will do it.I already use this option as specified on wiki: http://wiki.xen.org/xenwiki/XenPCIpassthrough>> should try first? > > <shakes his head> I think we have a good lead.So let''s go for testing that way :)>>> It seems that adding this options in /etc/modprobe.d/sky2.conf does not >>> work, neither using /etc/modules > > You could just pass as Linux kernel parameter ''sky2.copybreak=0''. But I am > not sure if that stays if the ''sky2'' driver is compiled as module.According to Ian, this will work only with statically compiled driver. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Le 01/02/2011 20:37, Konrad Rzeszutek Wilk a écrit :> On Tue, Feb 01, 2011 at 04:23:09PM +0000, Ian Campbell wrote: >> On Tue, 2011-02-01 at 15:38 +0000, Jean Baptiste Favre wrote: >>> OK, just found it: >>> after domU boot: >>> - log in >>> - ping -s 86 10.0.0.1 (fails) >>> - rmmod sky2 >>> - modprobe sky2 copybreak=0 (no packet copied) >>> - ping -s 86 10.0.0.1 (works) >>> >>> So it''s clearly related to that option. >> >> Awesome! >> >>> Now the question: what am I supposed to do ? >> >> I think the next step is to try and reproduce on native 32 bit, with RAM >> artificially limited via the mem= kernel command in option, this will >> let us determine if this is a generic issue or is somehow Xen specific. >> >> The main difference caused by the copybreak option is that for larger >> frames (i.e. always when copybreak==0) it hits a code path which uses >> pci_map_single and pci_map_page to access the received data. When len < >> copybreak it takes a path which uses pci_dma_sync_single_for_cpu, so it >> seems like the later path is broken somehow. If the issue does turn out > > It could also have gotten the direction reverted (the 3c5XX code had it > wrong at some point so..). Might make sense to compile the kernel with > CONFIG_DMA_API_DEBUG which is good at detecting these issues. > >> to be related to Xen then I think that points to the swiotlb code. > >> >> I assume you are not seeing "rx mapping error" in your domU dmesg? Did >> you post a full guest console log at some point? Comparing the logs for >> the 256MB, 398MB and 512MB guest RAM case might be useful. >> >> Konrad, is there some way to force swiotlb use even for native to make >> the native test cases more relevant? Or is there some other direction we > > swiotlb=force will do it.OK, just performed native test. Installed 32bits Debian Squeeze, add 2.6.37 32bits kernel from experimental, setup grub with following options: "mem=256M swiotlb=force" Ping tests work whatever can be packet size. If I understood well what you explain to me, it''s now clear that the problem is somewhat Xen related, isn''t it ? I''ll be able to continue any tests you want tomorrow. Do you still need me to compile domU kernel with "CONFIG_DMA_API_DEBUG" enabled ? If so, what tools will I need to get debug informations ? Now, it''s time to sleep here in France :) (or I''ll get killed by my girl friend :D ) Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, 2011-02-01 at 22:06 +0000, Jean Baptiste Favre wrote:> Le 01/02/2011 20:37, Konrad Rzeszutek Wilk a écrit : > > On Tue, Feb 01, 2011 at 04:23:09PM +0000, Ian Campbell wrote:> >> Konrad, is there some way to force swiotlb use even for native to make > >> the native test cases more relevant? Or is there some other direction we > > > > swiotlb=force will do it. > I already use this option as specified on wiki: > http://wiki.xen.org/xenwiki/XenPCIpassthroughYes, we were just confirming that you should also use this option for your native tests.> > >> should try first? > > > > <shakes his head> I think we have a good lead. > So let''s go for testing that way :) > > >>> It seems that adding this options in /etc/modprobe.d/sky2.conf does not > >>> work, neither using /etc/modules > > > > You could just pass as Linux kernel parameter ''sky2.copybreak=0''. But I am > > not sure if that stays if the ''sky2'' driver is compiled as module. > According to Ian, this will work only with statically compiled driver.Well, I''m only sure it works for a statically compiled driver. I don''t know about modules -- depends on whether modprobe groks /proc/cmdline or something, I suspect it does not though. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, 2011-02-01 at 22:04 +0000, Jean Baptiste Favre wrote:> Le 01/02/2011 17:23, Ian Campbell a écrit :> > I assume you are not seeing "rx mapping error" in your domU dmesg? Did > > you post a full guest console log at some point? Comparing the logs for > > the 256MB, 398MB and 512MB guest RAM case might be useful. > No sure I''ve ever posted that logs. But I can redo my tests :)yes, please do that. Please can you also collect and post the information from ifconfig and ethtool -S which I asked for earlier. Can you confirm that on the domU tcpdump shows no incoming frames at all, including no corrupted or strange frames. It has been suggested that the frames could be being received ok but contain garbage (e.g. due to swiotlb not syncing the memory properly) and hence are not ICMP Responses but appear as some random frame type. Please can you use "tcpdump -w" on both the gateway/peer side and the affected domU to capture a short session while the failing ping is running and post (or link to if large) the resulting pcap files. On the peer side you can use "ether host <domU-MAC>" (plus optionally "or ether broadcast") on the tcpdump command line to limit the capture to just the one peer and reduce the size of the capture. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, 2011-02-01 at 23:01 +0000, Jean Baptiste Favre wrote:> Le 01/02/2011 20:37, Konrad Rzeszutek Wilk a écrit : > > On Tue, Feb 01, 2011 at 04:23:09PM +0000, Ian Campbell wrote: > >> On Tue, 2011-02-01 at 15:38 +0000, Jean Baptiste Favre wrote: > >>> OK, just found it: > >>> after domU boot: > >>> - log in > >>> - ping -s 86 10.0.0.1 (fails) > >>> - rmmod sky2 > >>> - modprobe sky2 copybreak=0 (no packet copied) > >>> - ping -s 86 10.0.0.1 (works) > >>> > >>> So it''s clearly related to that option. > >> > >> Awesome! > >> > >>> Now the question: what am I supposed to do ? > >> > >> I think the next step is to try and reproduce on native 32 bit, with RAM > >> artificially limited via the mem= kernel command in option, this will > >> let us determine if this is a generic issue or is somehow Xen specific. > >> > >> The main difference caused by the copybreak option is that for larger > >> frames (i.e. always when copybreak==0) it hits a code path which uses > >> pci_map_single and pci_map_page to access the received data. When len < > >> copybreak it takes a path which uses pci_dma_sync_single_for_cpu, so it > >> seems like the later path is broken somehow. If the issue does turn out > > > > It could also have gotten the direction reverted (the 3c5XX code had it > > wrong at some point so..). Might make sense to compile the kernel with > > CONFIG_DMA_API_DEBUG which is good at detecting these issues. > > > >> to be related to Xen then I think that points to the swiotlb code. > > > >> > >> I assume you are not seeing "rx mapping error" in your domU dmesg? Did > >> you post a full guest console log at some point? Comparing the logs for > >> the 256MB, 398MB and 512MB guest RAM case might be useful. > >> > >> Konrad, is there some way to force swiotlb use even for native to make > >> the native test cases more relevant? Or is there some other direction we > > > > swiotlb=force will do it. > OK, just performed native test. > Installed 32bits Debian Squeeze, add 2.6.37 32bits kernel from > experimental, setup grub with following options: > "mem=256M swiotlb=force" > > Ping tests work whatever can be packet size. > > If I understood well what you explain to me, it''s now clear that the > problem is somewhat Xen related, isn''t it ?It certainly seems that way. I''m not 100% sure that swiotlb=force will have actually made the driver use the swiotlb, it may just have forced the swiotlb to be allocated. Konrad?> I''ll be able to continue any tests you want tomorrow. > Do you still need me to compile domU kernel with "CONFIG_DMA_API_DEBUG" > enabled ?Yes please.> If so, what tools will I need to get debug informations ?Uh, Konrad? Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello Ian, My domU config file: # cat /cluster/xen/xps-106.cfg kernel = ''/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'' ramdisk = ''/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'' builder = ''linux'' memory = ''398'' vcpus = ''1'' cpus = ''2'' localtime = 0 serial = ''pty'' boot = ''cdn'' disk = [ ''drbd:xps-106,xvda,w'' ] on_poweroff = ''destroy'' on_reboot = ''restart'' on_crash = ''restart'' extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force console=hvc0 xencons=tty" pci = [ ''04:00.0'' ] name = ''xps-106'' hostname = ''xps-106.clichy.jbfavre.org'' Le 02/02/2011 10:27, Ian Campbell a écrit :> On Tue, 2011-02-01 at 22:04 +0000, Jean Baptiste Favre wrote: > > Le 01/02/2011 17:23, Ian Campbell a écrit : > > >> I assume you are not seeing "rx mapping error" in your domU dmesg? Did > >> you post a full guest console log at some point? Comparing the logs for > >> the 256MB, 398MB and 512MB guest RAM case might be useful. > > No sure I''ve ever posted that logs. But I can redo my tests :) > > yes, please do that.Please find attached both console startup logs with 256M and 512M: 256M_domU_console_logs.txt 512M_domU_console_logs.txt For 512M, I saw some kernel CallTrace I can not explain. There are not present with 256M. For 398M memory, I can''t even start domU : # xm create /cluster/xen/xps-106.cfg -c Using config file "/cluster/xen/xps-106.cfg". [215739.007871] pciback 0000:04:00.0: device has been assigned to another domain! Over-writting the ownership, but beware. Started domain xps-106 (id=23) (XEN) mm.c:798:d23 Non-privileged (23) attempt to map I/O space 00000000 (XEN) mm.c:4644:d23 ptwr_emulate: could not get_page_from_l1e() As I told you, I''m still using Debian 2.6.37 kernel because I''ve some problem to compile 2.6.32.27 from Jeremy''s git repository. I hope I can get it compiled today so I''ll be able to test with that kernel as well.> > Please can you also collect and post the information from ifconfig and > ethtool -S which I asked for earlier.Attached as well: 256_domU_ifconfig_ping_ethtool.txt 512_domU_ifconfig_ping_ethtool.txt> Can you confirm that on the domU tcpdump shows no incoming frames at > all, including no corrupted or strange frames. It has been suggested > that the frames could be being received ok but contain garbage (e.g. due > to swiotlb not syncing the memory properly) and hence are not ICMP > Responses but appear as some random frame type. > > Please can you use "tcpdump -w" on both the gateway/peer side and the > affected domU to capture a short session while the failing ping is > running and post (or link to if large) the resulting pcap files. On the > peer side you can use "ether host <domU-MAC>" (plus optionally "or ether > broadcast") on the tcpdump command line to limit the capture to just the > one peer and reduce the size of the capture.I''ll update you as soon as I have tcpdump captures ready. Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, 2011-02-02 at 10:24 +0000, Jean Baptiste Favre wrote:> Hello Ian, > > My domU config file: > > # cat /cluster/xen/xps-106.cfg > kernel = ''/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'' > ramdisk = ''/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'' > builder = ''linux'' > memory = ''398'' > vcpus = ''1'' > cpus = ''2'' > localtime = 0 > serial = ''pty'' > boot = ''cdn'' > disk = [ ''drbd:xps-106,xvda,w'' ] > on_poweroff = ''destroy'' > on_reboot = ''restart'' > on_crash = ''restart'' > extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force > console=hvc0 xencons=tty" > pci = [ ''04:00.0'' ] > name = ''xps-106'' > hostname = ''xps-106.clichy.jbfavre.org'' > > > Le 02/02/2011 10:27, Ian Campbell a écrit : > > On Tue, 2011-02-01 at 22:04 +0000, Jean Baptiste Favre wrote: > > > Le 01/02/2011 17:23, Ian Campbell a écrit : > > > > >> I assume you are not seeing "rx mapping error" in your domU dmesg? Did > > >> you post a full guest console log at some point? Comparing the logs for > > >> the 256MB, 398MB and 512MB guest RAM case might be useful. > > > No sure I''ve ever posted that logs. But I can redo my tests :) > > > > yes, please do that. > Please find attached both console startup logs with 256M and 512M: > 256M_domU_console_logs.txt > 512M_domU_console_logs.txtThanks, I stripped the timestamps and cleaned up some differences due to missing carriage returns, those versions are attached. Interesting bits of diff (- == 256, + == 512) are: -Allocating PCI resources starting at 10800000 (gap: 10800000:ef800000) +Allocating PCI resources starting at 20800000 (gap: 20800000:df800000) So the PCI resources are higher and smaller in the 512MB case. -Placing 64MB software IO TLB between cbd7ee40 - cfd7ee40 -software IO TLB at phys 0xbd7ee40 - 0xfd7ee40 +Placing 64MB software IO TLB between dbb1a000 - dfb1a000 +software IO TLB at phys 0x1bb1a000 - 0x1fb1a000 Fair enough? sky2: driver version 1.28 -sky2 0000:00:00.0: BAR 0: set to [mem 0xfeb00000-0xfeb03fff 64bit] (PCI address [0xfeb00000-0xfeb03fff]) -sky2 0000:00:00.0: BAR 2: set to [io 0xe800-0xe8ff] (PCI address [0xe800-0xe8ff]) -sky2 0000:00:00.0: enabling device (0000 -> 0003) sky2 0000:00:00.0: Xen PCI enabling IRQ: 18 sky2 0000:00:00.0: Yukon-2 EC Ultra chip revision 3 sky2 0000:00:00.0: eth0: addr 00:1f:c6:eb:71:43 So there is some sort of remapping going on in the 256MB case? (or the 512MB logs are missing a bit, which can happen). Can you post the content of domU''s /proc/{interrupts,iomem,ioports} for both cases?> > > > Please can you also collect and post the information from ifconfig and > > ethtool -S which I asked for earlier. > Attached as well: > 256_domU_ifconfig_ping_ethtool.txt > 512_domU_ifconfig_ping_ethtool.txtThanks. The interesting bit here is that the 256MB case is registering RX dropped frames. 512_domU_ifconfig_ping_ethtool.txt did not include an ifconfig after the experiment but I assume the dropped frames are not present there. The rx_dropped statistic is only cranked in a small number of places, once in the driver and a handful in the networking core, only one of the core cases looks likely to be relevant. The following patch should help us figure out where the frames are dropped... Ian. diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c index 7d85a38..645d9e9 100644 --- a/drivers/net/sky2.c +++ b/drivers/net/sky2.c @@ -2339,6 +2339,7 @@ static struct sk_buff *receive_copy(struct sky2_port *sky2, re->skb->ip_summed = CHECKSUM_NONE; skb_put(skb, length); } + WARN(skb == NULL, "sky2: receive_copy failed netdev_alloc_skb_ip_align length %u\n", length); return skb; } @@ -2386,10 +2387,15 @@ static struct sk_buff *receive_new(struct sky2_port *sky2, nre.skb = sky2_rx_alloc(sky2); if (unlikely(!nre.skb)) + { + WARN(1, "sky2: receive_new failed sky2_rx_alloc\n"); goto nobuf; - + } if (sky2_rx_map_skb(sky2->hw->pdev, &nre, hdr_space)) + { + WARN(1, "sky2: receive_new failed sky2_rx_map_skb\n"); goto nomap; + } skb = re->skb; sky2_rx_unmap_skb(sky2->hw->pdev, re); diff --git a/net/core/dev.c b/net/core/dev.c index 54277df..9c99612 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -1516,6 +1516,7 @@ int dev_forward_skb(struct net_device *dev, struct sk_buff *skb) if (unlikely(!(dev->flags & IFF_UP) || (skb->len > (dev->mtu + dev->hard_header_len + VLAN_HLEN)))) { + printk(KERN_CRIT "dev_forward_skb dropping skb\n"); atomic_long_inc(&dev->rx_dropped); kfree_skb(skb); return NET_RX_DROP; @@ -2700,6 +2701,7 @@ enqueue: local_irq_restore(flags); + printk(KERN_CRIT "enqueue_to_backlog dropping skb\n"); atomic_long_inc(&skb->dev->rx_dropped); kfree_skb(skb); return NET_RX_DROP; @@ -3125,6 +3127,7 @@ ncls: if (pt_prev) { ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev); } else { + printk(KERN_CRIT "__netif_receive_skb dropping skb proto %#x\n", type); atomic_long_inc(&skb->dev->rx_dropped); kfree_skb(skb); /* Jamal, now you will not able to escape explaining _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, Le 02/02/2011 11:59, Ian Campbell a écrit :> [...] > >>> No sure I''ve ever posted that logs. But I can redo my tests :) > >> > >> yes, please do that. > > Please find attached both console startup logs with 256M and 512M: > > 256M_domU_console_logs.txt > > 512M_domU_console_logs.txt > > Thanks, I stripped the timestamps and cleaned up some differences due to > missing carriage returns, those versions are attached. > > Interesting bits of diff (- == 256, + == 512) are: > -Allocating PCI resources starting at 10800000 (gap: > 10800000:ef800000) > +Allocating PCI resources starting at 20800000 (gap: > 20800000:df800000) > > So the PCI resources are higher and smaller in the 512MB case. > > -Placing 64MB software IO TLB between cbd7ee40 - cfd7ee40 > -software IO TLB at phys 0xbd7ee40 - 0xfd7ee40 > +Placing 64MB software IO TLB between dbb1a000 - dfb1a000 > +software IO TLB at phys 0x1bb1a000 - 0x1fb1a000 > > Fair enough? > > sky2: driver version 1.28 > -sky2 0000:00:00.0: BAR 0: set to [mem 0xfeb00000-0xfeb03fff > 64bit] (PCI address [0xfeb00000-0xfeb03fff]) > -sky2 0000:00:00.0: BAR 2: set to [io 0xe800-0xe8ff] (PCI > address [0xe800-0xe8ff]) > -sky2 0000:00:00.0: enabling device (0000 -> 0003) > sky2 0000:00:00.0: Xen PCI enabling IRQ: 18 > sky2 0000:00:00.0: Yukon-2 EC Ultra chip revision 3 > sky2 0000:00:00.0: eth0: addr 00:1f:c6:eb:71:43 > > So there is some sort of remapping going on in the 256MB case? (or the > 512MB logs are missing a bit, which can happen). Can you post the > content of domU''s /proc/{interrupts,iomem,ioports} for both cases?Please find them attached. I also join last full console logs for both 256M and 512M: I saw "Call Trace" for 256M I did not had last time.> >> Please can you also collect and post the information from ifconfig and > >> ethtool -S which I asked for earlier. > > Attached as well: > > 256_domU_ifconfig_ping_ethtool.txt > > 512_domU_ifconfig_ping_ethtool.txt > > Thanks. The interesting bit here is that the 256MB case is registering > RX dropped frames. 512_domU_ifconfig_ping_ethtool.txt did not include an > ifconfig after the experiment but I assume the dropped frames are not > present there.I did not include second ifconfig for 512M as ping always works with 512M memory. As far as I can remember, I did not noticed anything special.> The rx_dropped statistic is only cranked in a small number of places, > once in the driver and a handful in the networking core, only one of the > core cases looks likely to be relevant. The following patch should help > us figure out where the frames are dropped...Let''s patch Debian 2.6.37 32bits kernel so :) Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > Ping tests work whatever can be packet size. > > > > If I understood well what you explain to me, it''s now clear that the > > problem is somewhat Xen related, isn''t it ? > > It certainly seems that way. I''m not 100% sure that swiotlb=force will > have actually made the driver use the swiotlb, it may just have forcedIt should have.> the swiotlb to be allocated. Konrad?Both. It would allocate it and force all DMA operations to go through the bounce buffer. Let me take a look at the driver itself. Ah, so I had a similar card: 3c59x which hit the same type of failure way back in 2.6.31 time. It had the pci_dma_sync_single_for_cpu(.. PCI_DMA_FROMDEVICE) turned around so it would never copy properly. But in my tree (2.6.38) the sky driver looks to be doing just right.. It has PCI_DMA_FROMDEVICE.. Well this looks odd: 2330 skb = netdev_alloc_skb_ip_align(sky2->netdev, length); 2331 if (likely(skb)) { 2332 pci_dma_sync_single_for_cpu(sky2->hw->pdev, re->data_addr, 2333 length, PCI_DMA_FROMDEVICE); =======> skb_copy_from_linear_data(re->skb, skb->data, length); <===2335 skb->ip_summed = re->skb->ip_summed; 2336 skb->csum = re->skb->csum; 2337 pci_dma_sync_single_for_device(sky2->hw->pdev, re->data_addr, 2338 length, PCI_DMA_FROMDEVICE); 2339 re->skb->ip_summed = CHECKSUM_NONE; 2340 skb_put(skb, length); 2341 } It copies from skb->data (just allocated) to re->skb (seems that the re->skb->data has been earlier DMA-mapped). But there is no data in skb->data as we just allocated it? Shouldn''t this be the other way around? Like this: diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c index 7d85a38..37c0631 100644 --- a/drivers/net/sky2.c +++ b/drivers/net/sky2.c @@ -2331,7 +2331,7 @@ static struct sk_buff *receive_copy(struct sky2_port *sky2, if (likely(skb)) { pci_dma_sync_single_for_cpu(sky2->hw->pdev, re->data_addr, length, PCI_DMA_FROMDEVICE); - skb_copy_from_linear_data(re->skb, skb->data, length); + skb_copy_from_linear_data(skb, re->skb->data, length); skb->ip_summed = re->skb->ip_summed; skb->csum = re->skb->csum; pci_dma_sync_single_for_device(sky2->hw->pdev, re->data_addr,> > > I''ll be able to continue any tests you want tomorrow. > > Do you still need me to compile domU kernel with "CONFIG_DMA_API_DEBUG" > > enabled ? > > Yes please. > > > If so, what tools will I need to get debug informations ? > > Uh, Konrad?Here, try this patch. Run it under baremetal and Xen and we can compare the outputs .. but I think the issue here is not off mis-using the DMA API but rather a driver bug. /* * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License v2.0 as published by * the Free Software Foundation * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. */ #include <linux/module.h> #include <linux/string.h> #include <linux/types.h> #include <linux/init.h> #include <linux/stat.h> #include <linux/err.h> #include <linux/ctype.h> #include <linux/slab.h> #include <linux/limits.h> #include <linux/device.h> #include <linux/pci.h> #include <linux/blkdev.h> #include <linux/device.h> #include <linux/init.h> #include <linux/mm.h> #include <linux/fcntl.h> #include <linux/slab.h> #include <linux/kmod.h> #include <linux/major.h> #include <linux/smp_lock.h> #include <linux/highmem.h> #include <linux/blkdev.h> #include <linux/module.h> #include <linux/blkpg.h> #include <linux/buffer_head.h> #include <linux/mpage.h> #include <linux/mount.h> #include <linux/uio.h> #include <linux/namei.h> #include <asm/uaccess.h> #include <linux/pagemap.h> #include <linux/pagevec.h> #include <linux/dma-debug.h> #define DUMP_DMA_FUN "0.1" MODULE_AUTHOR("Konrad Rzeszutek Wilk <konrad@virtualiron>"); MODULE_DESCRIPTION("dump dma"); MODULE_LICENSE("GPL"); MODULE_VERSION(DUMP_DMA_FUN); static int __init dump_dma_init(void) { debug_dma_dump_mappings(NULL); return 0; } static void __exit dump_dma_exit(void) { } module_init(dump_dma_init); module_exit(dump_dma_exit);> > Ian. > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, 2011-02-02 at 15:38 +0000, Konrad Rzeszutek Wilk wrote:> > > Ping tests work whatever can be packet size. > > > > > > If I understood well what you explain to me, it''s now clear that the > > > problem is somewhat Xen related, isn''t it ? > > > > It certainly seems that way. I''m not 100% sure that swiotlb=force will > > have actually made the driver use the swiotlb, it may just have forced > > It should have. > > > the swiotlb to be allocated. Konrad? > > Both. It would allocate it and force all DMA operations to go through > the bounce buffer. > > Let me take a look at the driver itself. Ah, so I had a similar card: 3c59x > which hit the same type of failure way back in 2.6.31 time. It had the > pci_dma_sync_single_for_cpu(.. PCI_DMA_FROMDEVICE) turned around so it would never > copy properly. But in my tree (2.6.38) the sky driver looks to be doing just > right.. It has PCI_DMA_FROMDEVICE.. > > Well this looks odd: > > 2330 skb = netdev_alloc_skb_ip_align(sky2->netdev, length); > 2331 if (likely(skb)) { > 2332 pci_dma_sync_single_for_cpu(sky2->hw->pdev, re->data_addr, > 2333 length, PCI_DMA_FROMDEVICE); > =======> skb_copy_from_linear_data(re->skb, skb->data, length); <===> 2335 skb->ip_summed = re->skb->ip_summed; > 2336 skb->csum = re->skb->csum; > 2337 pci_dma_sync_single_for_device(sky2->hw->pdev, re->data_addr, > 2338 length, PCI_DMA_FROMDEVICE); > 2339 re->skb->ip_summed = CHECKSUM_NONE; > 2340 skb_put(skb, length); > 2341 } > > It copies from skb->data (just allocated) to re->skb (seems that the re->skb->data > has been earlier DMA-mapped). > > But there is no data in skb->data as we just allocated it? Shouldn''t this > be the other way around? Like this:skb_copy_from_linear_data''s arguments are the other way round to what you may be expecting: static inline void skb_copy_from_linear_data(const struct sk_buff *skb, void *to, const unsigned int len) So the above line is correctly copying from re->skb->data to skb->data not the other way round. I stared at that exact line for ages this morning ;-) Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> >> Ping tests work whatever can be packet size. > >> > >> If I understood well what you explain to me, it''s now clear that the > >> problem is somewhat Xen related, isn''t it ? > > > > It certainly seems that way. I''m not 100% sure that swiotlb=force will > > have actually made the driver use the swiotlb, it may just have forced > > It should have. > > > the swiotlb to be allocated. Konrad? > > Both. It would allocate it and force all DMA operations to go through > the bounce buffer. > > Let me take a look at the driver itself. Ah, so I had a similar card: > 3c59x > which hit the same type of failure way back in 2.6.31 time. It had the > pci_dma_sync_single_for_cpu(.. PCI_DMA_FROMDEVICE) turned around so it > would never > copy properly. But in my tree (2.6.38) the sky driver looks to be > doing just > right.. It has PCI_DMA_FROMDEVICE.. > > Well this looks odd: > > 2330 skb = netdev_alloc_skb_ip_align(sky2->netdev, length); > 2331 if (likely(skb)) { > 2332 pci_dma_sync_single_for_cpu(sky2->hw->pdev, > re->data_addr, > 2333 length, > PCI_DMA_FROMDEVICE); > =======> skb_copy_from_linear_data(re->skb, skb->data, > length); <===> 2335 skb->ip_summed = re->skb->ip_summed; > 2336 skb->csum = re->skb->csum; > 2337 pci_dma_sync_single_for_device(sky2->hw->pdev, > re->data_addr, > 2338 length, > PCI_DMA_FROMDEVICE); > 2339 re->skb->ip_summed = CHECKSUM_NONE; > 2340 skb_put(skb, length); > 2341 } > > It copies from skb->data (just allocated) to re->skb (seems that the > re->skb->data > has been earlier DMA-mapped). > > But there is no data in skb->data as we just allocated it? Shouldn''t this > be the other way around? Like this: > > > diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c > index 7d85a38..37c0631 100644 > --- a/drivers/net/sky2.c > +++ b/drivers/net/sky2.c > @@ -2331,7 +2331,7 @@ static struct sk_buff *receive_copy(struct > sky2_port *sky2, > if (likely(skb)) { > pci_dma_sync_single_for_cpu(sky2->hw->pdev, re->data_addr, > length, PCI_DMA_FROMDEVICE); > - skb_copy_from_linear_data(re->skb, skb->data, length); > + skb_copy_from_linear_data(skb, re->skb->data, length); > skb->ip_summed = re->skb->ip_summed; > skb->csum = re->skb->csum; > pci_dma_sync_single_for_device(sky2->hw->pdev, re->data_addr, >But, how could a "simple" driver bug automagically appear with Xen and not as Baremetal ? And why only with 32bits kernels ? BTW, I''ll try your patch as soon as I can.> > > >> I''ll be able to continue any tests you want tomorrow. > >> Do you still need me to compile domU kernel with "CONFIG_DMA_API_DEBUG" > >> enabled ? > > > > Yes please. > > > >> If so, what tools will I need to get debug informations ? > > > > Uh, Konrad? > > Here, try this patch. Run it under baremetal and Xen and we can > compare the > outputs .. but I think the issue here is not off mis-using the DMA API but > rather a driver bug.OK, so if you first patch does not solve problem, I''ll compile this one as kernel module and load it, both on 32bits Baremetal and in 32bits DomU Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Feb 02, 2011 at 11:24:51AM +0100, Jean Baptiste Favre wrote:> Hello Ian, > > My domU config file: > > # cat /cluster/xen/xps-106.cfg > kernel = ''/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'' > ramdisk = ''/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'' > builder = ''linux'' > memory = ''398'' > vcpus = ''1'' > cpus = ''2'' > localtime = 0 > serial = ''pty'' > boot = ''cdn'' > disk = [ ''drbd:xps-106,xvda,w'' ] > on_poweroff = ''destroy'' > on_reboot = ''restart'' > on_crash = ''restart'' > extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force > console=hvc0 xencons=tty" > pci = [ ''04:00.0'' ] > name = ''xps-106'' > hostname = ''xps-106.clichy.jbfavre.org'' > > > Le 02/02/2011 10:27, Ian Campbell a écrit : > > On Tue, 2011-02-01 at 22:04 +0000, Jean Baptiste Favre wrote: > > > Le 01/02/2011 17:23, Ian Campbell a écrit : > > > > >> I assume you are not seeing "rx mapping error" in your domU dmesg? Did > > >> you post a full guest console log at some point? Comparing the logs for > > >> the 256MB, 398MB and 512MB guest RAM case might be useful. > > > No sure I''ve ever posted that logs. But I can redo my tests :) > > > > yes, please do that. > Please find attached both console startup logs with 256M and 512M: > 256M_domU_console_logs.txt > 512M_domU_console_logs.txt > > For 512M, I saw some kernel CallTrace I can not explain. There are not > present with 256M. > > For 398M memory, I can''t even start domU : > # xm create /cluster/xen/xps-106.cfg -c > Using config file "/cluster/xen/xps-106.cfg". > [215739.007871] pciback 0000:04:00.0: device has been assigned to > another domain! Over-writting the ownership, but beware. > Started domain xps-106 (id=23) > (XEN) mm.c:798:d23 Non-privileged (23) attempt to map I/O space 00000000 > (XEN) mm.c:4644:d23 ptwr_emulate: could not get_page_from_l1e() > > As I told you, I''m still using Debian 2.6.37 kernel because I''ve some > problem to compile 2.6.32.27 from Jeremy''s git repository. > I hope I can get it compiled today so I''ll be able to test with that > kernel as well.So I''ve tried this on my Abit IP-35 box which has a 04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 13) Subsystem: ABIT Computer Corp. Device 1085 Flags: bus master, fast devsel, latency 0, IRQ 29 Memory at fdefc000 (64-bit, non-prefetchable) [size=16K] I/O ports at be00 [size=256] Expansion ROM at <ignored> [disabled] Capabilities: [48] Power Management version 3 Capabilities: [50] Vital Product Data Capabilities: [5c] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [e0] Express Legacy Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Kernel driver in use: sky2 Kernel modules: sky2 And when I launch this guest with a 32-bit DomU: kernel="/mnt/lab/latest/vmlinuz" ramdisk="/mnt/lab/latest/initramfs.cpio.gz" extra="console=hvc0 debug iommu=soft" memory=320 vcpus=1 cpu=''2'' on_crash="preserve" #vif = [ ''bridge=switch'' ] pci = ["04:00.0"] vfb = [ ''vnc=1, vnclisten=0.0.0.0,vncunused=1''] And played around with the ''extra'' to add ''swiotlb=force''. The moment I had ''swiotlb=force'' I could not get any DHCP address from the NIC. If I did not have ''swiotlb=force'' it would work just fine (can ping any size, etc, this is with 320MB) For fun, I upped the memory (320->720) and kept ''swiotlb=force'' in effect. Same effect: can''t do DHCP. I look to have a different issue than you, which is that whenever I use swiotlb=force, things go haywire. Fyi, this is what DomU tells me: 12:22:41 # 9 :~/> dmesg |grep Memor[ 0.000000] Memory: 145640k/335872k available (3731k kernel code, 189784k reserved, 1565k data, 436k init, 0k highmem) 12:22:56 # 10 :~/> uname -aLinux (none) 2.6.38-rc2-00028-gf2a2d8b #2 SMP Wed Feb 2 12:10:25 EST 2011 i686 i686 i386 GNU/Linux rnet driver. [ 0.921406] udevd (1126): /proc/1126/oom_adj is deprecated, please use /proc/1126/oom_score_adj instead. [ 0.984886] sky2: driver version 1.28 [ 0.995595] sky2 0000:04:00.0: BAR 0: set to [mem 0xfdefc000-0xfdefffff 64bit] (PCI address [0xfdefc000-0xfdefffff]) [ 0.995662] sky2 0000:04:00.0: BAR 2: set to [io 0xbe00-0xbeff] (PCI address [0xbe00-0xbeff]) [ 0.995697] sky2 0000:04:00.0: enabling device (0000 -> 0003) [ 0.996440] sky2 0000:04:00.0: Xen PCI mapped GSI18 to IRQ27 (This is the #master branch from git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, Sorry for not answering earlier. I made a test with OpenWRT, applying patches bellow. DomU boots, can connect, but nothing changed. I still have incoming packets dropped without any console log. Looking deeper, I can see ARP request coming in on my GW, replies coming out. On the domU, arp command output is: # arp IP address HW type Flags HW address Mask Device 10.0.0.1 0x1 0x0 00:00:00:00:00:00 * eth0 So, now I''ve an Layer 2 issue as well :( I''m still trying to make my Debian kernel working so that I can test on Debian too. Regards, JB>From Konrad:diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c index 7d85a38..37c0631 100644 --- a/drivers/net/sky2.c +++ b/drivers/net/sky2.c @@ -2331,7 +2331,7 @@ static struct sk_buff *receive_copy(struct sky2_port *sky2, if (likely(skb)) { pci_dma_sync_single_for_cpu(sky2->hw->pdev, re->data_addr, length, PCI_DMA_FROMDEVICE); - skb_copy_from_linear_data(re->skb, skb->data, length); + skb_copy_from_linear_data(skb, re->skb->data, length); skb->ip_summed = re->skb->ip_summed; skb->csum = re->skb->csum; pci_dma_sync_single_for_device(sky2->hw->pdev, re->data_addr, And from Ian: diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c index 7d85a38..645d9e9 100644 --- a/drivers/net/sky2.c +++ b/drivers/net/sky2.c @@ -2339,6 +2339,7 @@ static struct sk_buff *receive_copy(struct sky2_port *sky2, re->skb->ip_summed = CHECKSUM_NONE; skb_put(skb, length); } + WARN(skb == NULL, "sky2: receive_copy failed netdev_alloc_skb_ip_align length %u\n", length); return skb; } @@ -2386,10 +2387,15 @@ static struct sk_buff *receive_new(struct sky2_port *sky2, nre.skb = sky2_rx_alloc(sky2); if (unlikely(!nre.skb)) + { + WARN(1, "sky2: receive_new failed sky2_rx_alloc\n"); goto nobuf; - + } if (sky2_rx_map_skb(sky2->hw->pdev, &nre, hdr_space)) + { + WARN(1, "sky2: receive_new failed sky2_rx_map_skb\n"); goto nomap; + } skb = re->skb; sky2_rx_unmap_skb(sky2->hw->pdev, re); diff --git a/net/core/dev.c b/net/core/dev.c index 54277df..9c99612 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -1516,6 +1516,7 @@ int dev_forward_skb(struct net_device *dev, struct sk_buff *skb) if (unlikely(!(dev->flags & IFF_UP) || (skb->len > (dev->mtu + dev->hard_header_len + VLAN_HLEN)))) { + printk(KERN_CRIT "dev_forward_skb dropping skb\n"); atomic_long_inc(&dev->rx_dropped); kfree_skb(skb); return NET_RX_DROP; @@ -2700,6 +2701,7 @@ enqueue: local_irq_restore(flags); + printk(KERN_CRIT "enqueue_to_backlog dropping skb\n"); atomic_long_inc(&skb->dev->rx_dropped); kfree_skb(skb); return NET_RX_DROP; @@ -3125,6 +3127,7 @@ ncls: if (pt_prev) { ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev); } else { + printk(KERN_CRIT "__netif_receive_skb dropping skb proto %#x\n", type); atomic_long_inc(&skb->dev->rx_dropped); kfree_skb(skb); /* Jamal, now you will not able to escape explaining Le 02/02/2011 18:42, Konrad Rzeszutek Wilk a écrit :> On Wed, Feb 02, 2011 at 11:24:51AM +0100, Jean Baptiste Favre wrote: >> Hello Ian, >> >> My domU config file: >> >> # cat /cluster/xen/xps-106.cfg >> kernel = ''/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'' >> ramdisk = ''/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'' >> builder = ''linux'' >> memory = ''398'' >> vcpus = ''1'' >> cpus = ''2'' >> localtime = 0 >> serial = ''pty'' >> boot = ''cdn'' >> disk = [ ''drbd:xps-106,xvda,w'' ] >> on_poweroff = ''destroy'' >> on_reboot = ''restart'' >> on_crash = ''restart'' >> extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force >> console=hvc0 xencons=tty" >> pci = [ ''04:00.0'' ] >> name = ''xps-106'' >> hostname = ''xps-106.clichy.jbfavre.org'' >> >> >> Le 02/02/2011 10:27, Ian Campbell a écrit : >>> On Tue, 2011-02-01 at 22:04 +0000, Jean Baptiste Favre wrote: >>>> Le 01/02/2011 17:23, Ian Campbell a écrit : >>> >>>>> I assume you are not seeing "rx mapping error" in your domU dmesg? Did >>>>> you post a full guest console log at some point? Comparing the logs for >>>>> the 256MB, 398MB and 512MB guest RAM case might be useful. >>>> No sure I''ve ever posted that logs. But I can redo my tests :) >>> >>> yes, please do that. >> Please find attached both console startup logs with 256M and 512M: >> 256M_domU_console_logs.txt >> 512M_domU_console_logs.txt >> >> For 512M, I saw some kernel CallTrace I can not explain. There are not >> present with 256M. >> >> For 398M memory, I can''t even start domU : >> # xm create /cluster/xen/xps-106.cfg -c >> Using config file "/cluster/xen/xps-106.cfg". >> [215739.007871] pciback 0000:04:00.0: device has been assigned to >> another domain! Over-writting the ownership, but beware. >> Started domain xps-106 (id=23) >> (XEN) mm.c:798:d23 Non-privileged (23) attempt to map I/O space 00000000 >> (XEN) mm.c:4644:d23 ptwr_emulate: could not get_page_from_l1e() >> >> As I told you, I''m still using Debian 2.6.37 kernel because I''ve some >> problem to compile 2.6.32.27 from Jeremy''s git repository. >> I hope I can get it compiled today so I''ll be able to test with that >> kernel as well. > > So I''ve tried this on my Abit IP-35 box which has a > > 04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 13) > Subsystem: ABIT Computer Corp. Device 1085 > Flags: bus master, fast devsel, latency 0, IRQ 29 > Memory at fdefc000 (64-bit, non-prefetchable) [size=16K] > I/O ports at be00 [size=256] > Expansion ROM at <ignored> [disabled] > Capabilities: [48] Power Management version 3 > Capabilities: [50] Vital Product Data > Capabilities: [5c] MSI: Enable+ Count=1/1 Maskable- 64bit+ > Capabilities: [e0] Express Legacy Endpoint, MSI 00 > Capabilities: [100] Advanced Error Reporting > Kernel driver in use: sky2 > Kernel modules: sky2 > > And when I launch this guest with a 32-bit DomU: > kernel="/mnt/lab/latest/vmlinuz" > ramdisk="/mnt/lab/latest/initramfs.cpio.gz" > extra="console=hvc0 debug iommu=soft" > memory=320 > vcpus=1 > cpu=''2'' > on_crash="preserve" > #vif = [ ''bridge=switch'' ] > pci = ["04:00.0"] > vfb = [ ''vnc=1, vnclisten=0.0.0.0,vncunused=1''] > > And played around with the ''extra'' to add ''swiotlb=force''. > > The moment I had ''swiotlb=force'' I could not get any DHCP > address from the NIC. If I did not have ''swiotlb=force'' it would > work just fine (can ping any size, etc, this is with 320MB) > > > For fun, I upped the memory (320->720) and kept ''swiotlb=force'' in effect. > Same effect: can''t do DHCP. > > I look to have a different issue than you, which is that whenever I use > swiotlb=force, things go haywire. > > Fyi, this is what DomU tells me: > > 12:22:41 # 9 :~/ >> dmesg |grep Memor > [ 0.000000] Memory: 145640k/335872k available (3731k kernel code, 189784k reserved, 1565k data, 436k init, 0k highmem) > > 12:22:56 # 10 :~/ >> uname -a > Linux (none) 2.6.38-rc2-00028-gf2a2d8b #2 SMP Wed Feb 2 12:10:25 EST 2011 i686 i686 i386 GNU/Linux > rnet driver. > [ 0.921406] udevd (1126): /proc/1126/oom_adj is deprecated, please use /proc/1126/oom_score_adj instead. > [ 0.984886] sky2: driver version 1.28 > [ 0.995595] sky2 0000:04:00.0: BAR 0: set to [mem 0xfdefc000-0xfdefffff 64bit] (PCI address [0xfdefc000-0xfdefffff]) > [ 0.995662] sky2 0000:04:00.0: BAR 2: set to [io 0xbe00-0xbeff] (PCI address [0xbe00-0xbeff]) > [ 0.995697] sky2 0000:04:00.0: enabling device (0000 -> 0003) > [ 0.996440] sky2 0000:04:00.0: Xen PCI mapped GSI18 to IRQ27 > > (This is the #master branch from git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git) > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 2011-02-04 at 08:43 +0000, Jean Baptiste Favre wrote:> >From Konrad: > diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c > index 7d85a38..37c0631 100644 > --- a/drivers/net/sky2.c > +++ b/drivers/net/sky2.c > @@ -2331,7 +2331,7 @@ static struct sk_buff *receive_copy(struct > sky2_port *sky2, > if (likely(skb)) { > pci_dma_sync_single_for_cpu(sky2->hw->pdev, re->data_addr, > length, PCI_DMA_FROMDEVICE); > - skb_copy_from_linear_data(re->skb, skb->data, length); > + skb_copy_from_linear_data(skb, re->skb->data, length); > skb->ip_summed = re->skb->ip_summed; > skb->csum = re->skb->csum; > pci_dma_sync_single_for_device(sky2->hw->pdev, re->data_addr,Please don''t apply this bit, the argument order to skb_copy.... can be a bit surprising if you assume it it is like memcpy. The original code is correct. This is probably where your L2 issue came from. Please can you try again with the just the WARN patch. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, Le 04/02/2011 09:53, Ian Campbell a écrit :> On Fri, 2011-02-04 at 08:43 +0000, Jean Baptiste Favre wrote: > >> >From Konrad: >> diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c >> index 7d85a38..37c0631 100644 >> --- a/drivers/net/sky2.c >> +++ b/drivers/net/sky2.c >> @@ -2331,7 +2331,7 @@ static struct sk_buff *receive_copy(struct >> sky2_port *sky2, >> if (likely(skb)) { >> pci_dma_sync_single_for_cpu(sky2->hw->pdev, re->data_addr, >> length, PCI_DMA_FROMDEVICE); >> - skb_copy_from_linear_data(re->skb, skb->data, length); >> + skb_copy_from_linear_data(skb, re->skb->data, length); >> skb->ip_summed = re->skb->ip_summed; >> skb->csum = re->skb->csum; >> pci_dma_sync_single_for_device(sky2->hw->pdev, re->data_addr, > > Please don''t apply this bit, the argument order to skb_copy.... can be a > bit surprising if you assume it it is like memcpy. The original code is > correct. This is probably where your L2 issue came from. > > Please can you try again with the just the WARN patch.Sure I can :) Did not understood Konrad''s patch was incorrect, sorry. Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello Ian,
Applyed your patches.
Now, I''ve:
# ping -s86 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 86 data bytes
__netif_receive_skb dropping skb proto 0x20
So problem seems to occur in net/core/dev.c file, according to the patch
bellow
@@ -3125,6 +3127,7 @@ ncls:
if (pt_prev) {
ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
} else {
printk(KERN_CRIT "__netif_receive_skb dropping skb proto %#x\n",
type);
atomic_long_inc(&skb->dev->rx_dropped);
kfree_skb(skb);
/* Jamal, now you will not able to escape explaining
Regards,
JB
Le 04/02/2011 09:54, Jean Baptiste Favre a écrit :> Hello,
>
> Le 04/02/2011 09:53, Ian Campbell a écrit :
>> On Fri, 2011-02-04 at 08:43 +0000, Jean Baptiste Favre wrote:
>>
>>> >From Konrad:
>>> diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c
>>> index 7d85a38..37c0631 100644
>>> --- a/drivers/net/sky2.c
>>> +++ b/drivers/net/sky2.c
>>> @@ -2331,7 +2331,7 @@ static struct sk_buff *receive_copy(struct
>>> sky2_port *sky2,
>>> if (likely(skb)) {
>>> pci_dma_sync_single_for_cpu(sky2->hw->pdev,
re->data_addr,
>>> length, PCI_DMA_FROMDEVICE);
>>> - skb_copy_from_linear_data(re->skb, skb->data, length);
>>> + skb_copy_from_linear_data(skb, re->skb->data, length);
>>> skb->ip_summed = re->skb->ip_summed;
>>> skb->csum = re->skb->csum;
>>> pci_dma_sync_single_for_device(sky2->hw->pdev,
re->data_addr,
>>
>> Please don''t apply this bit, the argument order to
skb_copy.... can be a
>> bit surprising if you assume it it is like memcpy. The original code is
>> correct. This is probably where your L2 issue came from.
>>
>> Please can you try again with the just the WARN patch.
> Sure I can :)
> Did not understood Konrad''s patch was incorrect, sorry.
>
> Regards,
> JB
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
On Fri, 2011-02-04 at 10:12 +0000, Jean Baptiste Favre wrote:> Hello Ian, > Applyed your patches.Thanks.> Now, I''ve: > # ping -s86 10.0.0.1 > PING 10.0.0.1 (10.0.0.1): 86 data bytes > __netif_receive_skb dropping skb proto 0x20 > > > So problem seems to occur in net/core/dev.c file, according to the patch > bellowInteresting. the number printed in the warning is type == skb->protocol == 0x20 which is not a valid protocol that I can find anywhere (nor is 0x2000 in case I''m mixing my endianesses up). Neither is 0x20 it a valid Ethernet frame length (min 64) so it''s not that sort of confusion AFAICT. skb->protocol is initialised in sky2_status_intr with the return value of "eth_type_trans(skb, dev)" which as far as I can tell cannot return 0x20. The domU network configuration is using the sky2 device directly, no bridging, VLAN, tunnels or anything else like that? Please can you post the output of "ethtool -k <sky2-device>". If either tx or rx checksumming are enabled please can you try disabling them ("ethtool -K <sky2-device> [tx|rx] off"). This setting controls the code path in sky2_skb_rx which can go either to netif_receive_skb or napi_gro_receive. The tcpdump captures from both ends would still be interesting to see. The following patch enhances the previous one with a few checks for protocol 0x20 getting set. Ian. diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c index 7d85a38..727922f 100644 --- a/drivers/net/sky2.c +++ b/drivers/net/sky2.c @@ -2339,6 +2339,7 @@ static struct sk_buff *receive_copy(struct sky2_port *sky2, re->skb->ip_summed = CHECKSUM_NONE; skb_put(skb, length); } + WARN(skb == NULL, "sky2: receive_copy failed netdev_alloc_skb_ip_align length %u\n", length); return skb; } @@ -2386,10 +2387,15 @@ static struct sk_buff *receive_new(struct sky2_port *sky2, nre.skb = sky2_rx_alloc(sky2); if (unlikely(!nre.skb)) + { + WARN(1, "sky2: receive_new failed sky2_rx_alloc\n"); goto nobuf; - + } if (sky2_rx_map_skb(sky2->hw->pdev, &nre, hdr_space)) + { + WARN(1, "sky2: receive_new failed sky2_rx_map_skb\n"); goto nomap; + } skb = re->skb; sky2_rx_unmap_skb(sky2->hw->pdev, re); @@ -2600,6 +2606,7 @@ static int sky2_status_intr(struct sky2_hw *hw, int to_do, u16 idx) } skb->protocol = eth_type_trans(skb, dev); + WARN_ONCE(skb->protocol == 0x20, "sky2_status_intr: skb->protocol is 0x20. ip_summed is %d\n", skb->ip_summed); sky2_skb_rx(sky2, status, skb); diff --git a/net/core/dev.c b/net/core/dev.c index 54277df..5d786b6 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -1516,6 +1516,7 @@ int dev_forward_skb(struct net_device *dev, struct sk_buff *skb) if (unlikely(!(dev->flags & IFF_UP) || (skb->len > (dev->mtu + dev->hard_header_len + VLAN_HLEN)))) { + printk(KERN_CRIT "dev_forward_skb dropping skb\n"); atomic_long_inc(&dev->rx_dropped); kfree_skb(skb); return NET_RX_DROP; @@ -1524,6 +1525,7 @@ int dev_forward_skb(struct net_device *dev, struct sk_buff *skb) skb->tstamp.tv64 = 0; skb->pkt_type = PACKET_HOST; skb->protocol = eth_type_trans(skb, dev); + WARN_ONCE(skb->protocol == 0x20, "dev_forward_skb protocol 0x20 ip_summed %d\n", skb->ip_summed); return netif_rx(skb); } EXPORT_SYMBOL_GPL(dev_forward_skb); @@ -2700,6 +2702,7 @@ enqueue: local_irq_restore(flags); + printk(KERN_CRIT "enqueue_to_backlog dropping skb\n"); atomic_long_inc(&skb->dev->rx_dropped); kfree_skb(skb); return NET_RX_DROP; @@ -3010,6 +3013,8 @@ static int __netif_receive_skb(struct sk_buff *skb) int ret = NET_RX_DROP; __be16 type; + WARN_ONCE(type == 0x20, "__netif_receive_skb protocol 0x20 ip summed %d at line %d\n", skb->ip_summed, __LINE__); + if (!netdev_tstamp_prequeue) net_timestamp_check(skb); @@ -3111,6 +3116,7 @@ ncls: } type = skb->protocol; + WARN_ONCE(type == 0x20, "__netif_receive_skb protocol 0x20 ip summed %d at line %d\n", skb->ip_summed, __LINE__); list_for_each_entry_rcu(ptype, &ptype_base[ntohs(type) & PTYPE_HASH_MASK], list) { if (ptype->type == type && (ptype->dev == null_or_orig || @@ -3125,6 +3131,7 @@ ncls: if (pt_prev) { ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev); } else { + printk(KERN_CRIT "__netif_receive_skb dropping skb proto %#x ip summed %d\n", type, skb->ip_summed); atomic_long_inc(&skb->dev->rx_dropped); kfree_skb(skb); /* Jamal, now you will not able to escape explaining @@ -3447,6 +3454,7 @@ gro_result_t napi_frags_finish(struct napi_struct *napi, struct sk_buff *skb, case GRO_NORMAL: case GRO_HELD: skb->protocol = eth_type_trans(skb, skb->dev); + WARN_ONCE(skb->protocol == 0x20, "napi_frags_finish protocol 0x20 ip_summed %d\n", skb->ip_summed); if (ret == GRO_HELD) skb_gro_pull(skb, -ETH_HLEN); @@ -3498,6 +3506,7 @@ struct sk_buff *napi_frags_skb(struct napi_struct *napi) * special handling. We''ll fix it up properly at the end. */ skb->protocol = eth->h_proto; + WARN_ONCE(skb->protocol == 0x20, "napi_frags_skb protocol 0x20 ip_summed %d\n", skb->ip_summed); out: return skb; _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, Le 04/02/2011 12:04, Ian Campbell a écrit :> On Fri, 2011-02-04 at 10:12 +0000, Jean Baptiste Favre wrote: >> Hello Ian, >> Applyed your patches. > > Thanks. > >> Now, I''ve: >> # ping -s86 10.0.0.1 >> PING 10.0.0.1 (10.0.0.1): 86 data bytes >> __netif_receive_skb dropping skb proto 0x20 >> >> >> So problem seems to occur in net/core/dev.c file, according to the patch >> bellow > > Interesting. the number printed in the warning is type == skb->protocol > == 0x20 which is not a valid protocol that I can find anywhere (nor is > 0x2000 in case I''m mixing my endianesses up). Neither is 0x20 it a valid > Ethernet frame length (min 64) so it''s not that sort of confusion > AFAICT. > > skb->protocol is initialised in sky2_status_intr with the return value > of "eth_type_trans(skb, dev)" which as far as I can tell cannot return > 0x20. > > The domU network configuration is using the sky2 device directly, no > bridging, VLAN, tunnels or anything else like that?At boot it uses bridge. For the test, I delete bridge and set IP address directly on eth0 root@OpenWrt:/# ifconfig br-lan down br-lan: port 1(eth0) entering forwarding state root@OpenWrt:/# brctl delbr br-lan device eth0 left promiscuous mode br-lan: port 1(eth0) entering disabled state root@OpenWrt:/# ifconfig eth0 10.0.0.8 netmask 255.255.255.0 root@OpenWrt:/# ping 10.0.0.1 PING 10.0.0.1 (10.0.0.1): 56 data bytes 64 bytes from 10.0.0.1: seq=0 ttl=64 time=10.455 ms 64 bytes from 10.0.0.1: seq=1 ttl=64 time=0.870 ms ^C --- 10.0.0.1 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.870/5.662/10.455 ms root@OpenWrt:/# ping -s86 10.0.0.1 PING 10.0.0.1 (10.0.0.1): 86 data bytes ^C --- 10.0.0.1 ping statistics --- 12 packets transmitted, 0 packets received, 100% packet loss> > Please can you post the output of "ethtool -k <sky2-device>".root@OpenWrt:/# ethtool -k eth0 Offload parameters for eth0: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off ntuple-filters: off receive-hashing: on> If either tx or rx checksumming are enabled please can you try disabling > them ("ethtool -K <sky2-device> [tx|rx] off"). This setting controls the > code path in sky2_skb_rx which can go either to netif_receive_skb or > napi_gro_receive.root@OpenWrt:/# ethtool -K eth0 tx off root@OpenWrt:/# ethtool -K eth0 rx off root@OpenWrt:/# ping -s86 10.0.0.1 PING 10.0.0.1 (10.0.0.1): 86 data bytes ^C --- 10.0.0.1 ping statistics --- 3 packets transmitted, 0 packets received, 100% packet loss root@OpenWrt:/# ethtool -k eth0 Offload parameters for eth0: rx-checksumming: off tx-checksumming: off scatter-gather: off tcp-segmentation-offload: off udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: off large-receive-offload: off ntuple-filters: off receive-hashing: on> The tcpdump captures from both ends would still be interesting to see. > > The following patch enhances the previous one with a few checks for > protocol 0x20 getting set. > > Ian.What is a bit strange here is that I don''t any more the KERN_CRIT printk message. Could be a false positive ? I''m currently compiling new kernel with your last patch. will keep you updated Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 2011-02-04 at 11:25 +0000, Jean Baptiste Favre wrote:> Hello, > > Le 04/02/2011 12:04, Ian Campbell a écrit : > > On Fri, 2011-02-04 at 10:12 +0000, Jean Baptiste Favre wrote: > >> Hello Ian, > >> Applyed your patches. > > > > Thanks. > > > >> Now, I''ve: > >> # ping -s86 10.0.0.1 > >> PING 10.0.0.1 (10.0.0.1): 86 data bytes > >> __netif_receive_skb dropping skb proto 0x20 > >> > >> > >> So problem seems to occur in net/core/dev.c file, according to the patch > >> bellow > > > > Interesting. the number printed in the warning is type == skb->protocol > > == 0x20 which is not a valid protocol that I can find anywhere (nor is > > 0x2000 in case I''m mixing my endianesses up). Neither is 0x20 it a valid > > Ethernet frame length (min 64) so it''s not that sort of confusion > > AFAICT. > > > > skb->protocol is initialised in sky2_status_intr with the return value > > of "eth_type_trans(skb, dev)" which as far as I can tell cannot return > > 0x20. > > > > The domU network configuration is using the sky2 device directly, no > > bridging, VLAN, tunnels or anything else like that? > At boot it uses bridge. > For the test, I delete bridge and set IP address directly on eth0OK, good. [...]> generic-segmentation-offload: on[...]> receive-hashing: onCan you also try turning these two off (independently and together).> What is a bit strange here is that I don''t any more the KERN_CRIT printk > message. > Could be a false positive ?Worth bearing in mind, lets see what the next test run produces.> I''m currently compiling new kernel with your last patch. will keep you > updatedThanks. Please gather the tcpdump''s too. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Le 04/02/2011 12:28, Ian Campbell a écrit :> On Fri, 2011-02-04 at 11:25 +0000, Jean Baptiste Favre wrote: >> Hello, >> >> Le 04/02/2011 12:04, Ian Campbell a écrit : >>> On Fri, 2011-02-04 at 10:12 +0000, Jean Baptiste Favre wrote: >>>> Hello Ian, >>>> Applyed your patches. >>> >>> Thanks. >>> >>>> Now, I''ve: >>>> # ping -s86 10.0.0.1 >>>> PING 10.0.0.1 (10.0.0.1): 86 data bytes >>>> __netif_receive_skb dropping skb proto 0x20 >>>> >>>> >>>> So problem seems to occur in net/core/dev.c file, according to the patch >>>> bellow >>> >>> Interesting. the number printed in the warning is type == skb->protocol >>> == 0x20 which is not a valid protocol that I can find anywhere (nor is >>> 0x2000 in case I''m mixing my endianesses up). Neither is 0x20 it a valid >>> Ethernet frame length (min 64) so it''s not that sort of confusion >>> AFAICT. >>> >>> skb->protocol is initialised in sky2_status_intr with the return value >>> of "eth_type_trans(skb, dev)" which as far as I can tell cannot return >>> 0x20. >>> >>> The domU network configuration is using the sky2 device directly, no >>> bridging, VLAN, tunnels or anything else like that? >> At boot it uses bridge. >> For the test, I delete bridge and set IP address directly on eth0 > > OK, good. > > [...] >> generic-segmentation-offload: on > [...] >> receive-hashing: on > > Can you also try turning these two off (independently and together).No change with ethtool -K gso off Can not change receive-hashing: # ethtool -K eth1 rxhash off Cannot set device flag settings: Invalid argument>> What is a bit strange here is that I don''t any more the KERN_CRIT printk >> message. >> Could be a false positive ? > > Worth bearing in mind, lets see what the next test run produces.Seems that I got this messge only with copybreak=0. With default value (128), no such message More, with copybreak=0, all packets are dropped (even a ping with default packet size is dropped. Same with ping -s1)>> I''m currently compiling new kernel with your last patch. will keep you >> updatedNo new messages, despite your patch :(> Thanks. > > Please gather the tcpdump''s too.Both tcpdump from GW and domU are Attached. Commands were: domU# tcpdump -n -w domU.cap -s0 -i eth0 ether host 00:1f:c6:eb:71:43 or ether broadcast gw# tcpdump -n -w gw.cap -s0 -i br0 ether host 00:1f:c6:eb:71:43 or ether broadcast _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 2011-02-04 at 13:15 +0000, Jean Baptiste Favre wrote:> > >> What is a bit strange here is that I don''t any more the KERN_CRIT printk > >> message. > >> Could be a false positive ? > > > > Worth bearing in mind, lets see what the next test run produces. > Seems that I got this messge only with copybreak=0. > With default value (128), no such message > > More, with copybreak=0, all packets are dropped (even a ping with > default packet size is dropped. Same with ping -s1)Hang on, I thought you previously said copybreak=0 made everything work ok. If that isn''t definitely the case then we may be following a red herring. Are you saying that copybreak=0 + this patch breaks? That would be very surprising since the patch doesn''t cause any flow control differences. Perhaps there is some difference between your self-built kernels and the Debian kernels you started with? Perhaps you should try the self built kernel with no patches, just to confirm it behaves the same as the Debian kernels?> > Thanks. > > > > Please gather the tcpdump''s too. > Both tcpdump from GW and domU are Attached.Were these collected with or without patches? With or without ethtool -K options? With or without copybreak? Please try and be explicit about everything you post, there are lots of variables in the air.> Commands were: > > domU# tcpdump -n -w domU.cap -s0 -i eth0 ether host 00:1f:c6:eb:71:43 or > ether broadcast > > gw# tcpdump -n -w gw.cap -s0 -i br0 ether host 00:1f:c6:eb:71:43 or > ether broadcastSo no 0x20 anywhere in what the gw sent as the reply and still no clue where the value came from... Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Le 04/02/2011 14:50, Ian Campbell a écrit :> On Fri, 2011-02-04 at 13:15 +0000, Jean Baptiste Favre wrote: > >> >>>> What is a bit strange here is that I don''t any more the KERN_CRIT printk >>>> message. >>>> Could be a false positive ? >>> >>> Worth bearing in mind, lets see what the next test run produces. >> Seems that I got this messge only with copybreak=0. >> With default value (128), no such message >> >> More, with copybreak=0, all packets are dropped (even a ping with >> default packet size is dropped. Same with ping -s1) > > Hang on, I thought you previously said copybreak=0 made everything work > ok. If that isn''t definitely the case then we may be following a red > herring.That''s something I''m investigating. Under Debian, copybreak=0 solve the problem Under OpenWRT, copybreak=0 + patch breaks. Will try without patch.> Are you saying that copybreak=0 + this patch breaks? That would be very > surprising since the patch doesn''t cause any flow control differences. > > Perhaps there is some difference between your self-built kernels and the > Debian kernels you started with? Perhaps you should try the self built > kernel with no patches, just to confirm it behaves the same as the > Debian kernels?Under Debian, I use 2.6.37 from experimental Under OpenWRT, use legacy 2.6.37, build env applies patches for OpenWRT and compile. OpenWRT provides complete build env, as I still have problem compiling Debian 32bits kernel from 64bits env. That''s why I switched back to openWRT for testing.>>> Thanks. >>> >>> Please gather the tcpdump''s too. >> Both tcpdump from GW and domU are Attached. > > Were these collected with or without patches? With or without ethtool -K > options? With or without copybreak? > > Please try and be explicit about everything you post, there are lots of > variables in the air.OK, sorry. Will redo all tests Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, Sorry for long silent period. I''m taking night classes to get an engineering degree and am in the midst of exams this whole week. I''ll be back available next week to perform required tests and will provide you an update as soon as possible. Regards, JB Le 04/02/2011 15:01, Jean Baptiste Favre a écrit :> Le 04/02/2011 14:50, Ian Campbell a écrit : >> On Fri, 2011-02-04 at 13:15 +0000, Jean Baptiste Favre wrote: >> >>> >>>>> What is a bit strange here is that I don''t any more the KERN_CRIT printk >>>>> message. >>>>> Could be a false positive ? >>>> >>>> Worth bearing in mind, lets see what the next test run produces. >>> Seems that I got this messge only with copybreak=0. >>> With default value (128), no such message >>> >>> More, with copybreak=0, all packets are dropped (even a ping with >>> default packet size is dropped. Same with ping -s1) >> >> Hang on, I thought you previously said copybreak=0 made everything work >> ok. If that isn''t definitely the case then we may be following a red >> herring. > That''s something I''m investigating. > Under Debian, copybreak=0 solve the problem > Under OpenWRT, copybreak=0 + patch breaks. Will try without patch. > >> Are you saying that copybreak=0 + this patch breaks? That would be very >> surprising since the patch doesn''t cause any flow control differences. >> >> Perhaps there is some difference between your self-built kernels and the >> Debian kernels you started with? Perhaps you should try the self built >> kernel with no patches, just to confirm it behaves the same as the >> Debian kernels? > Under Debian, I use 2.6.37 from experimental > Under OpenWRT, use legacy 2.6.37, build env applies patches for OpenWRT > and compile. > > OpenWRT provides complete build env, as I still have problem compiling > Debian 32bits kernel from 64bits env. That''s why I switched back to > openWRT for testing. > > >>>> Thanks. >>>> >>>> Please gather the tcpdump''s too. >>> Both tcpdump from GW and domU are Attached. >> >> Were these collected with or without patches? With or without ethtool -K >> options? With or without copybreak? >> >> Please try and be explicit about everything you post, there are lots of >> variables in the air. > OK, sorry. Will redo all tests > > Regards, > JB > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, Back online after my exams :) Had some time to perform tests with my Debian Squeeze 32bits domU and 2.6.37 kernel from experimental. DomU config is: ***************************************************************** kernel = ''/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'' ramdisk = ''/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'' #kernel = ''/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem-sky2'' #ramdisk = ''/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem-sky2'' builder = ''linux'' memory=268 vcpus = ''1'' cpus = ''2'' localtime = 0 serial = ''pty'' boot = ''cdn'' disk = [ ''drbd:xps-106,xvda,w'' ] on_poweroff = ''destroy'' on_reboot = ''restart'' on_crash = ''restart'' name = ''xps-106'' hostname = ''xps-106.clichy.jbfavre.org'' extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force console=hvc0 xencons=tty" pci = [ ''04:00.0'' ] ***************************************************************** Tests were done first with "legacy" kernel from Debian and with patched kernel (patches from Ian to get verbose debug). Each test consist in executing following command: ping -c5 -s150 10.0.0.1 ping -c5 -s85 10.0.0.1 10.0.0.1 is my gateway As you can see bellow, all verbose debug display "__netif_receive_skb dropping skb" with various "proto" value and always "ip summed 2". I can provide full detailled output if you want. Here are the results: Memory 2.6.37 2.6.37-patched 128 Works Works 130 NoBoot NoBoot 132 Works Works 134 NoBoot NoBoot 136 Works Works 138 NoBoot NoBoot 140 Fails __netif_receive_skb dropping skb 142 NoBoot NoBoot 144 Fails __netif_receive_skb dropping skb 146 NoBoot NoBoot 148 Fails __netif_receive_skb dropping skb 150 NoBoot NoBoot 152 Fails __netif_receive_skb dropping skb 154 NoBoot NoBoot 156 Fails __netif_receive_skb dropping skb 158 NoBoot NoBoot 160 Fails __netif_receive_skb dropping skb 192 Fails __netif_receive_skb dropping skb 224 Fails __netif_receive_skb dropping skb 256 Fails __netif_receive_skb dropping skb 264 Fails __netif_receive_skb dropping skb 266 NoBoot NoBoot 268 Works Works 272 Works Works 288 Works Works 320 Works Works 352 Works Works 384 Works Works 416 Works Works 448 Works Works 480 Works Works 512 Works Works Now, I''ll try to do the same tests with OpenWRT. I hope I''ll have similar results :) Regards, JB Le 09/02/2011 10:59, Jean Baptiste Favre a écrit :> Hello, > Sorry for long silent period. > > I''m taking night classes to get an engineering degree and am in the > midst of exams this whole week. > > I''ll be back available next week to perform required tests and will > provide you an update as soon as possible. > > Regards, > JB > > Le 04/02/2011 15:01, Jean Baptiste Favre a écrit : >> Le 04/02/2011 14:50, Ian Campbell a écrit : >>> On Fri, 2011-02-04 at 13:15 +0000, Jean Baptiste Favre wrote: >>> >>>> >>>>>> What is a bit strange here is that I don''t any more the KERN_CRIT printk >>>>>> message. >>>>>> Could be a false positive ? >>>>> >>>>> Worth bearing in mind, lets see what the next test run produces. >>>> Seems that I got this messge only with copybreak=0. >>>> With default value (128), no such message >>>> >>>> More, with copybreak=0, all packets are dropped (even a ping with >>>> default packet size is dropped. Same with ping -s1) >>> >>> Hang on, I thought you previously said copybreak=0 made everything work >>> ok. If that isn''t definitely the case then we may be following a red >>> herring. >> That''s something I''m investigating. >> Under Debian, copybreak=0 solve the problem >> Under OpenWRT, copybreak=0 + patch breaks. Will try without patch. >> >>> Are you saying that copybreak=0 + this patch breaks? That would be very >>> surprising since the patch doesn''t cause any flow control differences. >>> >>> Perhaps there is some difference between your self-built kernels and the >>> Debian kernels you started with? Perhaps you should try the self built >>> kernel with no patches, just to confirm it behaves the same as the >>> Debian kernels? >> Under Debian, I use 2.6.37 from experimental >> Under OpenWRT, use legacy 2.6.37, build env applies patches for OpenWRT >> and compile. >> >> OpenWRT provides complete build env, as I still have problem compiling >> Debian 32bits kernel from 64bits env. That''s why I switched back to >> openWRT for testing. >> >> >>>>> Thanks. >>>>> >>>>> Please gather the tcpdump''s too. >>>> Both tcpdump from GW and domU are Attached. >>> >>> Were these collected with or without patches? With or without ethtool -K >>> options? With or without copybreak? >>> >>> Please try and be explicit about everything you post, there are lots of >>> variables in the air. >> OK, sorry. Will redo all tests >> >> Regards, >> JB >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Feb 18, 2011 at 10:14:13PM +0100, Jean Baptiste Favre wrote:> Hello, > Back online after my exams :) > > Had some time to perform tests with my Debian Squeeze 32bits domU and > 2.6.37 kernel from experimental.I am bit lost now.. Can you refresh my memory whether the ''copy_break'' parameter worked or not? Experimental is the stock kernel or was that a new proposed kernel? Where are the sources for the experimental kernel?> DomU config is: > ***************************************************************** > kernel = ''/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'' > ramdisk = ''/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'' > #kernel = ''/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem-sky2'' > #ramdisk = ''/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem-sky2'' > builder = ''linux'' > memory=268 > vcpus = ''1'' > cpus = ''2'' > localtime = 0 > serial = ''pty'' > boot = ''cdn'' > disk = [ ''drbd:xps-106,xvda,w'' ] > on_poweroff = ''destroy'' > on_reboot = ''restart'' > on_crash = ''restart'' > name = ''xps-106'' > hostname = ''xps-106.clichy.jbfavre.org''I had a box with a sky2 adapter that looked to have a similar issue but found the culprit to be the switch. So at this point I am having no luck reproducing this. Would it be possible for you to stick the kernel + dist image somewhere so I can try it out on my box?> > extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force > console=hvc0 xencons=tty"Try without ''swiotlb=force'' on any kernel that is PVOPS. Only the older ones (lenny) required that. And you don''t need ''xencons=tty'' either with PVOPS kernels. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Le 25/02/2011 15:40, Konrad Rzeszutek Wilk a écrit :> On Fri, Feb 18, 2011 at 10:14:13PM +0100, Jean Baptiste Favre wrote: >> Hello, >> Back online after my exams :) >> >> Had some time to perform tests with my Debian Squeeze 32bits domU and >> 2.6.37 kernel from experimental. > > I am bit lost now.. Can you refresh my memory whether the ''copy_break'' > parameter worked or not?Copybreak param still solve problem: # ping 10.0.0.1 PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data. __netif_receive_skb dropping skb proto 0x6aac ip summed 2 __netif_receive_skb dropping skb proto 0x620c ip summed 2 __netif_receive_skb dropping skb proto 0x6aac ip summed 2 ^C --- 10.0.0.1 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 1999ms root@xps-106:~# rmmod sky2 [ 23.137515] sky2 0000:00:00.0: eth0: disabling interface root@xps-106:~# modprobe sky2 copybreak=0 sky2: driver version 1.28 sky2 0000:00:00.0: Xen PCI enabling IRQ: 18 xen_map_pirq_gsi: returning irq 18 for gsi 18 sky2 0000:00:00.0: Yukon-2 EC Ultra chip revision 3 xen_map_pirq_gsi: returning irq 55 for gsi 55 sky2 0000:00:00.0: eth0: addr 00:1f:c6:eb:71:43 sky2 0000:00:00.0: eth0: enabling interface ADDRCONF(NETDEV_UP): eth0: link is not ready sky2 0000:00:00.0: eth0: Link is up at 1000 Mbps, full duplex, flow control both ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready root@xps-106:~# ping 10.0.0.1 PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data. 64 bytes from 10.0.0.1: icmp_req=1 ttl=64 time=4.09 ms 64 bytes from 10.0.0.1: icmp_req=2 ttl=64 time=0.814 ms ^C --- 10.0.0.1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 0.814/2.456/4.098/1.642 ms> > Experimental is the stock kernel or was that a new proposed kernel? > Where are the sources for the experimental kernel?Kernel 2.6.37 has been installed from Debian experimental repository. Did not have time to compile my own from vanilla sources. Maybe will have some time this week, but not sure.>> DomU config is: >> ***************************************************************** >> kernel = ''/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'' >> ramdisk = ''/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'' >> #kernel = ''/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem-sky2'' >> #ramdisk = ''/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem-sky2'' >> builder = ''linux'' >> memory=268 >> vcpus = ''1'' >> cpus = ''2'' >> localtime = 0 >> serial = ''pty'' >> boot = ''cdn'' >> disk = [ ''drbd:xps-106,xvda,w'' ] >> on_poweroff = ''destroy'' >> on_reboot = ''restart'' >> on_crash = ''restart'' >> name = ''xps-106'' >> hostname = ''xps-106.clichy.jbfavre.org'' > > I had a box with a sky2 adapter that looked to have a similar issue but > found the culprit to be the switch. So at this point I am having no luck > reproducing this. Would it be possible for you to stick the kernel + dist > image somewhere so I can try it out on my box?As requested by you (or Ian, I don''t remeber exactly), I''ve tried booting the server with 32bits 2.6.37 kernel and 32bits Debian Squeeze. Even with mem=256 boot param, everything worked fine, so I''m not sure switch can be the culprit.>> extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force >> console=hvc0 xencons=tty" > > Try without ''swiotlb=force'' on any kernel that is PVOPS. Only the older > ones (lenny) required that. And you don''t need ''xencons=tty'' either with > PVOPS kernels.OK, will try that. Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, Le 25/02/2011 15:40, Konrad Rzeszutek Wilk a écrit :> On Fri, Feb 18, 2011 at 10:14:13PM +0100, Jean Baptiste Favre wrote: >> Hello, >> Back online after my exams :) >> >> Had some time to perform tests with my Debian Squeeze 32bits domU and >> 2.6.37 kernel from experimental. > > I am bit lost now.. Can you refresh my memory whether the ''copy_break'' > parameter worked or not? > > Experimental is the stock kernel or was that a new proposed kernel? > Where are the sources for the experimental kernel? > >> DomU config is: >> ***************************************************************** >> kernel = ''/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'' >> ramdisk = ''/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'' >> #kernel = ''/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem-sky2'' >> #ramdisk = ''/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem-sky2'' >> builder = ''linux'' >> memory=268 >> vcpus = ''1'' >> cpus = ''2'' >> localtime = 0 >> serial = ''pty'' >> boot = ''cdn'' >> disk = [ ''drbd:xps-106,xvda,w'' ] >> on_poweroff = ''destroy'' >> on_reboot = ''restart'' >> on_crash = ''restart'' >> name = ''xps-106'' >> hostname = ''xps-106.clichy.jbfavre.org'' > > I had a box with a sky2 adapter that looked to have a similar issue but > found the culprit to be the switch. So at this point I am having no luck > reproducing this. Would it be possible for you to stick the kernel + dist > image somewhere so I can try it out on my box? > >> >> extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force >> console=hvc0 xencons=tty" > > Try without ''swiotlb=force'' on any kernel that is PVOPS. Only the older > ones (lenny) required that. And you don''t need ''xencons=tty'' either with > PVOPS kernels.Thought I already tried without ''swiotlb=force'', but seems not... Just tried it on my Debian domU, and everything works now, whatever copybreak value can be. But trying the same with OpenWRT gives following results: - With ''swiotlb=force'': fails for size equal or greater than 128 bytes (or ping -s86) - Without ''swiotlb=force'': always fails I can provide OpenWRT disk img and kernel. For Debian domU, I can provide a dd from LVM and kernel. Let me know which ones you want. Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> >> extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force > >> console=hvc0 xencons=tty" > > > > Try without ''swiotlb=force'' on any kernel that is PVOPS. Only the older > > ones (lenny) required that. And you don''t need ''xencons=tty'' either with > > PVOPS kernels. > Thought I already tried without ''swiotlb=force'', but seems not... > > Just tried it on my Debian domU, and everything works now, whatever > copybreak value can be. > > But trying the same with OpenWRT gives following results: > - With ''swiotlb=force'': fails for size equal or greater than 128 bytes > (or ping -s86) > - Without ''swiotlb=force'': always fails > > I can provide OpenWRT disk img and kernel. For Debian domU, I can > provide a dd from LVM and kernel. Let me know which ones you want.Lets do both. Do you know where the sources for OpenWRT are located? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, Le 28/02/2011 16:01, Konrad Rzeszutek Wilk a écrit :>>>> extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force >>>> console=hvc0 xencons=tty" >>> >>> Try without ''swiotlb=force'' on any kernel that is PVOPS. Only the older >>> ones (lenny) required that. And you don''t need ''xencons=tty'' either with >>> PVOPS kernels. >> Thought I already tried without ''swiotlb=force'', but seems not... >> >> Just tried it on my Debian domU, and everything works now, whatever >> copybreak value can be. >> >> But trying the same with OpenWRT gives following results: >> - With ''swiotlb=force'': fails for size equal or greater than 128 bytes >> (or ping -s86) >> - Without ''swiotlb=force'': always fails >> >> I can provide OpenWRT disk img and kernel. For Debian domU, I can >> provide a dd from LVM and kernel. Let me know which ones you want. > > Lets do both. Do you know where the sources for OpenWRT are located?Openwrt .img and PV kernel are available at http://downloads.jbfavre.org/openwrt.tar.gz I''m making dd from my debian DomU and will upload it as debian.tar.gz at the same place. Will update you when completed. Openwrt source tree is available here: svn://svn.openwrt.org/openwrt/trunk You have all information to build it here: http://wiki.openwrt.org/doc/howto/build Or you have all steps I followed here: http://publications.jbfavre.org/virtualisation/xen_openwrt_domu_pci_passthrough.en Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> >> Lets do both. Do you know where the sources for OpenWRT are located? > > > > Openwrt .img and PV kernel are available at > > http://downloads.jbfavre.org/openwrt.tar.gzUsing that, and this xm file kernel="/mnt/tmp/openwrt/openwrt-x86-xen_domu-vmlinuz" root=''/dev/xvda2 rw'' memory=256 vcpus=1 localtime=0 disk=[''phy:/dev/sdc,xvda,w''] extra="console=hvc0 debug loglevel=10 iommu=soft" name="openwrt" on_crash="preserve" vfb = [ ''vnc=1, vnclisten=0.0.0.0,vncunused=1''] pci = [''04:00.0''] where 04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 13) I can''t get the sky2 adapter to work at all. Just to make sure it wasn''t your build ...> > > > I''m making dd from my debian DomU and will upload it as debian.tar.gz at > > the same place. Will update you when completed. > > > > Openwrt source tree is available here: > > svn://svn.openwrt.org/openwrt/trunk > > > > You have all information to build it here: > > http://wiki.openwrt.org/doc/howto/build > > > > Or you have all steps I followed here: > > http://publications.jbfavre.org/virtualisation/xen_openwrt_domu_pci_passthrough.en... I tried to follow those directions and found that it would not work. I can''t get the xen-pcifront.ko file at all on any of the *combined.img images. I made this patch thinking it was due to the name of the module being different: Index: target/linux/x86/Makefile ==================================================================--- target/linux/x86/Makefile (revision 25855) +++ target/linux/x86/Makefile (working copy) @@ -12,7 +12,7 @@ FEATURES:=squashfs jffs2 ext4 vdi vmdk pcmcia targz SUBTARGETS=generic olpc xen_domu ep80579 net5501 kvm_guest geos -LINUX_VERSION:=2.6.32.29 +LINUX_VERSION:=2.6.37 include $(INCLUDE_DIR)/target.mk Index: package/kernel/modules/virtual.mk ==================================================================--- package/kernel/modules/virtual.mk (revision 25855) +++ package/kernel/modules/virtual.mk (working copy) @@ -168,7 +168,7 @@ TITLE:=Xen PCI device frontend DEPENDS:=@TARGET_x86_xen_domu @LINUX_2_6_37||LINUX_2_6_38 KCONFIG:=CONFIG_XEN_PCIDEV_FRONTEND - FILES:=$(LINUX_DIR)/drivers/xen/platform-pci.ko + FILES:=$(LINUX_DIR)/drivers/pci/xen-pcifront.ko AUTOLOAD:=$(call AutoLoad,10,xen-pcifront) endef but it still would not include the xen-pcifront.ko file on the *combined-ext4.img.gz file. Any ideas what I am doing wrong? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, Le 03/03/2011 23:12, Konrad Rzeszutek Wilk a écrit :>>>> Lets do both. Do you know where the sources for OpenWRT are located? >>> >>> Openwrt .img and PV kernel are available at >>> http://downloads.jbfavre.org/openwrt.tar.gz > > Using that, and this xm file > > kernel="/mnt/tmp/openwrt/openwrt-x86-xen_domu-vmlinuz" > root=''/dev/xvda2 rw'' > memory=256 > vcpus=1 > localtime=0 > disk=[''phy:/dev/sdc,xvda,w''] > extra="console=hvc0 debug loglevel=10 iommu=soft" > name="openwrt" > on_crash="preserve" > vfb = [ ''vnc=1, vnclisten=0.0.0.0,vncunused=1''] > pci = [''04:00.0''] > > where > 04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 13) > > I can''t get the sky2 adapter to work at all. > > Just to make sure it wasn''t your build ... >>> >>> I''m making dd from my debian DomU and will upload it as debian.tar.gz at >>> the same place. Will update you when completed. >>> >>> Openwrt source tree is available here: >>> svn://svn.openwrt.org/openwrt/trunk >>> >>> You have all information to build it here: >>> http://wiki.openwrt.org/doc/howto/build >>> >>> Or you have all steps I followed here: >>> http://publications.jbfavre.org/virtualisation/xen_openwrt_domu_pci_passthrough.en > > ... I tried to follow those directions and found that it would not work. > I can''t get the xen-pcifront.ko file at all on any of the *combined.img images. > > > I made this patch thinking it was due to the name of the module being different: > > Index: target/linux/x86/Makefile > ==================================================================> --- target/linux/x86/Makefile (revision 25855) > +++ target/linux/x86/Makefile (working copy) > @@ -12,7 +12,7 @@ > FEATURES:=squashfs jffs2 ext4 vdi vmdk pcmcia targz > SUBTARGETS=generic olpc xen_domu ep80579 net5501 kvm_guest geos > > -LINUX_VERSION:=2.6.32.29 > +LINUX_VERSION:=2.6.37 > > include $(INCLUDE_DIR)/target.mk > > Index: package/kernel/modules/virtual.mk > ==================================================================> --- package/kernel/modules/virtual.mk (revision 25855) > +++ package/kernel/modules/virtual.mk (working copy) > @@ -168,7 +168,7 @@ > TITLE:=Xen PCI device frontend > DEPENDS:=@TARGET_x86_xen_domu @LINUX_2_6_37||LINUX_2_6_38 > KCONFIG:=CONFIG_XEN_PCIDEV_FRONTEND > - FILES:=$(LINUX_DIR)/drivers/xen/platform-pci.ko > + FILES:=$(LINUX_DIR)/drivers/pci/xen-pcifront.ko > AUTOLOAD:=$(call AutoLoad,10,xen-pcifront) > endef > > but it still would not include the xen-pcifront.ko file on the *combined-ext4.img.gz > file. Any ideas what I am doing wrong?I don''t remember such problem. Could this module be BTW, I''m checking that rebuilding OpenWRT. When you execute make menuconfig, choose Target=x86 and SubTarget="Xen paravirt Guest". Then go to Kernel Modules -> Virtualisation Support Xen PCI frontend is disabled by default. Did you activated it ? OpenWRT global config cat be found in .config: # grep xen -i .config CONFIG_TARGET_x86_xen_domu=y CONFIG_TARGET_x86_xen_domu_Default=y CONFIG_DEFAULT_kmod-xen-evtchn=y CONFIG_DEFAULT_kmod-xen-fs=y CONFIG_DEFAULT_kmod-xen-kbddev=y CONFIG_DEFAULT_kmod-xen-netdev=y CONFIG_X86_GRUB_BOOTOPTS="xencons=hvc" CONFIG_PACKAGE_kmod-xen-evtchn=y # CONFIG_PACKAGE_kmod-xen-fbdev is not set CONFIG_PACKAGE_kmod-xen-fs=y CONFIG_PACKAGE_kmod-xen-kbddev=y CONFIG_PACKAGE_kmod-xen-netdev=y CONFIG_PACKAGE_kmod-xen-pcidev=y Other way may be to fill target/linux/x86/xen_domu/config-default with Xen CONFIG_* values. This will be used as kernel config Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> I don''t remember such problem. Could this module be > > BTW, I''m checking that rebuilding OpenWRT.OK. To eliminate this being the kernel, I took the trunk/build_dir/toolchain-i386_gcc-linaro_uClibc-0.9.32/linux-2.6.37 (which has a bunch of patches on top of stock 2.6.37) compiled it outside the OpenWRT and tried to passthrough the same adapter. It worked. But let me try to follow your directions below tomorrow and see what progress I can make.> > When you execute make menuconfig, choose Target=x86 and SubTarget="Xen > paravirt Guest". Then go to Kernel Modules -> Virtualisation Support > Xen PCI frontend is disabled by default. Did you activated it ? > > OpenWRT global config cat be found in .config: > # grep xen -i .config > CONFIG_TARGET_x86_xen_domu=y > CONFIG_TARGET_x86_xen_domu_Default=y > CONFIG_DEFAULT_kmod-xen-evtchn=y > CONFIG_DEFAULT_kmod-xen-fs=y > CONFIG_DEFAULT_kmod-xen-kbddev=y > CONFIG_DEFAULT_kmod-xen-netdev=y > CONFIG_X86_GRUB_BOOTOPTS="xencons=hvc" > CONFIG_PACKAGE_kmod-xen-evtchn=y > # CONFIG_PACKAGE_kmod-xen-fbdev is not set > CONFIG_PACKAGE_kmod-xen-fs=y > CONFIG_PACKAGE_kmod-xen-kbddev=y > CONFIG_PACKAGE_kmod-xen-netdev=y > CONFIG_PACKAGE_kmod-xen-pcidev=y > > > Other way may be to fill target/linux/x86/xen_domu/config-default with > Xen CONFIG_* values. This will be used as kernel config > > Regards, > JB_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello Konrad, Le 03/03/2011 23:58, Konrad Rzeszutek Wilk a écrit :>> I don''t remember such problem. Could this module be >> >> BTW, I''m checking that rebuilding OpenWRT. > > OK. > > To eliminate this being the kernel, I took the > > trunk/build_dir/toolchain-i386_gcc-linaro_uClibc-0.9.32/linux-2.6.37 > (which has a bunch of patches on top of stock 2.6.37) > > compiled it outside the OpenWRT and tried to passthrough the same > adapter. It worked. But let me try to follow your directions > below tomorrow and see what progress I can make.Hello, One tought or two between wake-up and leaving for work :-) trunk/build_dir/toolchain-i386_gcc-linaro_uClibc-0.9.32/linux-2.6.37 is kernel builddir. So, if it works from here, there''s no reason it finaly doesn''t. One tought: in "make menuconfig", you will setup the Openwrt''s package/features list. If you have one package marked "M", that means it''ll be build as separate package (*.opkg). If marked with "y", it''ll be integrated within FS image. For you kernel module xen-pcifront, you have to make sure that: - it''s marked in trunk/.config - it''s marked in kernel config file You can check after compliation: ls -l build_dir/linux-x86_xen_domu/packages/ipkg-x86/kmod-xen-pcidev/ total 12 drwxr-xr-x 2 jbfavre jbfavre 4096 Mar 4 01:09 CONTROL drwxr-xr-x 3 jbfavre jbfavre 4096 Mar 4 01:09 etc drwxr-xr-x 3 jbfavre jbfavre 4096 Mar 4 01:09 lib jbfavre@xps-103:~/openwrt/trunk$ ls -l build_dir/linux-x86_xen_domu/packages/ipkg-x86/kmod-xen-pcidev/lib/modules/2.6.37/platform-pci.ko -rw-r--r-- 1 jbfavre jbfavre 4936 Mar 4 01:09 build_dir/linux-x86_xen_domu/packages/ipkg-x86/kmod-xen-pcidev/lib/modules/2.6.37/platform-pci.ko So, not sure you patch is mandatory, is it ? Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Mar 04, 2011 at 08:25:49AM +0100, Jean Baptiste Favre wrote:> Hello Konrad, > > Le 03/03/2011 23:58, Konrad Rzeszutek Wilk a écrit : > >> I don''t remember such problem. Could this module be > >> > >> BTW, I''m checking that rebuilding OpenWRT. > > > > OK. > > > > To eliminate this being the kernel, I took the > > > > trunk/build_dir/toolchain-i386_gcc-linaro_uClibc-0.9.32/linux-2.6.37 > > (which has a bunch of patches on top of stock 2.6.37) > > > > compiled it outside the OpenWRT and tried to passthrough the same > > adapter. It worked. But let me try to follow your directions > > below tomorrow and see what progress I can make. > Hello, > One tought or two between wake-up and leaving for work :-)I haven''t gotten back on this - sorry about this. The other thing that might be going wrong is that CONFIG_PCI_QUIRKS is not turned on in the OpenWRT build. Perhaps there are some work-arounds for the driver that aren''t enabled with OpenWRT build? You could try to build a debian kernel using the OpenWRT .config and seeing if it works right?> > trunk/build_dir/toolchain-i386_gcc-linaro_uClibc-0.9.32/linux-2.6.37 > is kernel builddir. So, if it works from here, there''s no reason it > finaly doesn''t. > > One tought: in "make menuconfig", you will setup the Openwrt''s > package/features list. If you have one package marked "M", that means > it''ll be build as separate package (*.opkg). If marked with "y", it''ll > be integrated within FS image. > > For you kernel module xen-pcifront, you have to make sure that: > - it''s marked in trunk/.config > - it''s marked in kernel config file > > You can check after compliation: > ls -l build_dir/linux-x86_xen_domu/packages/ipkg-x86/kmod-xen-pcidev/ > total 12 > drwxr-xr-x 2 jbfavre jbfavre 4096 Mar 4 01:09 CONTROL > drwxr-xr-x 3 jbfavre jbfavre 4096 Mar 4 01:09 etc > drwxr-xr-x 3 jbfavre jbfavre 4096 Mar 4 01:09 lib > jbfavre@xps-103:~/openwrt/trunk$ ls -l > build_dir/linux-x86_xen_domu/packages/ipkg-x86/kmod-xen-pcidev/lib/modules/2.6.37/platform-pci.ko > > -rw-r--r-- 1 jbfavre jbfavre 4936 Mar 4 01:09 > build_dir/linux-x86_xen_domu/packages/ipkg-x86/kmod-xen-pcidev/lib/modules/2.6.37/platform-pci.ko > > So, not sure you patch is mandatory, is it ?Probably not. Will get back to this after the XenHackothon.> > Regards, > JB > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, Sorry for long silence: had some hollidays :) On 16/03/2011 04:14, Konrad Rzeszutek Wilk wrote:> On Fri, Mar 04, 2011 at 08:25:49AM +0100, Jean Baptiste Favre wrote: >> Hello Konrad, >> >> Le 03/03/2011 23:58, Konrad Rzeszutek Wilk a écrit : >>>> I don''t remember such problem. Could this module be >>>> >>>> BTW, I''m checking that rebuilding OpenWRT. >>> >>> OK. >>> >>> To eliminate this being the kernel, I took the >>> >>> trunk/build_dir/toolchain-i386_gcc-linaro_uClibc-0.9.32/linux-2.6.37 >>> (which has a bunch of patches on top of stock 2.6.37) >>> >>> compiled it outside the OpenWRT and tried to passthrough the same >>> adapter. It worked. But let me try to follow your directions >>> below tomorrow and see what progress I can make. >> Hello, >> One tought or two between wake-up and leaving for work :-) > > I haven''t gotten back on this - sorry about this. > > The other thing that might be going wrong is that CONFIG_PCI_QUIRKS is not > turned on in the OpenWRT build. Perhaps there are some work-arounds for the > driver that aren''t enabled with OpenWRT build? You could try to > build a debian kernel using the OpenWRT .config and seeing if > it works right?In my setup, CONFIG_PCI_QUIRKS is enabled. As default compilation did not worked, I took all XEN and PCI related config options from Debian kernel and manually added them in target/linux/x86/xen_domu/config-default As a consequence, its activated in final kernel but with no change :(>> trunk/build_dir/toolchain-i386_gcc-linaro_uClibc-0.9.32/linux-2.6.37 >> is kernel builddir. So, if it works from here, there''s no reason it >> finaly doesn''t. >> >> One tought: in "make menuconfig", you will setup the Openwrt''s >> package/features list. If you have one package marked "M", that means >> it''ll be build as separate package (*.opkg). If marked with "y", it''ll >> be integrated within FS image. >> >> For you kernel module xen-pcifront, you have to make sure that: >> - it''s marked in trunk/.config >> - it''s marked in kernel config file >> >> You can check after compliation: >> ls -l build_dir/linux-x86_xen_domu/packages/ipkg-x86/kmod-xen-pcidev/ >> total 12 >> drwxr-xr-x 2 jbfavre jbfavre 4096 Mar 4 01:09 CONTROL >> drwxr-xr-x 3 jbfavre jbfavre 4096 Mar 4 01:09 etc >> drwxr-xr-x 3 jbfavre jbfavre 4096 Mar 4 01:09 lib >> jbfavre@xps-103:~/openwrt/trunk$ ls -l >> build_dir/linux-x86_xen_domu/packages/ipkg-x86/kmod-xen-pcidev/lib/modules/2.6.37/platform-pci.ko >> >> -rw-r--r-- 1 jbfavre jbfavre 4936 Mar 4 01:09 >> build_dir/linux-x86_xen_domu/packages/ipkg-x86/kmod-xen-pcidev/lib/modules/2.6.37/platform-pci.ko >> >> So, not sure you patch is mandatory, is it ? > > Probably not. > > Will get back to this after the XenHackothon.Hope this goes well :) Regards, JB _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel