Hi! I have bridging problems with the Dom0 kernels from Jeremy’s tree. I wrote a mail to xen-user (MSG-ID <20091220T1944.GA.ab998.stse@fsing.rootsland.net>, 20 Dec 2009), but without solutions. So I try xen-devel this time. My hardware setup: A PC with two NICs (Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller) is used as working environment (Dom0) and as firewall/proxy/DSL-router (DomU). The two NICs are bridged between Dom0 and DomU. Bridge eth0 containing peth0 and vif1.0 has an IP address in Dom0 and DomU. The DomU IP address is the gateway address in Dom0. Bridge xenbr1 containing eth1 and vif1.1 has no IP address in Dom0 and DomU and is only used to connect the DSL modem to DomU. The IP address is given to the PPP interface in DomU. Linux distribution is Debian/Testing (64bit) with XEN version 3.4.2 in December and 3.4.3rc3 now. The kernels are always self-compiled. My working setup: Dom0 with kernel 2.6.29.5 with xen-patches-2.6.29-6.tar.bz2 and DomU with standard kernel 2.6.32.x (and the 2.6.29.5 xen kernel before). The hypervisor was 3.4.2 and is now 3.4.3rc3. Here everything works as expected. DomU acts as firewall and is using correct masquerading for all internet traffic. My non-working setup: Dom0 with the PV-Ops kernel from Jeremy’s tree (I tried the follwoing kernels: 2.6.31.5-00500-g34013be, 2.6.31.6-00696-g41a0695 (tested in December) and now from xen/stable the versions 2.6.32.10-02792-gf112549 and 2.6.32.10-02798-gd945b01). DomU kernel and hypervisor are the same as in the working setup. What is working? IP connection between Dom0 and DomU is working and between DomU and the internet. Traffic from Dom0 to the internet is working if DomU is used as a proxy (e.g. HTTP traffice with a squid in DomU). What is not working? Direct IP connection between Dom0 and the internet (tested with ping and „telnet <host> <port>”. If I trace in DomU I see the packets leaving the ppp0 interface (correctly masqueraded), but I see no answering packets. If I trace in Dom0 using the bridge interfaces between the DSL modem and DomU (xenbr1, eth1, vif1.1, see hardware setup above), I don’t see the packets anymore. I only see packets from traffic generated directly by DomU. The DomU configuration between the working and non-working setup is not changed, only the Dom0 kernel is changed. So if anyone has an idea, what this could be and how to fix it, I will be glad. Further information: The NIC and the bridge driver are the same in all kernels from 2.6.29.5 until 2.6.32.10: osgiliath:~# ethtool -i eth1 driver: r8169 version: 2.3LK-NAPI firmware-version: bus-info: 0000:03:00.0 osgiliath:~# ethtool -i xenbr1 driver: bridge version: 2.3 firmware-version: N/A bus-info: N/A The only difference in the output of „ethtool eth1” are additional information about „link partner advertised modes” in the 2.6.3x kernels. „ethtool -k eth1” shows the error message „Cannot get device flags: Operation not supported” in the working setup for the working Dom0 kernel. All other output is identical in all kernel versions: osgiliath:~# ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: off scatter-gather: off tcp-segmentation-offload: off udp-fragmentation-offload: off generic-segmentation-offload: off generic-receive-offload: off large-receive-offload: off Switching rx-checksumming off does not help. dmesg and the log files don’t show error messages in the non-working setup. Shade and sweet water! Stephan -- | Stephan Seitz E-Mail: stse@fsing.rootsland.net | | PGP Public Keys: http://fsing.rootsland.net/~stse/pgp.html | _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
S.H. Verbrugge
2010-Mar-29 18:18 UTC
Re: [Xen-devel] Jeremy''s GIT-tree and network problems
On Mon, Mar 29, 2010 at 05:40:23PM +0200, Stephan Seitz wrote:> Hi! > > I have bridging problems with the Dom0 kernels from Jeremy???s tree. > I wrote a mail to xen-user (MSG-ID > <20091220T1944.GA.ab998.stse@fsing.rootsland.net>, 20 Dec 2009), but > without solutions. So I try xen-devel this time. > > > My hardware setup: > A PC with two NICs (Realtek Semiconductor Co., Ltd. RTL8111/8168B > PCI Express Gigabit Ethernet controller) is used as working > environment (Dom0) and as firewall/proxy/DSL-router (DomU). > The two NICs are bridged between Dom0 and DomU. > Bridge eth0 containing peth0 and vif1.0 has an IP address in Dom0 > and DomU. The DomU IP address is the gateway address in Dom0. > Bridge xenbr1 containing eth1 and vif1.1 has no IP address in Dom0 > and DomU and is only used to connect the DSL modem to DomU. The IP > address is given to the PPP interface in DomU. > Linux distribution is Debian/Testing (64bit) with XEN version 3.4.2 > in December and 3.4.3rc3 now. The kernels are always self-compiled. > > > My working setup: > Dom0 with kernel 2.6.29.5 with xen-patches-2.6.29-6.tar.bz2 and DomU > with standard kernel 2.6.32.x (and the 2.6.29.5 xen kernel before). > The hypervisor was 3.4.2 and is now 3.4.3rc3. > Here everything works as expected. DomU acts as firewall and is > using correct masquerading for all internet traffic. > > > My non-working setup: > Dom0 with the PV-Ops kernel from Jeremy???s tree (I tried the > follwoing kernels: 2.6.31.5-00500-g34013be, 2.6.31.6-00696-g41a0695 > (tested in December) and now from xen/stable the versions > 2.6.32.10-02792-gf112549 and 2.6.32.10-02798-gd945b01). DomU kernel > and hypervisor are the same as in the working setup. > > What is working? > IP connection between Dom0 and DomU is working and between DomU and > the internet. Traffic from Dom0 to the internet is working if DomU > is used as a proxy (e.g. HTTP traffice with a squid in DomU). > > What is not working? > Direct IP connection between Dom0 and the internet (tested with ping > and ???telnet <host> <port>???. > If I trace in DomU I see the packets leaving the ppp0 interface > (correctly masqueraded), but I see no answering packets. > If I trace in Dom0 using the bridge interfaces between the DSL modem > and DomU (xenbr1, eth1, vif1.1, see hardware setup above), I don???t > see the packets anymore. I only see packets from traffic generated > directly by DomU. > The DomU configuration between the working and non-working setup is > not changed, only the Dom0 kernel is changed. > > > So if anyone has an idea, what this could be and how to fix it, I > will be glad. > > > Further information: > The NIC and the bridge driver are the same in all kernels from > 2.6.29.5 until 2.6.32.10: > > osgiliath:~# ethtool -i eth1 > driver: r8169 > version: 2.3LK-NAPI > firmware-version: > bus-info: 0000:03:00.0 > osgiliath:~# ethtool -i xenbr1 > driver: bridge > version: 2.3 > firmware-version: N/A > bus-info: N/A > > The only difference in the output of ???ethtool eth1??? are > additional information about ???link partner advertised modes??? in > the 2.6.3x kernels. > > ???ethtool -k eth1??? shows the error message ???Cannot get device > flags: Operation not supported??? in the working setup for the > working Dom0 kernel. All other output is identical in all kernel > versions: > > osgiliath:~# ethtool -k eth1 > Offload parameters for eth1: > rx-checksumming: on > tx-checksumming: off > scatter-gather: off > tcp-segmentation-offload: off > udp-fragmentation-offload: off > generic-segmentation-offload: off > generic-receive-offload: off > large-receive-offload: off > > Switching rx-checksumming off does not help. > > dmesg and the log files don???t show error messages in the > non-working setup. > > Shade and sweet water! > > Stephan > > -- > | Stephan Seitz E-Mail: stse@fsing.rootsland.net | > | PGP Public Keys: http://fsing.rootsland.net/~stse/pgp.html |Hello :-) I seem to have the exact same problem. Could you check your dmesg (dom0) to see if you have similar messages? http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00966.html -- /\/\ Hostingvereniging Soleus | Community-driven < ** > http://soleus.nu | Virtual Private Servers \/\/ Sen (IEF) Verbrugge (CT ProLead) | & more ... _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Mar-29 18:42 UTC
Re: [Xen-devel] Jeremy''s GIT-tree and network problems
On 03/29/2010 08:40 AM, Stephan Seitz wrote:> Hi! > > I have bridging problems with the Dom0 kernels from Jeremy’s tree. I > wrote a mail to xen-user (MSG-ID > <20091220T1944.GA.ab998.stse@fsing.rootsland.net>, 20 Dec 2009), but > without solutions. So I try xen-devel this time. > > > My hardware setup: > A PC with two NICs (Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI > Express Gigabit Ethernet controller) is used as working environment > (Dom0) and as firewall/proxy/DSL-router (DomU). > The two NICs are bridged between Dom0 and DomU. > Bridge eth0 containing peth0 and vif1.0 has an IP address in Dom0 and > DomU. The DomU IP address is the gateway address in Dom0. > Bridge xenbr1 containing eth1 and vif1.1 has no IP address in Dom0 and > DomU and is only used to connect the DSL modem to DomU. The IP address > is given to the PPP interface in DomU. > Linux distribution is Debian/Testing (64bit) with XEN version 3.4.2 in > December and 3.4.3rc3 now. The kernels are always self-compiled. > > > My working setup: > Dom0 with kernel 2.6.29.5 with xen-patches-2.6.29-6.tar.bz2 and DomU > with standard kernel 2.6.32.x (and the 2.6.29.5 xen kernel before). > The hypervisor was 3.4.2 and is now 3.4.3rc3. > Here everything works as expected. DomU acts as firewall and is using > correct masquerading for all internet traffic. > > > My non-working setup: > Dom0 with the PV-Ops kernel from Jeremy’s tree (I tried the follwoing > kernels: 2.6.31.5-00500-g34013be, 2.6.31.6-00696-g41a0695 (tested in > December) and now from xen/stable the versions > 2.6.32.10-02792-gf112549 and 2.6.32.10-02798-gd945b01). DomU kernel > and hypervisor are the same as in the working setup. > > What is working? > IP connection between Dom0 and DomU is working and between DomU and > the internet. Traffic from Dom0 to the internet is working if DomU is > used as a proxy (e.g. HTTP traffice with a squid in DomU). > > What is not working? > Direct IP connection between Dom0 and the internet (tested with ping > and „telnet <host> <port>”. > If I trace in DomU I see the packets leaving the ppp0 interface > (correctly masqueraded), but I see no answering packets. > If I trace in Dom0 using the bridge interfaces between the DSL modem > and DomU (xenbr1, eth1, vif1.1, see hardware setup above), I don’t see > the packets anymore. I only see packets from traffic generated > directly by DomU. > The DomU configuration between the working and non-working setup is > not changed, only the Dom0 kernel is changed. > > > So if anyone has an idea, what this could be and how to fix it, I will > be glad. > > > Further information: > The NIC and the bridge driver are the same in all kernels from > 2.6.29.5 until 2.6.32.10: > > osgiliath:~# ethtool -i eth1 > driver: r8169 > version: 2.3LK-NAPI > firmware-version: > bus-info: 0000:03:00.0 > osgiliath:~# ethtool -i xenbr1 > driver: bridge > version: 2.3 > firmware-version: N/A > bus-info: N/A > > The only difference in the output of „ethtool eth1” are additional > information about „link partner advertised modes” in the 2.6.3x kernels. > > „ethtool -k eth1” shows the error message „Cannot get device flags: > Operation not supported” in the working setup for the working Dom0 > kernel. All other output is identical in all kernel versions: > > osgiliath:~# ethtool -k eth1 > Offload parameters for eth1: > rx-checksumming: on > tx-checksumming: off > scatter-gather: off > tcp-segmentation-offload: off > udp-fragmentation-offload: off > generic-segmentation-offload: off > generic-receive-offload: off > large-receive-offload: off > > Switching rx-checksumming off does not help.Have you tried carpet-bombing the ethtools: turn off everything on all the dom0 interfaces (both the bridge(s) and all the component interfaces) and all the domU interfaces? It does look like some kind of checksum problem (or perhaps other offload?). Fortunately it looks like this is going to get some systematic attention. I''d really like any reasonable (ie, not inherently broken for other reasons) network setup to just work. Thanks, J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stephan Seitz
2010-Mar-30 12:26 UTC
[Xen-devel] Re: Jeremy''s GIT-tree and network problems
On Mon, Mar 29, 2010 at 08:18:15PM +0200, S.H. Verbrugge wrote:>Could you check your dmesg (dom0) to see if you have similar messages? >http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00966.htmlYou mean „Attempting to checksum a non-TCP/UDP packet, dropping a protocol 1 packet”? Yes, I see one shortly before the NFSv4 server is started. But it is only one line and doesn’t show again if I start my connection tests. Shade and sweet water! Stephan -- | Stephan Seitz E-Mail: stse@fsing.rootsland.net | | PGP Public Keys: http://fsing.rootsland.net/~stse/pgp.html | _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stephan Seitz
2010-Mar-30 12:33 UTC
[Xen-devel] Re: Jeremy''s GIT-tree and network problems
Hi Jeremy! On Mon, Mar 29, 2010 at 11:42:40AM -0700, Jeremy Fitzhardinge wrote:>Have you tried carpet-bombing the ethtools: turn off everything on all >the dom0 interfaces (both the bridge(s) and all the component >interfaces) and all the domU interfaces? It does look like some kind of >checksum problem (or perhaps other offload?).Well, the physical interfaces only had RX on. The virtual interfaces in Dom0 and DomU had RX, TX, SG, TSO and GSO on. Trying to switch off RX on the virtual interfaces produced an error message, but after switching off the other features, RX was off as well. But after everything was off (in Dom0 bridge eth0 with peth0 and vif1.0 and bridge xenbr1 with eth1 and vif1.1 and in DomU eth0 and eth1), the setup didn’t start working. So if this is some kind of checksum problem, switching the checksum and offload features off don’t change it. Do you have other suggestions?>Fortunately it looks like this is going to get some systematic >attention. I''d really like any reasonable (ie, not inherently broken for >other reasons) network setup to just work.I hope so. Anf if there are patches to test, I will test it. Shade and sweet water! Stephan -- | Stephan Seitz E-Mail: stse@fsing.rootsland.net | | PGP Public Keys: http://fsing.rootsland.net/~stse/pgp.html | _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Mar-30 14:30 UTC
Re: [Xen-devel] Re: Jeremy''s GIT-tree and network problems
On Tue, Mar 30, 2010 at 02:26:00PM +0200, Stephan Seitz wrote:> On Mon, Mar 29, 2010 at 08:18:15PM +0200, S.H. Verbrugge wrote: >> Could you check your dmesg (dom0) to see if you have similar messages? >> http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00966.html > > You mean „Attempting to checksum a non-TCP/UDP packet, dropping a > protocol 1 packet”? Yes, I see one shortly before the NFSv4 server is > started. But it is only one line and doesn’t show again if I start my > connection tests.So here is an interesting observation (And I am late in this thread so if I am going off on a tangent, please correct me): We compiled both in DomU and Dom0 the kernel to not have any traces of iptables nor igbtables. The performance of scp-ing a tarball from Dom0 to DomU went from 0.5MB to 17MB/s (which is about the disk speed). I don''t know if you feel inclined to actually try such a drastic option to see if that is your culprit or not, but it is worth keeping in the back of your head. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stephan Seitz
2010-Mar-30 19:39 UTC
[Xen-devel] Re: Jeremy''s GIT-tree and network problems
On Tue, Mar 30, 2010 at 10:30:06AM -0400, Konrad Rzeszutek Wilk wrote:>So here is an interesting observation (And I am late in this thread so >if I am going off on a tangent, please correct me): We compiled both in >DomU and Dom0 the kernel to not have any traces of iptables nor >igbtables. The performance of scp-ing a tarball from Dom0 to DomU went >from 0.5MB to 17MB/s (which is about the disk speed).Interesting. Copying a kernel.tar.bz2 from Dom0 to DomU with scp gives a speed of 30.7MB/s, even if both kernels have iptables support.>I don''t know if you feel inclined to actually try such a drastic option >to see if that is your culprit or not, but it is worth keeping in the >back of your head.But I fear, that this is not my problem, and I need iptables in DomU at least for masquerading. Shade and sweet water! Stephan -- | Stephan Seitz E-Mail: stse@fsing.rootsland.net | | PGP Public Keys: http://fsing.rootsland.net/~stse/pgp.html | _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stephan Seitz
2010-May-03 18:48 UTC
[Xen-devel] Re: Jeremy''s GIT-tree and network problems
On Mon, Mar 29, 2010 at 11:42:40AM -0700, Jeremy Fitzhardinge wrote:>Fortunately it looks like this is going to get some systematic >attention. I''d really like any reasonable (ie, not inherently broken for >other reasons) network setup to just work.Do you know what is the status of this problem At the moment I’m using 2.6.33.2 with xen-patches-2.6.33-1.tar.bz2 and everything is working fine. So if you think, the bug is fixed in your tree, I’ll try it. Shade and sweet water! Stephan -- | Stephan Seitz E-Mail: stse@fsing.rootsland.net | | PGP Public Keys: http://fsing.rootsland.net/~stse/pgp.html | _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel