Hi, I''m running Xen 3.1 on Kernel 2.6.22 (Ubuntu 7.10) and have an extremly strange performance-problem. The setup is using vif-route with ip_forward and proxy_arp, so every domU has it''s own vif-Interface. The relevant line in the xen-config all read similiar to this: "vif = [ ''ip=12.34.56.78, mac=00:16:3e:11:22:33'' ]" The guest then have the default-gateway set to an ip in the same subnet (/29) that is an alias for eth0 in the dom0. Network-connectivity is fine, if I download something in the domU, I get near wirespeed (up to 11MB/s). If I download something from the domU (e.g. via apache from a remote site), I get maybe 40kB/s. So I started up tcpdump and I see many Retransmit, Out-of-order, Dup and ICMP "fragmentation needed" packets. As every host on the way between the domU and the testmachine is MTU 1500, I have no idea what causes the problem. I tried setting the domU as well as the dom0 to MTU 1300, without any change. Is this a possible Xen-problem (configuration error on my end?), a problem with the kernel or (rather unlikely as this is a big colocation) a network problem? To be honest, I have no idea what to look for next, so I would really appreciate any ideas. Thanks. -- Fridtjof Busse _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
* Ralf Kaestner <Ralf.Kaestner@web.de>:> Fridtjof Busse wrote: > > So I started up tcpdump and I see many Retransmit, Out-of-order, > > Dup and ICMP "fragmentation needed" packets. As every host on the > > way between > > Are there TCP checksum errors as well in the network trace? > Whats the guest OS, Linux or Windows? > Has the guest enabled any TCP task offloading (if yes try turning it > off)?I haven''t seen any incorrect checksums so far (doesn''t mean there aren''t any). It more looks like the packets get somehow duplicated. The guests are all linux without any special "optimizations", the only parameters I touched are proxy_arp and forwarding. -- Fridtjof Busse _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
* Fridtjof Busse <fbusse@gmx.de>:> Hi, > I''m running Xen 3.1 on Kernel 2.6.22 (Ubuntu 7.10) and have an > extremly strange performance-problem.Some more informations: The domUs are all Linux, running the same kernel/initrd as the dom0. Even the guests have extrem slow network between each other, only remote downloads work fast. Does anybody has an idea what I can look for? Is anybody maybe running a similiar setup and is willing to share some configs? Or is the entire kernel maybe "bogus"? I could even supply a testmachine. Thanks. -- Fridtjof Busse _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
* Fridtjof Busse <fbusse@gmx.de>:> * Fridtjof Busse <fbusse@gmx.de>: > > Hi, > > I''m running Xen 3.1 on Kernel 2.6.22 (Ubuntu 7.10) and have an > > extremly strange performance-problem.Unfortunatly, I didn''t receive any more replays, so some more info: I switched from routing to bridging (basically put a dummy0-interface into my subnet) and I still have the exact same problem! Besides if intact packets, wireshark basically reads like this: TCP Previous segment lost TCP Dup ACK TCP Dup ACK TCP Retransmissions Communication with the dom0 is flawless, so it''s not a generic network problem. I also noticed a window size of only 501 (Win=501 in wireshark). Help! -- Fridtjof Busse _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
* Ralf Kaestner <Ralf.Kaestner@web.de>:> Has the guest enabled any TCP task offloading (if yes try turning it > off)?You already gave me the right hint, without me noticing: ''ethtool -K eth0 tx off'' in the domUs. That immediatly fixed the problem. Thanks! Question remains: Why does it fix the problem? I never had to do that with kernel 2.6.18 (Xen 3.1.0) from XenSource. -- Fridtjof Busse _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I am having a similar issue I see the retransmissions and dup acks basically everything you are seeing. The 2 things that are different on my end is that I''m using centos5 and piranha(for a virtual failover IP). I see the duplicate and retranssmissions on eth1:1(the vip from piranha). Would the below fix for you also apply to me(ethtool -K eth0 tx off). Also would I apply this to the VIP? ethtool -K eth1:1 tx off? Fridtjof Busse wrote:> * Ralf Kaestner <Ralf.Kaestner@web.de>: > >> Has the guest enabled any TCP task offloading (if yes try turning it >> off)? >> > > You already gave me the right hint, without me noticing: > ''ethtool -K eth0 tx off'' in the domUs. > That immediatly fixed the problem. Thanks! > Question remains: Why does it fix the problem? I never had to do that > with kernel 2.6.18 (Xen 3.1.0) from XenSource. > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
* James Wilson <jwilson@transolutions.net>:> I am having a similar issue I see the retransmissions and dup acks > basically everything you are seeing. The 2 things that are different > on my end is that I''m using centos5 and piranha(for a virtual > failover IP). I see the duplicate and retranssmissions on eth1:1(the > vip from piranha). Would the below fix for you also apply to me > (ethtool -K eth0 tx off). Also would I apply this to the VIP? ethtool > -K eth1:1 tx off?Running ethtool in one of my domUs seems to have fixed the problem for the others as well (maybe due to bridge-setup?). You can simply try running ethtool, it should not break anything that''s not already broken :) -- Fridtjof Busse _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users