Dr. Volker Jaenisch
2008-Sep-21 22:00 UTC
[Xen-users] Good unidirectional TCP performance, weird asymetric performance going bidirectional
Hi Xen-Users! After several hours of tuning and testing I stuck with a serious performance degration doing bidirectional tcp networking between dom0 to domU as well as domU to domU. Overview: ======dom0 to domU: ------------------------- unidirectional : dom0 -> domU : 578 Mbits/sec dom0 <- domU : 1.22 Gbits/sec Cool isnt it? bidirectional: dom0 <=> domU: dom0 -> domU : 1.22 Gbits/sec dom0 <- domU : 38.2 Mbits/sec Ups! But things it can become worse... domU1 to domU2: ------------------------------ unidirectional: domU1 -> domU2 : 410,2 Mbits/sec domU1 <- domU2 : 378.1 Mbits/sec Can easily live with that. bidirectional: domU1 <=> domU2 : domU1 -> domU2 : 42,3 Mbits/sec domU1 <- domU2 : 38.2 Mbits/sec But what is that? Some problems looking similiar to that have been discussed in postings in this list and elsewhere. I have read lots of them. But none of the mentioned solutions (TCP-tuning, ethtool tweaking, etc) have helped to get rid of this behavior. May be I missed something. This behavior is reproducable with xen 3.1, xen 3.2 on two different machines. * May anybody confirm this findings? * Anybody an idea? Any help appreciated. Best Regards, Volker In the following the details ... iperf yields the following reproductable numbers.>From dom0 to domU:=============zeus:/etc/xen# iperf -c 192.168.2.22 -p5555 ------------------------------------------------------------ Client connecting to 192.168.2.22, TCP port 5555 TCP window size: 27.2 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.2.20 port 59804 connected with 192.168.2.22 port 5555 [ 3] 0.0-10.0 sec 689 MBytes 578 Mbits/sec>From domU to dom0:=============apollo:~# iperf -c 192.168.2.20 -p6666 -t60 ------------------------------------------------------------ Client connecting to 192.168.2.20, TCP port 6666 TCP window size: 23.3 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.2.22 port 36007 connected with 192.168.2.20 port 6666 [ 3] 0.0-60.0 sec 8.55 GBytes 1.22 Gbits/sec So far nothing to complain about. But domU to dom0 full duplex (both directions simultaneous) ================apollo:~# iperf -c 192.168.2.20 -p6666 -t60 -d ------------------------------------------------------------ Client connecting to 192.168.2.20, TCP port 6666 TCP window size: 23.3 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.2.22 port 36007 connected with 192.168.2.20 port 6666 [ 4] local 192.168.2.22 port 5555 connected with 192.168.2.20 port 34223 [ 3] 0.0-60.0 sec 8.55 GBytes 1.22 Gbits/sec [ 4] 0.0-60.0 sec 274 MBytes 38.2 Mbits/sec This is weird. The domU to dom0 performance is still there. But the dom0 to domU performance drops to less than 10%. Same holds for domU to domU communication: About 400MBit/sec in every one direction and 40Mbit/sec bidirectionally. This behavior is independent of the client server role of iperf. Same happens if I start two simultaneous seperate iperf runs in opposite direction by hand. This eliminates iperf as source of the problem. The test-setup is as follows: * AMD64 Opteron Dual Core server. * ASUS Server-Mainboard. * 4GB RAM. * Running debian etch. Debian Kernel 2.6.18-6-xen-amd64 for dom0 and domUs. * xen3.2 hypervisor from debian package. (Same behavior with xen3.1) * Dom0 2 GB RAM, tagged on CPU 0 * DomU 2 GB RAM, tagged on CPU 1 * Network connected via bridge br0. bridge name bridge id STP enabled interfaces br0 8000.001bfcdbd279 no eth0 vif9.0 ethtool -K eth tx off applied on the domUs interfaces to prevent TCP checksum errors. Checked with tcpdump for errors. No obvisouly errors in the stream. A few out of order packages but nothing serious. One fact is remarkable: The CPU utilisation is lower on the domU side and strikingly low for the domU to domU case. dom0 to domU case: dom0 domU 96% 80% domU to domU case: domU domU 15% 15% (Data from xentop) I''m not sure * if this low cpu utilisation is due to the low network performance OR * if something limits the CPU bandwith in the domU and so causes the degraded network performance?? All Machines (dom0, and domUs) have exactly the same TCP tuning parameters. zeus:/etc/xen# sysctl -a | grep net.core net.core.netdev_budget = 300 net.core.somaxconn = 128 net.core.xfrm_aevent_rseqth = 2 net.core.xfrm_aevent_etime = 10 net.core.optmem_max = 20480 net.core.message_burst = 10 net.core.message_cost = 5 net.core.netdev_max_backlog = 1000 net.core.dev_weight = 64 net.core.rmem_default = 126976 net.core.wmem_default = 126976 net.core.rmem_max = 131071 net.core.wmem_max = 131071 zeus:/etc/xen# sysctl -a | grep net.ipv4.tcp net.ipv4.tcp_slow_start_after_idle = 1 net.ipv4.tcp_dma_copybreak = 4096 net.ipv4.tcp_workaround_signed_windows = 0 net.ipv4.tcp_base_mss = 512 net.ipv4.tcp_mtu_probing = 0 net.ipv4.tcp_abc = 0 net.ipv4.tcp_congestion_control = bic net.ipv4.tcp_tso_win_divisor = 3 net.ipv4.tcp_moderate_rcvbuf = 1 net.ipv4.tcp_no_metrics_save = 0 net.ipv4.tcp_low_latency = 0 net.ipv4.tcp_frto = 0 net.ipv4.tcp_tw_reuse = 0 net.ipv4.tcp_adv_win_scale = 2 net.ipv4.tcp_app_win = 31 net.ipv4.tcp_rmem = 4096 87380 4194304 net.ipv4.tcp_wmem = 4096 16384 4194304 net.ipv4.tcp_mem = 196608 262144 3932160 net.ipv4.tcp_dsack = 1 net.ipv4.tcp_ecn = 0 net.ipv4.tcp_reordering = 3 net.ipv4.tcp_fack = 1 net.ipv4.tcp_orphan_retries = 0 net.ipv4.tcp_max_syn_backlog = 1024 net.ipv4.tcp_rfc1337 = 0 net.ipv4.tcp_stdurg = 0 net.ipv4.tcp_abort_on_overflow = 0 net.ipv4.tcp_tw_recycle = 0 net.ipv4.tcp_syncookies = 0 net.ipv4.tcp_fin_timeout = 60 net.ipv4.tcp_retries2 = 15 net.ipv4.tcp_retries1 = 3 net.ipv4.tcp_keepalive_intvl = 75 net.ipv4.tcp_keepalive_probes = 9 net.ipv4.tcp_keepalive_time = 7200 net.ipv4.tcp_max_tw_buckets = 180000 net.ipv4.tcp_max_orphans = 65536 net.ipv4.tcp_synack_retries = 5 net.ipv4.tcp_syn_retries = 5 net.ipv4.tcp_retrans_collapse = 1 net.ipv4.tcp_sack = 1 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_timestamps = 1 TCP-Tuning has no influence on this behavior. Send-, and Receive-Buffer-Sizes, Backlog length, txqueuelength etc. does not shift this behavior a bit. The bottleneck is nor the machine nor the bridge-setup. The same machine using the same bridge setup runs that fast in bidirectional tests ------------------------------------------------------------ Client connecting to 192.168.2.202, TCP port 5555 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 5] local 192.168.2.200 port 50169 connected with 192.168.2.202 port 5555 [ 4] local 192.168.2.200 port 5555 connected with 192.168.2.202 port 53744 [ ID] Interval Transfer Bandwidth [ 5] 0.0-10.0 sec 830 MBytes 696 Mbits/sec [ 4] 0.0-10.0 sec 1.02 GBytes 878 Mbits/sec with two openVZ containers having real veth interfaces. We tested openVZ for quality zope application hosting to get rid of the kernel RAM overhead of XEN. We expected the network of openVZ to be weak and half-baked. Please don''t get me wrong : this is no XEN bashing attempt. We have XEN in production since years and like to have it as long as it stays open source and is avaible in recent kernels :-). We simply like to understand what our xen machines do. Best Regards Volker -- =================================================== inqbus it-consulting +49 ( 341 ) 5643800 Dr. Volker Jaenisch http://www.inqbus.de Herloßsohnstr. 12 0 4 1 5 5 Leipzig N O T - F Ä L L E +49 ( 170 ) 3113748 =================================================== _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2008-Sep-22 08:11 UTC
Re: [Xen-users] Good unidirectional TCP performance, weird asymetric performance going bidirectional
On Mon, Sep 22, 2008 at 12:00:28AM +0200, Dr. Volker Jaenisch wrote:> Hi Xen-Users! > > After several hours of tuning and testing I stuck with a serious > performance degration > doing bidirectional tcp networking between dom0 to domU as well as domU > to domU. > > Overview: > ======> dom0 to domU: > ------------------------- > unidirectional : > dom0 -> domU : 578 Mbits/sec > dom0 <- domU : 1.22 Gbits/sec > Cool isnt it? > > bidirectional: > dom0 <=> domU: > dom0 -> domU : 1.22 Gbits/sec > dom0 <- domU : 38.2 Mbits/sec > > Ups! But things it can become worse... > > domU1 to domU2: > ------------------------------ > unidirectional: > domU1 -> domU2 : 410,2 Mbits/sec > domU1 <- domU2 : 378.1 Mbits/sec > Can easily live with that. > > bidirectional: > domU1 <=> domU2 : > domU1 -> domU2 : 42,3 Mbits/sec > domU1 <- domU2 : 38.2 Mbits/sec > But what is that? > > Some problems looking similiar to that have been discussed in postings > in this list and elsewhere. I have read lots of them. > But none of the mentioned solutions (TCP-tuning, ethtool tweaking, etc) > have helped to get rid of this behavior. > May be I missed something. > > This behavior is reproducable with xen 3.1, xen 3.2 on two different > machines. > > * May anybody confirm this findings? > * Anybody an idea? >Have you tried changing the vif* interface txqueuelen in dom? I''m wondering if that would help.. "ifconfig vifX.Y txqueuelen 256" or even bigger value.. default value of 64 isn''t too much.. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2008-Sep-22 09:04 UTC
Re: [Xen-users] Good unidirectional TCP performance, weird asymetric performance going bidirectional
On Mon, Sep 22, 2008 at 11:11:37AM +0300, Pasi Kärkkäinen wrote:> > > > * May anybody confirm this findings? > > * Anybody an idea? > > > > Have you tried changing the vif* interface txqueuelen in dom? I''m wondering > if that would help.. > > "ifconfig vifX.Y txqueuelen 256" or even bigger value.. default value of 64 > isn''t too much.. >Whoops, I was supposed to write "txqueuelen in dom0".. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Michael Monnerie
2008-Sep-22 10:06 UTC
Re: [Xen-users] Good unidirectional TCP performance, weird asymetric performance going bidirectional
On Montag, 22. September 2008 Dr. Volker Jaenisch wrote:> After several hours of tuning and testing I stuckI didn''t test myself, but look at this: http://www.xen.org/files/xensummitboston08/Deshane-XenSummit08- Slides.pdf We used VMware until recently and want to switch to XEN now, but currently I find it not-so-easy-as-I-thought. :-( mfg zmi -- // Michael Monnerie, Ing.BSc ----- http://it-management.at // Tel: 0660 / 415 65 31 .network.your.ideas. // PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import" // Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4 // Keyserver: www.keyserver.net Key-ID: 1C1209B4 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Dr. Volker Jaenisch
2008-Sep-22 12:54 UTC
Re: [Xen-users] Good unidirectional TCP performance, weird asymetric performance going bidirectional
Hello Pasi! Pasi Kärkkäinen wrote:> Hmm.. did you play with the domain weights in credit scheduler settings? > > Maybe dom0 needs more CPU time to process/forward packets? >Mh. My feeling is that the domU does not get enougth CPU to process/forward. This feeling is due to the fact that sending data from the domU to the dom0 is very fast. The opposite direction is slower. I played with the sched-credit. Raising and lowering the dom0 weigth by a factor of 2 does not change the behavior.> Other than that can''t really think about much else.. maybe this should be > asked on xen-devel list.. >Good idea. Will do that. Thank you very much. Best regards, Volker -- =================================================== inqbus it-consulting +49 ( 341 ) 5643800 Dr. Volker Jaenisch http://www.inqbus.de Herloßsohnstr. 12 0 4 1 5 5 Leipzig N O T - F Ä L L E +49 ( 170 ) 3113748 =================================================== _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Dr. Volker Jaenisch
2008-Sep-22 12:54 UTC
Re: [Xen-users] Good unidirectional TCP performance, weird asymetric performance going bidirectional
Hi Pasi! THanks for the fast reply! Pasi Kärkkäinen schrieb:> On Mon, Sep 22, 2008 at 11:11:37AM +0300, Pasi Kärkkäinen wrote: > >> Have you tried changing the vif* interface txqueuelen in dom? I''m wondering >> if that would help.. >>Yes I did. I changed the txqueulength from 0 (yes ifconfig shows me ZERO) to 1000. Should I change the txqueulength of the bridge too? zeus:~# ifconfig br0 Link encap:Ethernet HWaddr 00:1B:FC:DB:D2:79 inet addr:192.168.2.20 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: fe80::21b:fcff:fedb:d279/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2758816 errors:0 dropped:0 overruns:0 frame:0 TX packets:831349 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:3711167604 (3.4 GiB) TX bytes:861930334 (822.0 MiB) eth0 Link encap:Ethernet HWaddr 00:1B:FC:DB:D2:79 inet6 addr: fe80::21b:fcff:fedb:d279/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1333 errors:0 dropped:0 overruns:0 frame:0 TX packets:908 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:178350 (174.1 KiB) TX bytes:124770 (121.8 KiB) Interrupt:23 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:34 errors:0 dropped:0 overruns:0 frame:0 TX packets:34 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2576 (2.5 KiB) TX bytes:2576 (2.5 KiB) vif2.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2757485 errors:0 dropped:0 overruns:0 frame:0 TX packets:829015 errors:0 dropped:1588 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3711013112 (3.4 GiB) TX bytes:860609840 (820.7 MiB) Did no change at all (txqueuelength of vif and br0 changed to 1000): apollo2:~# dom0 <=> domU iperf -c 192.168.2.20 -p5555 -t5 ------------------------------------------------------------ Client connecting to 192.168.2.20, TCP port 5555 TCP window size: 23.3 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.2.22 port 39015 connected with 192.168.2.20 port 5555 [ 3] 0.0- 5.0 sec 788 MBytes 1.32 Gbits/sec apollo2:~# iperf -c 192.168.2.20 -p5555 -t5 -d ------------------------------------------------------------ Server listening on TCP port 5555 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ ------------------------------------------------------------ Client connecting to 192.168.2.20, TCP port 5555 TCP window size: 77.7 KByte (default) ------------------------------------------------------------ [ 5] local 192.168.2.22 port 39016 connected with 192.168.2.20 port 5555 [ 4] local 192.168.2.22 port 5555 connected with 192.168.2.20 port 48509 [ 5] 0.0- 5.0 sec 894 MBytes 1.50 Gbits/sec [ 4] 0.0- 5.1 sec 25.3 MBytes 41.4 Mbits/sec domU <=> dom0 apollo2:~# iperf -s -p5555 ------------------------------------------------------------ Server listening on TCP port 5555 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 192.168.2.22 port 5555 connected with 192.168.2.20 port 59129 ------------------------------------------------------------ Client connecting to 192.168.2.20, TCP port 5555 TCP window size: 38.8 KByte (default) ------------------------------------------------------------ [ 6] local 192.168.2.22 port 39017 connected with 192.168.2.20 port 5555 [ 6] 0.0-10.0 sec 1.68 GBytes 1.44 Gbits/sec [ 4] 0.0-10.0 sec 40.3 MBytes 33.8 Mbits/sec dom0 -> domU [ 5] local 192.168.2.22 port 5555 connected with 192.168.2.20 port 59130 [ 5] 0.0-10.0 sec 703 MBytes 590 Mbits/sec In openVZ the txqueuelength is also 0 and the bidirectional transfer rate is not asymetric. Any other ideas? Best Regards Volker -- =================================================== inqbus it-consulting +49 ( 341 ) 5643800 Dr. Volker Jaenisch http://www.inqbus.de Herloßsohnstr. 12 0 4 1 5 5 Leipzig N O T - F Ä L L E +49 ( 170 ) 3113748 =================================================== _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users