Artur Schiefer
2006-Mar-09 16:00 UTC
[Re: [Xen-users] Slow domU network (even within domU)]
hello, thank you but that didn''t help neither. another strange observation i made is when do i an iperf within the domu i get the following results: iperf to localhost 2.8 gbit iperf to domu-real-ip 200 mbit (see results at the end). ciao, artur Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ VM_OTRS:~# iperf -c 192.168.70.50 ------------------------------------------------------------ Client connecting to 192.168.70.50, TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.70.50 port 41309 connected with 192.168.70.50 port 5001 [ 4] local 192.168.70.50 port 5001 connected with 192.168.70.50 port 41309 [ 3] 0.0-10.0 sec 239 MBytes 200 Mbits/sec [ 4] 0.0-10.0 sec 239 MBytes 200 Mbits/sec VM_OTRS:~# iperf -c 127.0.0.1 ------------------------------------------------------------ Client connecting to 127.0.0.1, TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 3] local 127.0.0.1 port 56978 connected with 127.0.0.1 port 5001 [ 4] local 127.0.0.1 port 5001 connected with 127.0.0.1 port 56978 [ 3] 0.0-10.0 sec 3.25 GBytes 2.79 Gbits/sec [ 4] 0.0-10.0 sec 3.25 GBytes 2.79 Gbits/sec Am Donnerstag, den 09.03.2006, 09:14 -0500 schrieb Himanshu Raj:> You will have to tune your TCP parameters like window sizes etc. Idon''t have a> ready ref on this - but try the following settings in your domU. > > Following sysctl parameters must be tuned in order to get gigE bw fromdomUs.> Either set via sysctl -w or put in /etc/sysctl.conf. > > # increase TCP maximum buffer size > net.core.rmem_max=16777216 > net.core.wmem_max=16777216 > > # increase Linux autotuning TCP buffer limits > # min, default, and maximum number of bytes to use > net.ipv4.tcp_rmem="4096 87380 16777216" > net.ipv4.tcp_wmem="4096 65536 16777216" > > -Himanshu > > On Wed, Mar 08, 2006 at 02:27:29PM +0100, Artur Schiefer wrote: > > hello philipp, > > > > this is my xentop output when i am runnig the test. do you achieve > > higher rates (should only be limited by memory throughtput). > > > > ciao, > > artur > > > > xentop - 14:21:58 Xen 3.0.1 > > 2 domains: 1 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0 > > shutdown > > Mem: 1048104k total, 1039148k used, 8956k free CPUs: 4 @ 2799MHz > > NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) > > MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) SSID > > Domain-0 -----r 1170 109.3 885948 84.5 no limit > > n/a 4 8 0 0 0 > > vm_apache ------ 690 72.4 131068 12.5 139264 > > 13.3 2 1 1196352 31262 0 > > Am Mittwoch, den 08.03.2006, 12:52 +0100 schrieb Philipp Jggi: > > > > > > Did you recoded the xentop output during your test? How muchmem-max> > > do you have for dom0 and your domU? > > > > > > bye Philipp > > > > > > > > > > > > > > > Artur Schiefer <aschiefer@vub.de> > > > Sent by: > > > xen-users-bounces@lists.xensource.com > > > > > > 03/08/2006 12:47 PM > > > Please respond to > > > aschiefer@vub.de > > > > > > > > > > > > > > > To > > > xen-users > > > <xen-users@lists.xensource.com> > > > cc > > > > > > Subject > > > [Xen-users] Slow > > > domU network > > > > > > > > > > > > > > > > > > > > > > > > > > > hello, > > > > > > when i test network performance between dom0 <-> domU (xen3.0.1tha3> > > debian patches)with iperf i get only about 450 Mbits of throughputand> > > lots of dropped packets on the vif-interface (bridged). Inopposite to> > > that, when i run the same test beetween two dom0 withbonded/teamed> > > gigabit-nics i am able to achieve 1.6 Gbits of throughput. > > > has someone made this obsevation as well (what is yourthroughput)?> > > any solutions ? > > > > > > cheers, > > > artur > > > > > > > > > > > > > > > _______________________________________________ > > > Xen-users mailing list > > > Xen-users@lists.xensource.com > > > http://lists.xensource.com/xen-users > > > > > > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Artur Schiefer
2006-Mar-09 16:26 UTC
Re: [Re: [Xen-users] Slow domU network (even within domU)]
ok, solved the problem for within domu communcation, was an autotuning effect. Am Donnerstag, den 09.03.2006, 17:00 +0100 schrieb Artur Schiefer: _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2006-Mar-09 18:38 UTC
Re: [Re: [Xen-users] Slow domU network (even within domU)]
On Thu, Mar 09, 2006 at 05:26:03PM +0100, Artur Schiefer wrote:> ok, solved the problem for within domu communcation, was an autotuning > effect. >Autotuning effect? Please be more specific because many of the xen users seem to have problems with network performance.. -- Pasi ^ . . Linux / - \ Choice.of.the .Next.Generation. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
hello, somehow the linux network core tries to adjust itself to its environment. as i recently learned theese parameters are optimized for 100 mbit networking. from another poster i got the info that you have to set the following parameters in your "/etc/sysctl.conf" to achieve gbit speed: # increase TCP maximum buffer size net.core.rmem_max=16777216 net.core.wmem_max=16777216 # increase Linux autotuning TCP buffer limits # min, default, and maximum number of bytes to use net.ipv4.tcp_rmem=4096 87380 16777216 net.ipv4.tcp_wmem=4096 65536 16777216 BUT THIS ONLY SOLVES THE PROBLEM FOR WITHIN DOMU COMMUNCATION (i.e. sender and receiver are the SAME virtual maschine, which should be a theoretical problem in most cases). I tuned domU and dom0 this way problem between dom0 and domU still exists. i also played with tcp window sizes (-w option in iperf) from 64kb to 2MB but the performance differences are negligable. Am Donnerstag, den 09.03.2006, 20:38 +0200 schrieb Pasi Kärkkäinen:> On Thu, Mar 09, 2006 at 05:26:03PM +0100, Artur Schiefer wrote: > > ok, solved the problem for within domu communcation, was an autotuning > > effect. > > > > Autotuning effect? Please be more specific because many of the xen users > seem to have problems with network performance.. > > -- Pasi > > ^ > . . > Linux > / - \ > Choice.of.the > .Next.Generation. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Mar 10, 2006 at 10:43:17AM +0100, Artur Schiefer wrote:> hello, > > somehow the linux network core tries to adjust itself to its > environment. as i recently learned theese parameters are optimized for > 100 mbit networking. from another poster i got the info that you have to > set the following parameters in your "/etc/sysctl.conf" to achieve gbit > speed: > # increase TCP maximum buffer size > net.core.rmem_max=16777216 > net.core.wmem_max=16777216 > > # increase Linux autotuning TCP buffer limits > # min, default, and maximum number of bytes to use > net.ipv4.tcp_rmem=4096 87380 16777216 > net.ipv4.tcp_wmem=4096 65536 16777216 > > BUT THIS ONLY SOLVES THE PROBLEM FOR WITHIN DOMU COMMUNCATION (i.e. > sender and receiver are the SAME virtual maschine, which should be a > theoretical problem in most cases). I tuned domU and dom0 this way > problem between dom0 and domU still exists. i also played with tcp > window sizes (-w option in iperf) from 64kb to 2MB but the performance > differences are negligable. >OK. Thanks. Did you try iperf in UDP mode between dom0 and domU? server: iperf -s -u -i 1 client: iperf -c server_ip -u -b 90M -t 20 to check for packet loss @ 90 Mbit/sec for 20 seconds. - Pasi> > Am Donnerstag, den 09.03.2006, 20:38 +0200 schrieb Pasi Kärkkäinen: > > On Thu, Mar 09, 2006 at 05:26:03PM +0100, Artur Schiefer wrote: > > > ok, solved the problem for within domu communcation, was an autotuning > > > effect. > > > > > > > Autotuning effect? Please be more specific because many of the xen users > > seem to have problems with network performance.. > > > > -- Pasi > > > > ^ > > . . > > Linux > > / - \ > > Choice.of.the > > .Next.Generation. > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Artur Schiefer wrote:> hello, > > somehow the linux network core tries to adjust itself to its > environment. as i recently learned theese parameters are optimized for > 100 mbit networking. from another poster i got the info that you have to > set the following parameters in your "/etc/sysctl.conf" to achieve gbit > speed: > # increase TCP maximum buffer size > net.core.rmem_max=16777216 > net.core.wmem_max=16777216 > > # increase Linux autotuning TCP buffer limits > # min, default, and maximum number of bytes to use > net.ipv4.tcp_rmem=4096 87380 16777216 > net.ipv4.tcp_wmem=4096 65536 16777216I would highly recommend also increasing the default size that the buffer is. The TCP buffer increase relies on learning from the behaviour of the traffic - but a large number of connections do not last very long. If you have a moderate sized or smallish file to send, you don''t catch up fast enough. Try: net.ipv4.tcp_rmem = 1398080 net.ipv4.tcp_wmem = 1398080 thanks, Nivedita> BUT THIS ONLY SOLVES THE PROBLEM FOR WITHIN DOMU COMMUNCATION (i.e. > sender and receiver are the SAME virtual maschine, which should be a > theoretical problem in most cases). I tuned domU and dom0 this way > problem between dom0 and domU still exists. i also played with tcp > window sizes (-w option in iperf) from 64kb to 2MB but the performance > differences are negligable. > > > Am Donnerstag, den 09.03.2006, 20:38 +0200 schrieb Pasi Kärkkäinen: > >>On Thu, Mar 09, 2006 at 05:26:03PM +0100, Artur Schiefer wrote: >> >>>ok, solved the problem for within domu communcation, was an autotuning >>>effect. >>> >> >>Autotuning effect? Please be more specific because many of the xen users >>seem to have problems with network performance.. >> >>-- Pasi >> >> ^ >> . . >> Linux >> / - \ >> Choice.of.the >> .Next.Generation. >> >>_______________________________________________ >>Xen-users mailing list >>Xen-users@lists.xensource.com >>http://lists.xensource.com/xen-users > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users