G. Michael Carter
2008-Jun-25 20:41 UTC
[Xen-users] Network nightmare. Is NAT better than bridge for DomU to DomU?
I''m having lots of fun with my xen machine... I''ve done all the tips I can find with sysctl and txqueuelen. External machine to Xen Dom0 is about 40-50MB/s and to DomU''s anywhere between 10-40MB/s But DomU to DomU is being a real pain. Most times I only get 6MB/s. One thing I''ve noticed (I think) is when my webserver domU talks to my db domU speeds start fluctuating really bad and the network traffic is rather small. virt-top will report 60MB/s transmition on the vir interfaces even though the file is only transferring at 2MB/s I was wondering if I set up a local virtual network would it be faster than the bridge from DomU <-> DomU network? Any one have any tips other than what I''ve found about sysctl and txqueuelen? Michael _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
G. Michael Carter
2008-Jun-25 21:03 UTC
Re: [Xen-users] Network nightmare. Is NAT better than bridge for DomU to DomU?
Something to add... I just noticed when the traffic is going from DomU to Dom0/U the ST percentage on top goes to 50%. Only happens for DomU to Dom0/U traffic. What''s ST? Is it interrupt related? Michael G. Michael Carter wrote:> I''m having lots of fun with my xen machine... > > I''ve done all the tips I can find with sysctl and txqueuelen. > External machine to Xen Dom0 is about 40-50MB/s and to DomU''s anywhere > between 10-40MB/s > But DomU to DomU is being a real pain. Most times I only get 6MB/s. > One thing I''ve noticed (I think) is when my webserver domU talks to my > db domU speeds start fluctuating really bad and the network traffic is > rather small. virt-top will report 60MB/s transmition on the vir > interfaces even though the file is only transferring at 2MB/s > > I was wondering if I set up a local virtual network would it be faster > than the bridge from DomU <-> DomU network? Any one have any tips > other than what I''ve found about sysctl and txqueuelen? > > Michael > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users-- *G. Michael Carter* *Projects* .. GarageSales http://www.garagesaleorganizer.ca Blog/Diaries http://theshadow.carterfamily.ca Family Pictures http://photoalbum.carterfamily.ca GeoCaching http://www.geocache.carterfamily.ca *Contact Info* Office 1-905-890-8100 x 28494 Work Site http://www.oracle.com 1-905-267-8494 Personal http://www.carterfamily.ca Cell 519-215-1869 AIM MikeyCarter Fax 1-519-941-0009 OIM Michael.Carter _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2008-Jun-26 04:07 UTC
Re: [Xen-users] Network nightmare. Is NAT better than bridge for DomU to DomU?
G. Michael Carter wrote:> I''m having lots of fun with my xen machine... > > I''ve done all the tips I can find with sysctl and txqueuelen. > External machine to Xen Dom0 is about 40-50MB/s and to DomU''s anywhere > between 10-40MB/s > But DomU to DomU is being a real pain. Most times I only get 6MB/s.What OS/Xen versions? What type of domU (PV Linux, HVM)? How do you test network performance? On my system (RHEL5.2/x86_64/xen-3.2.1/Xeon 5130), using iperf, i get 400 - 900 Mbps. Performance is cpu-bound (since they''re all software NICs connected to the same bridge). Here''s the relevant content from my sysctl.conf ### network read/write buffer ### # increase TCP max buffer size net.core.rmem_max = 262144 net.core.wmem_max = 262144 net.core.rmem_default= 262144 net.core.wmem_default= 262144 # increase Linux autotuning TCP buffer limits # min, default, and max number of bytes to use net.ipv4.tcp_wmem=4096 87380 262144 net.ipv4.tcp_rmem=4096 87380 262144 Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Markus Hochholdinger
2008-Jun-26 12:13 UTC
Re: [Xen-users] Network nightmare. Is NAT better than bridge for DomU to DomU?
Hi, Am Mittwoch, 25. Juni 2008 23:03 schrieb G. Michael Carter:> Something to add... I just noticed when the traffic is going from DomU > to Dom0/U the ST percentage on top goes to 50%. Only happens for DomU > to Dom0/U traffic. What''s ST? Is it interrupt related?perhaps you only have one cpu? man vmstat [..] st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown. [..] So your cpu cycles from your domU are stolen from dom0 to mange the virtual network!?> G. Michael Carter wrote: > > I''m having lots of fun with my xen machine... > > > > I''ve done all the tips I can find with sysctl and txqueuelen. > > External machine to Xen Dom0 is about 40-50MB/s and to DomU''s anywhere > > between 10-40MB/s > > But DomU to DomU is being a real pain. Most times I only get 6MB/s. > > One thing I''ve noticed (I think) is when my webserver domU talks to my > > db domU speeds start fluctuating really bad and the network traffic is > > rather small. virt-top will report 60MB/s transmition on the vir > > interfaces even though the file is only transferring at 2MB/s > > > > I was wondering if I set up a local virtual network would it be faster > > than the bridge from DomU <-> DomU network? Any one have any tips > > other than what I''ve found about sysctl and txqueuelen? > > > > Michael > > > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users-- greetings eMHa _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2008-Jun-27 02:38 UTC
Re: [Xen-users] Network nightmare. Is NAT better than bridge for DomU to DomU?
G. Michael Carter wrote:> OS: > *Dom0* - Fedora 8 (x86_64, xen:3.1.2-2.fc8) > > *DomU*: > Fedora 9 (x2 - i686, xen:3.2.0-12.fc9) > Oracle Enterprise Linux 5.1 (x86_64, xen:3.0.3-64.el5_2.1), > CentOS 5 (i686, xen: 3.0.3-64.el5_2.1), > Fedora 7 (x86_64, xen: 3.1.2-2.fc7) > > I know xen on the Dom0 is the only important one. All PV running on > a Intel Dual Core Pentium D 2.8Ghz (only one without HVM grrr) > > To test I''ve tried iperf, scp files, and DB link from Oracle -> Oracle. > > iperf I get 80-320 Mbps. >At first glance I assume it''s cpu problem. Try this : - disable hyperthread. That should make xen reports only two cpus (which is the number of cores you have) - assign dom0 one cpu only, on a dedicated core with xm vcpu-set and xm-vcpu-pin (e.g. cpu 0) - assign both test domU to other cpu (in your case this would be cpu 1) with xm vcpu-pin or modifying domU config file Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users