Martin Goldstone
2007-May-23 15:05 UTC
[Xen-users] Slow network performance between HVM guests
Hi all, We''re running Xen 3.1 on Centos 5 here (x86_64, kernel 2.6.18), and we''re seeing some odd networking issues with our Windows HVM guests. Perhaps someone here has some ideas they can offer. Briefly, the specs of the host are: Dual Xeon 3 GHz (Dual Core + HT), 4GB RAM, 73 GB HDD. Network is e1000. The Windows HVM guests are Windows Server 2003 R2 Standard x86-64, with 2 VCPUs, 1GB RAM, 16GB HDD (in a file), rtl8139 network. Basically, file transfer speeds between guests (both on the same bridge) on the same host are approximately 10% (if not less) of the speed of file transfers from a HVM guest to another system (virtualised or not) away from that host. Any ideas, or is this normal behaviour? This is affecting us with Xen 3.03 and 3.0.4.1 as well, so it''s not just a 3.1 thing. I''ve disabled iptables on dom0 to see if that makes a difference (it doesn''t). We''ve tried the 32 bit version of Windows, and we''ve reduced the number of VCPUs to 1, and increased to 4, all without success. We originally thought it might have something to do with another issue we were experiencing (ping times reported in Windows were very strange, show latency of several thousand ms, and showing negative latency times as well) but that issue disappeared after setting the number of VCPUs to 1 (apparently there is a bug in the Windows Multiprocessor ACPI HAL (incidentally, does anyone know if this bug affects anything other than the displayed latency times?)). We haven''t had any success tracking this down so far. Any ideas? Thanks in advance for any help, Martin _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Petersson, Mats
2007-May-23 15:17 UTC
RE: [Xen-users] Slow network performance between HVM guests
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > Martin Goldstone > Sent: 23 May 2007 16:05 > To: xen-users@lists.xensource.com > Subject: [Xen-users] Slow network performance between HVM guests > > Hi all, > > We''re running Xen 3.1 on Centos 5 here (x86_64, kernel 2.6.18), and > we''re seeing some odd networking issues with our Windows HVM guests. > Perhaps someone here has some ideas they can offer. > > Briefly, the specs of the host are: Dual Xeon 3 GHz (Dual Core + HT), > 4GB RAM, 73 GB HDD. Network is e1000. > > The Windows HVM guests are Windows Server 2003 R2 Standard > x86-64, with > 2 VCPUs, 1GB RAM, 16GB HDD (in a file), rtl8139 network. > > Basically, file transfer speeds between guests (both on the > same bridge) > on the same host are approximately 10% (if not less) of the speed of > file transfers from a HVM guest to another system > (virtualised or not) > away from that host. Any ideas, or is this normal behaviour?Most likely it''s because the network access is going through QEMU-DM, which means that Dom0 has to "emulate" the network device. With BOTH devices being emulated on the same Dom0, you get latency added in both ends, and there''s less likely to be any "overlap" between them. Do you by any chance also restrict the Dom0 to a single core? Also, are the two guest domains running on the same or different cores? If both domains use the same core, they would obviously "stop each other" from running.> This is > affecting us with Xen 3.03 and 3.0.4.1 as well, so it''s not > just a 3.1 > thing. I''ve disabled iptables on dom0 to see if that makes a > difference > (it doesn''t). We''ve tried the 32 bit version of Windows, and we''ve > reduced the number of VCPUs to 1, and increased to 4, all without > success. We originally thought it might have something to do with > another issue we were experiencing (ping times reported in > Windows were > very strange, show latency of several thousand ms, and > showing negative > latency times as well) but that issue disappeared after setting the > number of VCPUs to 1 (apparently there is a bug in the Windows > Multiprocessor ACPI HAL (incidentally, does anyone know if this bug > affects anything other than the displayed latency times?)).The negative latency is probably the one reported in the internal Intel bug-tracker here: http://losvmm-bridge.sh.intel.com/bugzilla/show_bug.cgi?id=991 But it''s not easy to know, since it''s not accessible from outside Intel (or at least not from the AMD network, but I doubt that it''s really a "hide this from AMD" attempt, but rather that the link is an internal Intel site). My guess would be that the time is measured using timestamp counting, and it fails because it''s taking the TSC from two different processor cores at different times, which leads to varying results. But that''s speculation, and not based on any real understanding of why this is. -- Mats> > We haven''t had any success tracking this down so far. Any ideas? > > Thanks in advance for any help, > > Martin > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Martin Goldstone
2007-May-23 16:17 UTC
Re: [Xen-users] Slow network performance between HVM guests
Petersson, Mats wrote:> > > >> -----Original Message----- >> From: xen-users-bounces@lists.xensource.com >> [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of >> Martin Goldstone >> Sent: 23 May 2007 16:05 >> To: xen-users@lists.xensource.com >> Subject: [Xen-users] Slow network performance between HVM guests >> >> Hi all, >> >> We''re running Xen 3.1 on Centos 5 here (x86_64, kernel 2.6.18), and >> we''re seeing some odd networking issues with our Windows HVM guests. >> Perhaps someone here has some ideas they can offer. >> >> Briefly, the specs of the host are: Dual Xeon 3 GHz (Dual Core + HT), >> 4GB RAM, 73 GB HDD. Network is e1000. >> >> The Windows HVM guests are Windows Server 2003 R2 Standard >> x86-64, with >> 2 VCPUs, 1GB RAM, 16GB HDD (in a file), rtl8139 network. >> >> Basically, file transfer speeds between guests (both on the >> same bridge) >> on the same host are approximately 10% (if not less) of the speed of >> file transfers from a HVM guest to another system >> (virtualised or not) >> away from that host. Any ideas, or is this normal behaviour? >> > > Most likely it''s because the network access is going through QEMU-DM, > which means that Dom0 has to "emulate" the network device. With BOTH > devices being emulated on the same Dom0, you get latency added in both > ends, and there''s less likely to be any "overlap" between them. > > Do you by any chance also restrict the Dom0 to a single core? > >Although we had Dom0 set to take all the cpus (basically, the default), I''ve just checked and it was only using 2 vcpus (ie 1 core) (I guess it got knocked down to that by me starting up other guests and didn''t take the CPUs back when they were shut down). I''ve now set it to have 4 VCPUs (ie a whole cpu) and pinned it to the 1st CPU (0-3).> Also, are the two guest domains running on the same or different cores? > If both domains use the same core, they would obviously "stop each > other" from running. > > >I had forgotten to pin the vcpus (2 in each guest), so its possible they started blocking each other. I''ve pinned them now to separate cores, and I am seeing better performance (about double the speed that I was getting before, perhaps even faster than that).>> This is >> affecting us with Xen 3.03 and 3.0.4.1 as well, so it''s not >> just a 3.1 >> thing. I''ve disabled iptables on dom0 to see if that makes a >> difference >> (it doesn''t). We''ve tried the 32 bit version of Windows, and we''ve >> reduced the number of VCPUs to 1, and increased to 4, all without >> success. We originally thought it might have something to do with >> another issue we were experiencing (ping times reported in >> Windows were >> very strange, show latency of several thousand ms, and >> showing negative >> latency times as well) but that issue disappeared after setting the >> number of VCPUs to 1 (apparently there is a bug in the Windows >> Multiprocessor ACPI HAL (incidentally, does anyone know if this bug >> affects anything other than the displayed latency times?)). >> > > The negative latency is probably the one reported in the internal Intel > bug-tracker here: > http://losvmm-bridge.sh.intel.com/bugzilla/show_bug.cgi?id=991 > But it''s not easy to know, since it''s not accessible from outside Intel > (or at least not from the AMD network, but I doubt that it''s really a > "hide this from AMD" attempt, but rather that the link is an internal > Intel site). > > My guess would be that the time is measured using timestamp counting, > and it fails because it''s taking the TSC from two different processor > cores at different times, which leads to varying results. But that''s > speculation, and not based on any real understanding of why this is. > >Hmm, that Intel site does appear to be internal only, oh well. I guess I''ll just have to keep my eye on it to see whether its affecting anything else (apart from SiSoftware Sandra, which is registering about 20% packet loss in its performance index tests). Thanks very much for your help Mart> -- > Mats > >> We haven''t had any success tracking this down so far. Any ideas? >> >> Thanks in advance for any help, >> >> Martin >> >> >> >> > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users