Hi all, I''m trying to troubleshoot some performance issues I''m getting with HVM under Xen 4.1.2 (FC16 packages) - I''m seeing these issues in Windows and Ubuntu. Disk IO is slow (15 MB/sec) when using the "normal way" of disk = [''phy:whatever'']. I understand by googling that the conventional way to resolve this is to export a pvscsi device to the domU, though I''m guessing I''d need PV drivers in the HVM domU first? Further, on Windows I''m thinking that might create a problem if I up and "move" the system disk from the conventional qemu IDE controller to SCSI. If anyone has any comments or experience with this, I''d love to hear them. Also with regard to network performance, that too is very slow using the traditional vif = [ ''type=iommu''] method. I''ve read that PV drivers are the best way to get "normal" network throughput comparable to what you''d see on the dom0. Is that correct, or is there some other driver that''s better than qemu/iommu? Thanks in advance. -Ray
On Thu, May 3, 2012 at 3:07 PM, Ray Barnes <tical.net@gmail.com> wrote:> Hi all, > > I''m trying to troubleshoot some performance issues I''m getting with > HVM under Xen 4.1.2 (FC16 packages) - I''m seeing these issues in > Windows and Ubuntu. Disk IO is slow (15 MB/sec) when using the > "normal way" of disk = [''phy:whatever'']. I understand by googling > that the conventional way to resolve this is to export a pvscsi device > to the domU,err... no. Why would you think that? IIRC pvscsi is different from normal pv block device, suitable for some things (e.g. tape backup, DVD writer). To get better performance you only need pv block device.> though I''m guessing I''d need PV drivers in the HVM domU > first?exatcly.> Further, on Windows I''m thinking that might create a problem > if I up and "move" the system disk from the conventional qemu IDE > controller to SCSI.Nope. in the case of pv block device, pv driver installer will handle it automatically, so it''s mostly click-next-reboot. -- Fajar
On 5/3/2012 6:26 PM, Ray Barnes wrote:> Thanks for the replies. I found and installed James'' PV drivers, > version 0.11.0.357. It did wonders for my disk throughput. But it > also created a new problem with transmission throughput on the NIC. > Prior to the PV installation, I had a minimum of 50 megabits > bi-directional throughput, with the dom0 uplinked to the network via > FastE. Now, post-installation, although my download throughput is > much faster and more consistent, my upload throughput out to the > Internet is extremely slow, on the order of 5 megabits/sec for a > single thread. I can not duplicate this locally to my LAN - I''m able > to push 80 to 90 mbits/sec to the local LAN. But every single test > I''m running to speedtest.net results in uploads between 1-5 mbits/sec. > For reference, I''m able to do 50+ mbits/sec upload to the same > targets via a nonvirtualized Windows box on my network. The problem > is definitely confined to the Windows domU (version 2003 R2). > > I''m invoking the vif like this: > > vif = [ ''type=iommu, mac=00:16:3e:00:00:09, bridge=something, > script=vif-something''] > > I''ve also tried it with ''model=intel'' appended but with no change. > Also, against my better judgment, I implemented the registry changes > mentioned at http://msdn.microsoft.com/en-us/library/ee377084%28v=bts.10%29.aspx > for better network performance (via the supplied PowerShell script at > the bottom), no change also. > > Any idea what''s happening?I''ve seen problems with GSO and TSO causing corrupted checksums for packets with certain properties in the past, which might be affecting you here (this issue, if seen, would cause random dropped packets, killing transfer speeds). Try turning off TSO/GSO/GRO/LRO both in dom0 (using ethool -K) and on the virtual network adapter in the domU. Since you have the GPLPV drivers installed, you can also remove "type=iommu, " from the vif line. I don''t know if it hurts to have it there, but it shouldn''t hurt to remove it. -John
Thanks for the reply. I did find large sending offloads configured in the domU (under Device Manager). Based on something else I googled, I turned them off and was able to resolve this fully. My theory is that those offloads were latent, which would explain why something with microseconds of latency on my LAN would transfer quickly, but a couple of msec further over the wire would not. Thanks again. -Ray On Fri, May 4, 2012 at 4:12 AM, John Weekes <lists.xen@nuclearfallout.net> wrote:> On 5/3/2012 6:26 PM, Ray Barnes wrote: >> >> Thanks for the replies. I found and installed James'' PV drivers, >> version 0.11.0.357. It did wonders for my disk throughput. But it >> also created a new problem with transmission throughput on the NIC. >> Prior to the PV installation, I had a minimum of 50 megabits >> bi-directional throughput, with the dom0 uplinked to the network via >> FastE. Now, post-installation, although my download throughput is >> much faster and more consistent, my upload throughput out to the >> Internet is extremely slow, on the order of 5 megabits/sec for a >> single thread. I can not duplicate this locally to my LAN - I''m able >> to push 80 to 90 mbits/sec to the local LAN. But every single test >> I''m running to speedtest.net results in uploads between 1-5 mbits/sec. >> For reference, I''m able to do 50+ mbits/sec upload to the same >> targets via a nonvirtualized Windows box on my network. The problem >> is definitely confined to the Windows domU (version 2003 R2). >> >> I''m invoking the vif like this: >> >> vif = [ ''type=iommu, mac=00:16:3e:00:00:09, bridge=something, >> script=vif-something''] >> >> I''ve also tried it with ''model=intel'' appended but with no change. >> Also, against my better judgment, I implemented the registry changes >> mentioned at >> http://msdn.microsoft.com/en-us/library/ee377084%28v=bts.10%29.aspx >> for better network performance (via the supplied PowerShell script at >> the bottom), no change also. >> >> Any idea what''s happening? > > > I''ve seen problems with GSO and TSO causing corrupted checksums for packets > with certain properties in the past, which might be affecting you here (this > issue, if seen, would cause random dropped packets, killing transfer > speeds). Try turning off TSO/GSO/GRO/LRO both in dom0 (using ethool -K) and > on the virtual network adapter in the domU. > > Since you have the GPLPV drivers installed, you can also remove "type=iommu, > " from the vif line. I don''t know if it hurts to have it there, but it > shouldn''t hurt to remove it. > > -John