-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I''m trying to diagnose a reasonably large performance drop between two ostensibly similar servers; one running CentOS 5.5 on bare metal, the other running Fedora 14 as a Xen DomU under CentOS 5.5 (Xen 3.1.2-194.26.1.el5) I have a pair of older Dell PowerEdge 1750 servers, each with 4GB ram and 2x 2.4GHz Xeon CPU''s with HyperThreading enabled. CPU''s do not support any VT. Disks are 3x U320 10k SCSI disks in software RAID-5. Both servers are connected to a Cisco 2950 100Mbit switch Server A is running kernel-2.6.18-194.26.1.el5 Server B is running kernel-xen-2.6.18-194.26.1.el5. Dom0 is pinned to one CPU, with 512MB ram. I have one DomU; - - B1 - Fedora 14 allocated the remaining 3 CPU''s and 3400MB ram. I have Apache httpd installed on Server A and B1. Both instances are running the same configuration based on mpm-worker and supporting keep-alives. I have a third box, also Dual-Xeon/HT from which I''m running the ab benchmark; ab -c40 -n100000 -k http://<ip>/index.html The index.html file simply contains the hostname of the server, so should be cached once read. iperf reports a healthy 95Mbit in each direction between the Client and DomU. Server A can support around 10,200 requests/second. The DomU only reaches a maximum of 3,800 requests/second. During the test, xentop reports no disk activity but CPU usage going up to 300%; consistent with 3 CPU''s. Can anyone shed any light on why I see such a drop in performance between these two servers? I could understand, to an extent, seeing the DomU getting 3/4 of the performance of the bare metal server given it has one CPU less, but this is less than 1/2 the performance. Regards, Mark. - -- Mark Watts, BSc RHCE http://www.linux-corner.info/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJM+UemAAoJEA67+nBFe32moeoH+QEijJ7oZwQ8LXxTu+8bLiPm 73tGFOfnvCsqnijWbH0Y8f2VUoId684r7IcXVyNYvbK4JaUvKkHrXdAWknyO8RD2 HFx3HjMC/J7RjzBEtporD/x/VfRMNO9Nsp7zycIs4rRhI+2lUuJha98V+ium9uYS RrifrSCnw87haFwipPtwsOsIOS/IkWW6vX9ZJFqvQTNhE2j1e56dj3CUv18oIwZK MzJa0/ucZyWi9kaIK3D70D72gGC1g7Oaeg3gB9NSfhVJTMX+ekNOGL+J4APluAs5 /R2lx94hfbcTI7m2FuerF1sRrBZ/7iMfb7HXpy25NiuowhFq6brBXzPgEteI4Zg=R49C -----END PGP SIGNATURE----- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jeff Sturm
2010-Dec-03 22:18 UTC
RE: [Xen-users] Linux DomU vs Bare Metal performance issues
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com [mailto:xen-users- > bounces@lists.xensource.com] On Behalf Of Mark Watts > Sent: Friday, December 03, 2010 2:40 PM > To: xen-users@lists.xensource.com > Subject: [Xen-users] Linux DomU vs Bare Metal performance issues > > Can anyone shed any light on why I see such a drop in performancebetween these> two servers? I could understand, to an extent, seeing the DomU getting3/4 of the> performance of the bare metal server given it has one CPU less, butthis is less than> 1/2 the performance.Understand what you''re really testing here is the performance of the virtual network driver, since Apache does almost no work to serve a simple static page. You''re exercising the TCP stack and network drivers more than anything else. Out of curiosity, can you easily run the same test against dom0? The dom0 runs a Xen kernel, on top of the hypervisor, but with a physical network interface (rather than virtual). One thing you can try is to pin the dom0 and/or domU to a physical core, using "xm vcpu-pin". That gave me better throughput on some of my tests. (I have doubts that hyperthreading scales as well as multicore processors, however I don''t have any hyperthreaded systems of my own to test.) Jeff _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mark Watts
2010-Dec-04 08:10 UTC
Re: [Xen-users] Linux DomU vs Bare Metal performance issues
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/12/2010 22:18, Jeff Sturm wrote:>> -----Original Message----- >> From: xen-users-bounces@lists.xensource.com [mailto:xen-users- >> bounces@lists.xensource.com] On Behalf Of Mark Watts >> Sent: Friday, December 03, 2010 2:40 PM >> To: xen-users@lists.xensource.com >> Subject: [Xen-users] Linux DomU vs Bare Metal performance issues >> >> Can anyone shed any light on why I see such a drop in performance > between these >> two servers? I could understand, to an extent, seeing the DomU getting > 3/4 of the >> performance of the bare metal server given it has one CPU less, but > this is less than >> 1/2 the performance. > > Understand what you''re really testing here is the performance of the > virtual network driver, since Apache does almost no work to serve a > simple static page. You''re exercising the TCP stack and network drivers > more than anything else.Sure, but iperf shows near-enough line speed in both directions. Shouldn''t this ''prove'' the virtual network driver is up to scratch for this system?> Out of curiosity, can you easily run the same test against dom0? The > dom0 runs a Xen kernel, on top of the hypervisor, but with a physical > network interface (rather than virtual).I can, certainly.> One thing you can try is to pin the dom0 and/or domU to a physical core, > using "xm vcpu-pin". That gave me better throughput on some of my > tests.Dom-0 is already pinned to one CPU with the Dom-U given the remaining three.> (I have doubts that hyperthreading scales as well as multicore > processors, however I don''t have any hyperthreaded systems of my own to > test.)I can certainly try this again with HT switched off on both boxes, although that''ll have to wait ''till monday. Mark. - -- Mark Watts, BSc RHCE http://www.linux-corner.info/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJM+fduAAoJEA67+nBFe32mpIYIALk6PrqhwqIQquq3gma6wGcl VCSHXYo/ZiZlAT/jTwYvY7qoR0CQ32VBipWoQnYoDRLKPMxhXS3Nt+AHeflW0ucG mx2lwFfqFf9aftDWMPqlHUOzZoMulDIloXObvzcfUVEHOzuyInlIo3WRnImUoC+U YAOes8ohiyDP823fRukXIH/JkCDNOrXquoB7tqtls4Q4KOOb5EwPnZ/BJDivIGpK IFNn3s3paN7F0quRYFFDM3j/dcZf8Jh8rJOY3XZ8Y06YCl1sAIsbOeTwFhfmglYn qspf9EYGqnUJ8w+GLlkWT7znmTH30Xr+87z/LE2/w/4sxN+SUti3D2reI/yniJ0=RtOZ -----END PGP SIGNATURE----- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mark Watts
2010-Dec-05 23:25 UTC
Re: [Xen-users] Linux DomU vs Bare Metal performance issues
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 04/12/2010 08:10, Mark Watts wrote:> On 03/12/2010 22:18, Jeff Sturm wrote: >>> -----Original Message----- >>> From: xen-users-bounces@lists.xensource.com [mailto:xen-users- >>> bounces@lists.xensource.com] On Behalf Of Mark Watts >>> Sent: Friday, December 03, 2010 2:40 PM >>> To: xen-users@lists.xensource.com >>> Subject: [Xen-users] Linux DomU vs Bare Metal performance issues >>> >>> Can anyone shed any light on why I see such a drop in performance >> between these >>> two servers? I could understand, to an extent, seeing the DomU getting >> 3/4 of the >>> performance of the bare metal server given it has one CPU less, but >> this is less than >>> 1/2 the performance. > >> Understand what you''re really testing here is the performance of the >> virtual network driver, since Apache does almost no work to serve a >> simple static page. You''re exercising the TCP stack and network drivers >> more than anything else. > > Sure, but iperf shows near-enough line speed in both directions. > Shouldn''t this ''prove'' the virtual network driver is up to scratch for > this system? > >> Out of curiosity, can you easily run the same test against dom0? The >> dom0 runs a Xen kernel, on top of the hypervisor, but with a physical >> network interface (rather than virtual). > > I can, certainly. > >> One thing you can try is to pin the dom0 and/or domU to a physical core, >> using "xm vcpu-pin". That gave me better throughput on some of my >> tests. > > Dom-0 is already pinned to one CPU with the Dom-U given the remaining three. > >> (I have doubts that hyperthreading scales as well as multicore >> processors, however I don''t have any hyperthreaded systems of my own to >> test.) > > I can certainly try this again with HT switched off on both boxes, > although that''ll have to wait ''till monday.Some updates: Running the same ab test against the Dom-0 gets me around 4,400 req/sec - - a modest increase over the Dom-U (~3,800). Rebooting my Dom-0 system into the standard CentOS 5.5 kernel sees the same 10,200 req/sec, so its clearly something related to this kernel which causes the slow down. Any ideas? Mark. - -- Mark Watts, BSc RHCE http://www.linux-corner.info/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJM/B9wAAoJEA67+nBFe32maJ4IAKMbxFXA2Lr/tVUW0LjPrJIC R8WPVMzKeajtrgixWtaLL/8ABinqfZGEu2/UE3FXSYKQF9frlXIaXuFsPafRztb2 W/Li3XPDWSTUeSaMfHo8SDhft9GMmNjXY2XYRd91hnM4RtKRCiM26/nXfbaqb8M6 2Hz4UEpRyIoPHHD/XgfKpjIXWgEzWEylVgMO1JuTpxj8MqWUuQBIm4ISlq16KcOd m14CBltZa0iup07UHRiFc9lm5zxVqwKN6KUMPXCjqgF8YyYh3/OFymkZ5JnxkUuC Yfft1Rhi6nUfS6L0Eb5GzRhIQzxoJ4ckOkPNECbXSPIRaI4rUabJwNG6PAQ8d1I=xhLd -----END PGP SIGNATURE----- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jeff Sturm
2010-Dec-07 04:57 UTC
RE: [Xen-users] Linux DomU vs Bare Metal performance issues
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com [mailto:xen-users- > bounces@lists.xensource.com] On Behalf Of Mark Watts > Sent: Sunday, December 05, 2010 6:26 PM > To: xen-users@lists.xensource.com > Subject: Re: [Xen-users] Linux DomU vs Bare Metal performance issues > > > I can certainly try this again with HT switched off on both boxes, > > although that'll have to wait 'till monday. > > Some updates: > > Running the same ab test against the Dom-0 gets me around 4,400 req/sec > - - a modest increase over the Dom-U (~3,800). > Rebooting my Dom-0 system into the standard CentOS 5.5 kernel sees the same > 10,200 req/sec, so its clearly something related to this kernel which causes the slow > down.Now, that's interesting-- my first guess would've been the net drivers, but since dom0 has a hardware NIC and exhibits similar performance, it sounds like we can rule that out. There are many differences in the CentOS standard kernel configs vs. the dom0 configs. For instance, the dom0 kernel undefines CONFIG_X86_HT and lowers CONFIG_HZ to 250. I don't know which of these settings, or the hypervisor for that matter, contributes the most to the performance drop you observe--I'd honestly be guessing. -Jeff _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2010-Dec-16 18:04 UTC
Re: [Xen-users] Linux DomU vs Bare Metal performance issues
On Fri, Dec 03, 2010 at 07:40:23PM +0000, Mark Watts wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > Hi, I''m trying to diagnose a reasonably large performance drop between > two ostensibly similar servers; one running CentOS 5.5 on bare metal, > the other running Fedora 14 as a Xen DomU under CentOS 5.5 (Xen > 3.1.2-194.26.1.el5) > > I have a pair of older Dell PowerEdge 1750 servers, each with 4GB ram > and 2x 2.4GHz Xeon CPU''s with HyperThreading enabled. CPU''s do not > support any VT. Disks are 3x U320 10k SCSI disks in software RAID-5. > Both servers are connected to a Cisco 2950 100Mbit switch > > Server A is running kernel-2.6.18-194.26.1.el5 > > Server B is running kernel-xen-2.6.18-194.26.1.el5. > Dom0 is pinned to one CPU, with 512MB ram. > I have one DomU; > - - B1 - Fedora 14 allocated the remaining 3 CPU''s and 3400MB ram. > > I have Apache httpd installed on Server A and B1. Both instances are > running the same configuration based on mpm-worker and supporting > keep-alives. >I can''t remember from top of my head.. does that create a lot of new processes, or does it re-use the same existing processes? -- Pasi> I have a third box, also Dual-Xeon/HT from which I''m running the ab > benchmark; ab -c40 -n100000 -k http://<ip>/index.html > The index.html file simply contains the hostname of the server, so > should be cached once read. > > iperf reports a healthy 95Mbit in each direction between the Client and > DomU. > > Server A can support around 10,200 requests/second. > The DomU only reaches a maximum of 3,800 requests/second. > > During the test, xentop reports no disk activity but CPU usage going up > to 300%; consistent with 3 CPU''s. > > > Can anyone shed any light on why I see such a drop in performance > between these two servers? I could understand, to an extent, seeing the > DomU getting 3/4 of the performance of the bare metal server given it > has one CPU less, but this is less than 1/2 the performance. > > Regards, > > Mark. > > - -- > Mark Watts, BSc RHCE > http://www.linux-corner.info/ > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (MingW32) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iQEcBAEBAgAGBQJM+UemAAoJEA67+nBFe32moeoH+QEijJ7oZwQ8LXxTu+8bLiPm > 73tGFOfnvCsqnijWbH0Y8f2VUoId684r7IcXVyNYvbK4JaUvKkHrXdAWknyO8RD2 > HFx3HjMC/J7RjzBEtporD/x/VfRMNO9Nsp7zycIs4rRhI+2lUuJha98V+ium9uYS > RrifrSCnw87haFwipPtwsOsIOS/IkWW6vX9ZJFqvQTNhE2j1e56dj3CUv18oIwZK > MzJa0/ucZyWi9kaIK3D70D72gGC1g7Oaeg3gB9NSfhVJTMX+ekNOGL+J4APluAs5 > /R2lx94hfbcTI7m2FuerF1sRrBZ/7iMfb7HXpy25NiuowhFq6brBXzPgEteI4Zg> =R49C > -----END PGP SIGNATURE----- > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users