Hi, I''ve been investigating networking performance for Windows 2003 guests on a Solaris dom0 and I''ve uncovered a couple of things which I don''t really understand and I''m looking for some explanations. This first question is really directed at virtinst developers... Windows guests have different ACPI and APIC settings depending on which variant of Windows is to be installed: variant: unspecified -> ACPI: true, APIC: true variant: winxp -> ACPI: false, APIC: false variant: win2k -> ACPI: false, APIC: false variant: win2k3 -> ACPI: true, APIC: true variant: vista -> ACPI: true, APIC: true The question is: why don''t all Windows variants have the same ACPI and APIC settings? The reason I''m curious, is that it seems like different combinations of APIC and ACPI produce different functionality in the guest and different levels of performance. I did some tests with win2k3 guests and measured networking performance using iperf. I used the same dom0 system (snv_103) and created three W2K3 guests, each with 1 CPU but different APIC and ACPI settings: apic/acpi setting results (Mb/s throughput) apic: 0 acpi: 0 834, 833 apic: 1 acpi: 1 718, 704 apic: 0 acpi: 1 872, 876 (higher numbers are better) I only had time to perform two runs (the test is the iperf-2.0.2 1MB msg test that I ran to microbenchmark TCP performance) for each guest, but you can see that there is a fairly large and consistent (across two runs for each guest) performance difference. It''s about 20% faster when apic is not specified for the guest. My next question is: What is really happening when APIC is specified for a windows guest and why does performance vary so much according to whether it''s specified or not? Thanks for any help or insight anyone can offer. Gary -- Gary Pennington Solaris Core OS Sun Microsystems Gary.Pennington@sun.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 05/12/2008 14:03, "Gary Pennington" <Gary.Pennington@sun.com> wrote:> My next question is: What is really happening when APIC is specified for > a windows guest and why does performance vary so much according to whether > it''s specified or not?Older Windows kernels update the APIC TPR a lot, and unless you have a very modern Intel processor every one of those TPR updates causes a vmexit. Modern Windows (including possibly latest w2k3 service pack, but I''m not totally certain) includes lazy TPR, which gets rid of the vast majority of TPR updates, and hence will go much faster. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
IIRC, only w2k3sp2+, vista, and w2k8 have lazy TPR. Not positive about w2k3sp1, but I think it doesn''t. WinXP does not have lazy TPR in any service pack, AFAIK. What service pack of w2k3 did you do those tests with? Hopefully, sometime in the next few weeks, I''ll be able to release my ''xenalyze'' tool, which will help a lot with analyzing what''s really going on with these kinds of workloads. -George On Fri, Dec 5, 2008 at 2:45 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:> On 05/12/2008 14:03, "Gary Pennington" <Gary.Pennington@sun.com> wrote: > >> My next question is: What is really happening when APIC is specified for >> a windows guest and why does performance vary so much according to whether >> it''s specified or not? > > Older Windows kernels update the APIC TPR a lot, and unless you have a very > modern Intel processor every one of those TPR updates causes a vmexit. > > Modern Windows (including possibly latest w2k3 service pack, but I''m not > totally certain) includes lazy TPR, which gets rid of the vast majority of > TPR updates, and hence will go much faster. > > -- Keir > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Dec 05, 2008 at 04:42:57PM +0000, George Dunlap wrote:> IIRC, only w2k3sp2+, vista, and w2k8 have lazy TPR. Not positive > about w2k3sp1, but I think it doesn''t. WinXP does not have lazy TPR > in any service pack, AFAIK. > > What service pack of w2k3 did you do those tests with? >I''m not sure, because the ISO was provided by a colleague. I think it''s the original release of w2k3, with no service packs. However, what I was mostly curious about was why specifying apic when the guest was created made so much different to performance. Gary> Hopefully, sometime in the next few weeks, I''ll be able to release my > ''xenalyze'' tool, which will help a lot with analyzing what''s really > going on with these kinds of workloads. > > -George > > On Fri, Dec 5, 2008 at 2:45 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote: > > On 05/12/2008 14:03, "Gary Pennington" <Gary.Pennington@sun.com> wrote: > > > >> My next question is: What is really happening when APIC is specified for > >> a windows guest and why does performance vary so much according to whether > >> it''s specified or not? > > > > Older Windows kernels update the APIC TPR a lot, and unless you have a very > > modern Intel processor every one of those TPR updates causes a vmexit. > > > > Modern Windows (including possibly latest w2k3 service pack, but I''m not > > totally certain) includes lazy TPR, which gets rid of the vast majority of > > TPR updates, and hence will go much faster. > > > > -- Keir > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel-- Gary Pennington Solaris Core OS Sun Microsystems Gary.Pennington@sun.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Dec 5, 2008 at 4:57 PM, Gary Pennington <Gary.Pennington@sun.com> wrote:> However, what I was mostly curious about was why specifying apic when the > guest was created made so much different to performance.Because of how the interrupt blocking works with APIC vs non-apic. For APIC (on Intel boxes), you set the level of interrupts that you want to receive (called the TPR, or Task Priority Register) by writing to an MMIO register. Under Xen, this means a vmexit every time the TPR is written. w2k3sp0 writes to the TPR every time the priority changes, which happens a lot. Each write causes a vmexit/update/vmenter cycle, which is pretty expensive. w2k3sp2 has a technique called "lazy TPR", where it manages to avoid the vast majority of TPR writes. This results in a much lower overhead, probably on bare hardware as well, but especially in HVM-virtualized environments. If you take a xen trace of w2k3sp0, you will see a ton of MMIO writes to ffe00080; that''s the TPR register (if I recall correctly from the top of my head). xenalyze (Coming Soon[tm]) would look at the trace and tell you that you were spending a non-negligible percentage of your time handling MMIO writes to the TPR register. If you disable the APIC, the processor uses other methods of setting interrupt levels (don''t know these off the top of my head) which are virtualized as a part of the HVM spec, and are therefore don''t involve the hyperivsor at all. As Keir said, newer Intel hardware has TPR virtualization, so even with the APIC accessing the TPR doesn''t involve the hypervisor. -George> > Gary > >> Hopefully, sometime in the next few weeks, I''ll be able to release my >> ''xenalyze'' tool, which will help a lot with analyzing what''s really >> going on with these kinds of workloads. >> >> -George >> >> On Fri, Dec 5, 2008 at 2:45 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote: >> > On 05/12/2008 14:03, "Gary Pennington" <Gary.Pennington@sun.com> wrote: >> > >> >> My next question is: What is really happening when APIC is specified for >> >> a windows guest and why does performance vary so much according to whether >> >> it''s specified or not? >> > >> > Older Windows kernels update the APIC TPR a lot, and unless you have a very >> > modern Intel processor every one of those TPR updates causes a vmexit. >> > >> > Modern Windows (including possibly latest w2k3 service pack, but I''m not >> > totally certain) includes lazy TPR, which gets rid of the vast majority of >> > TPR updates, and hence will go much faster. >> > >> > -- Keir >> > >> > >> > >> > _______________________________________________ >> > Xen-devel mailing list >> > Xen-devel@lists.xensource.com >> > http://lists.xensource.com/xen-devel >> > >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel > > -- > Gary Pennington > Solaris Core OS > Sun Microsystems > Gary.Pennington@sun.com > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Dec 05, 2008 at 05:06:28PM +0000, George Dunlap wrote:> On Fri, Dec 5, 2008 at 4:57 PM, Gary Pennington <Gary.Pennington@sun.com> wrote: > > However, what I was mostly curious about was why specifying apic when the > > guest was created made so much different to performance. > > Because of how the interrupt blocking works with APIC vs non-apic. > For APIC (on Intel boxes), you set the level of interrupts that you > want to receive (called the TPR, or Task Priority Register) by writing > to an MMIO register. Under Xen, this means a vmexit every time the > TPR is written. w2k3sp0 writes to the TPR every time the priority > changes, which happens a lot. Each write causes a > vmexit/update/vmenter cycle, which is pretty expensive. > > w2k3sp2 has a technique called "lazy TPR", where it manages to avoid > the vast majority of TPR writes. This results in a much lower > overhead, probably on bare hardware as well, but especially in > HVM-virtualized environments. > > If you take a xen trace of w2k3sp0, you will see a ton of MMIO writes > to ffe00080; that''s the TPR register (if I recall correctly from the > top of my head). xenalyze (Coming Soon[tm]) would look at the trace > and tell you that you were spending a non-negligible percentage of > your time handling MMIO writes to the TPR register. > > If you disable the APIC, the processor uses other methods of setting > interrupt levels (don''t know these off the top of my head) which are > virtualized as a part of the HVM spec, and are therefore don''t involve > the hyperivsor at all. As Keir said, newer Intel hardware has TPR > virtualization, so even with the APIC accessing the TPR doesn''t > involve the hypervisor. >Thanks very much for the explanation. That makes it clear(ish) why specifying APIC alters performance so much. Now what I need to understand is why certain Windows variants default to specify APIC and others don''t... It seems like this area is a bit of a minefield and it would be good to get authoritative data on how to specify apic/acpi for windows guests, depending on hardware available and flavour of windows... Gary> -George > > > > > Gary > > > >> Hopefully, sometime in the next few weeks, I''ll be able to release my > >> ''xenalyze'' tool, which will help a lot with analyzing what''s really > >> going on with these kinds of workloads. > >> > >> -George > >> > >> On Fri, Dec 5, 2008 at 2:45 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote: > >> > On 05/12/2008 14:03, "Gary Pennington" <Gary.Pennington@sun.com> wrote: > >> > > >> >> My next question is: What is really happening when APIC is specified for > >> >> a windows guest and why does performance vary so much according to whether > >> >> it''s specified or not? > >> > > >> > Older Windows kernels update the APIC TPR a lot, and unless you have a very > >> > modern Intel processor every one of those TPR updates causes a vmexit. > >> > > >> > Modern Windows (including possibly latest w2k3 service pack, but I''m not > >> > totally certain) includes lazy TPR, which gets rid of the vast majority of > >> > TPR updates, and hence will go much faster. > >> > > >> > -- Keir > >> > > >> > > >> > > >> > _______________________________________________ > >> > Xen-devel mailing list > >> > Xen-devel@lists.xensource.com > >> > http://lists.xensource.com/xen-devel > >> > > >> > >> _______________________________________________ > >> Xen-devel mailing list > >> Xen-devel@lists.xensource.com > >> http://lists.xensource.com/xen-devel > > > > -- > > Gary Pennington > > Solaris Core OS > > Sun Microsystems > > Gary.Pennington@sun.com > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > >-- Gary Pennington Solaris Core OS Sun Microsystems Gary.Pennington@sun.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel