Xen devel - Jun 2010 - [PATCH] Add the USE_PLATFORM_TIMER flag to the FADT, to make w2k3 use the PMTIMER

because the default timesource (TSC) drifts under load.

The flag is only defined in ACPI 3.0, and we provide ACPI 2.0 tables,
but Windows seems happy enough to obey it anyway.

Signed-off-by: Tim Deegan <Tim.Deegan@citrix.com>




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Could you say more about this?  Which default timesource is
drifting?  Aren''t all HVM platform timers emulated on top
of Xen system time?  Does w2k3 use rdtsc directly... under
what circumstances?

Thanks,
Dan
> -----Original Message-----
> From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
> Sent: Wednesday, June 02, 2010 4:19 AM
> To: xen-devel@lists.xensource.com
> Subject: [Xen-devel] [PATCH] Add the USE_PLATFORM_TIMER flag to the
> FADT, to make w2k3 use the PMTIMER
> 
> because the default timesource (TSC) drifts under load.
> 
> The flag is only defined in ACPI 3.0, and we provide ACPI 2.0 tables,
> but Windows seems happy enough to obey it anyway.
> 
> Signed-off-by: Tim Deegan <Tim.Deegan@citrix.com>
> 
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, 

At 18:46 +0100 on 02 Jun (1275504378), Dan Magenheimer
wrote:
> Could you say more about this?  Which default timesource is
> drifting?  Aren''t all HVM platform timers emulated on top
> of Xen system time?  Does w2k3 use rdtsc directly... under
> what circumstances?
w2k3 will use RDTSC for its fine-grained time source (the one that''s
exposed as a performance counter) even if the ACPI PMTIMER is available,
unless it''s encouraged by this ACPI flag or the "/PMTIMER"
boot.ini
switch.  W2k3sp2 and later Windowses use the PMTIMER by default,
presumably because of the let''s-not-go-into-it-again difficulties with
using RDSTC as a time source.

http://blogs.technet.com/b/perfguru/archive/2008/02/18/explanation-for-the-usepmtimer-switch-in-the-boot-ini.aspx

Under stress testing, we found that this time source could drift (by
several percent) relative to real time, where the PMTIMER-based one did
not.

Yes, HVM timers are all based on the same source, except the RTC, which
is still implemented as a recurring timer event.  I suspect that the
drift is in the way Windows interprets the TSC values.  It could be, for
example, that it''s warping forward to avoid cross-CPU
time-going-backward issues.

Cheers,

Tim.
> > -----Original Message-----
> > From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
> > Sent: Wednesday, June 02, 2010 4:19 AM
> > To: xen-devel@lists.xensource.com
> > Subject: [Xen-devel] [PATCH] Add the USE_PLATFORM_TIMER flag to the
> > FADT, to make w2k3 use the PMTIMER
> > 
> > because the default timesource (TSC) drifts under load.
> > 
> > The flag is only defined in ACPI 3.0, and we provide ACPI 2.0 tables,
> > but Windows seems happy enough to obey it anyway.
> > 
> > Signed-off-by: Tim Deegan <Tim.Deegan@citrix.com>
> > 
> > 
-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> presumably because of the let''s-not-go-into-it-again difficulties
with
> using RDSTC as a time source.
I promise, I''ll try to be good :-)

I''m worried about this as a possible symptom of a bigger problem,
even if your patch provides an adequate workaround for one OS.
> w2k3 will use RDTSC for its fine-grained time source (the one
that''s
> exposed as a performance counter) even if the ACPI PMTIMER is
> available,
Wow, this is a surprise.  I thought Windows had eschewed all uses
of rdtsc.  I guess it is just too tempting ;-)
> Yes, HVM timers are all based on the same source, except the RTC, which
> is still implemented as a recurring timer event.  I suspect that the
> drift is in the way Windows interprets the TSC values.  It could be,
> for
> example, that it''s warping forward to avoid cross-CPU
> time-going-backward issues.
So we''re guessing that Windows is recognizing that raw TSC
is skewed and (like Xen and Solaris) providing some per-CPU
algorithm to correct for it.  Then, sadly, Xen changes
the vcpu-to-pcpu mapping and really confuses the correction
algorithm.

What tsc_mode is in use when this problem is encountered?
I''d think if tsc is emulated, there shouldn''t be a problem
(and since pmtimer is emulated anyway, shouldn''t be any slower).
> -----Original Message-----
> From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
> Sent: Thursday, June 03, 2010 2:58 AM
> To: Dan Magenheimer
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] [PATCH] Add the USE_PLATFORM_TIMER flag to the
> FADT, to make w2k3 use the PMTIMER
> 
> Hi,
> 
> At 18:46 +0100 on 02 Jun (1275504378), Dan Magenheimer wrote:
> > Could you say more about this?  Which default timesource is
> > drifting?  Aren''t all HVM platform timers emulated on top
> > of Xen system time?  Does w2k3 use rdtsc directly... under
> > what circumstances?
> 
> w2k3 will use RDTSC for its fine-grained time source (the one
that''s
> exposed as a performance counter) even if the ACPI PMTIMER is
> available,
> unless it''s encouraged by this ACPI flag or the
"/PMTIMER" boot.ini
> switch.  W2k3sp2 and later Windowses use the PMTIMER by default,
> presumably because of the let''s-not-go-into-it-again difficulties
with
> using RDSTC as a time source.
> 
> http://blogs.technet.com/b/perfguru/archive/2008/02/18/explanation-for-
> the-usepmtimer-switch-in-the-boot-ini.aspx
> 
> Under stress testing, we found that this time source could drift (by
> several percent) relative to real time, where the PMTIMER-based one did
> not.
> 
> Yes, HVM timers are all based on the same source, except the RTC, which
> is still implemented as a recurring timer event.  I suspect that the
> drift is in the way Windows interprets the TSC values.  It could be,
> for
> example, that it''s warping forward to avoid cross-CPU
> time-going-backward issues.
> 
> Cheers,
> 
> Tim.
> 
> > > -----Original Message-----
> > > From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
> > > Sent: Wednesday, June 02, 2010 4:19 AM
> > > To: xen-devel@lists.xensource.com
> > > Subject: [Xen-devel] [PATCH] Add the USE_PLATFORM_TIMER flag to
the
> > > FADT, to make w2k3 use the PMTIMER
> > >
> > > because the default timesource (TSC) drifts under load.
> > >
> > > The flag is only defined in ACPI 3.0, and we provide ACPI 2.0
> tables,
> > > but Windows seems happy enough to obey it anyway.
> > >
> > > Signed-off-by: Tim Deegan <Tim.Deegan@citrix.com>
> > >
> > >
> 
> --
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, XenServer Engineering
> Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 17:41 +0100 on 03 Jun (1275586899), Dan Magenheimer
wrote:
> So we''re guessing that Windows is recognizing that raw TSC
> is skewed and (like Xen and Solaris) providing some per-CPU
> algorithm to correct for it.  Then, sadly, Xen changes
> the vcpu-to-pcpu mapping and really confuses the correction
> algorithm.
Something like that, yes, though the TSC offset code is supposed to mask
that at least to the TSC-sync precision that Xen can manage.  Or
possibly just the jitter added by scheduling and descheduling adds up
over time, because the guest only ever speeds up the stragglers -- I''ve
seen Xen''s own system time do this on some older boxes, though not to
nearly the same extent.
> What tsc_mode is in use when this problem is encountered?
IIRC this was HVMPTM_no_delay_for_missed_ticks (on Xen 3.4 so no
emulated RDTSC).
> I''d think if tsc is emulated, there shouldn''t be a
problem
> (and since pmtimer is emulated anyway, shouldn''t be any slower).
That sounds plausible.

Cheers,

Tim.
> > -----Original Message-----
> > From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
> > Sent: Thursday, June 03, 2010 2:58 AM
> > To: Dan Magenheimer
> > Cc: xen-devel@lists.xensource.com
> > Subject: Re: [Xen-devel] [PATCH] Add the USE_PLATFORM_TIMER flag to
the
> > FADT, to make w2k3 use the PMTIMER
> > 
> > Hi,
> > 
> > At 18:46 +0100 on 02 Jun (1275504378), Dan Magenheimer wrote:
> > > Could you say more about this?  Which default timesource is
> > > drifting?  Aren''t all HVM platform timers emulated on
top
> > > of Xen system time?  Does w2k3 use rdtsc directly... under
> > > what circumstances?
> > 
> > w2k3 will use RDTSC for its fine-grained time source (the one
that''s
> > exposed as a performance counter) even if the ACPI PMTIMER is
> > available,
> > unless it''s encouraged by this ACPI flag or the
"/PMTIMER" boot.ini
> > switch.  W2k3sp2 and later Windowses use the PMTIMER by default,
> > presumably because of the let''s-not-go-into-it-again
difficulties with
> > using RDSTC as a time source.
> > 
> >
http://blogs.technet.com/b/perfguru/archive/2008/02/18/explanation-for-
> > the-usepmtimer-switch-in-the-boot-ini.aspx
> > 
> > Under stress testing, we found that this time source could drift (by
> > several percent) relative to real time, where the PMTIMER-based one
did
> > not.
> > 
> > Yes, HVM timers are all based on the same source, except the RTC,
which
> > is still implemented as a recurring timer event.  I suspect that the
> > drift is in the way Windows interprets the TSC values.  It could be,
> > for
> > example, that it''s warping forward to avoid cross-CPU
> > time-going-backward issues.
> > 
> > Cheers,
> > 
> > Tim.
> > 
> > > > -----Original Message-----
> > > > From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
> > > > Sent: Wednesday, June 02, 2010 4:19 AM
> > > > To: xen-devel@lists.xensource.com
> > > > Subject: [Xen-devel] [PATCH] Add the USE_PLATFORM_TIMER flag
to the
> > > > FADT, to make w2k3 use the PMTIMER
> > > >
> > > > because the default timesource (TSC) drifts under load.
> > > >
> > > > The flag is only defined in ACPI 3.0, and we provide ACPI
2.0
> > tables,
> > > > but Windows seems happy enough to obey it anyway.
> > > >
> > > > Signed-off-by: Tim Deegan <Tim.Deegan@citrix.com>
> > > >
> > > >
> > 
> > --
> > Tim Deegan <Tim.Deegan@citrix.com>
> > Principal Software Engineer, XenServer Engineering
> > Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)
-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel