Cui, Dexuan
2007-Jan-10 11:29 UTC
[Xen-devel] [Patch] Fix the slow wall clock time issue in x64 SMP Vista
In x64 SMP Vista HVM guest (vcpus=2 in the configuration file), the wall clock time is 50% slower than that in the real world. The attached patch fixes the issue. -- Dexuan Signed-off-by: Dexuan Cui <dexuan.cui@intel.com> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2007-Jan-10 11:36 UTC
Re: [Xen-devel] [Patch] Fix the slow wall clock time issue in x64 SMP Vista
On 10/1/07 11:29, "Cui, Dexuan" <dexuan.cui@intel.com> wrote:> In x64 SMP Vista HVM guest (vcpus=2 in the configuration file), the wall > clock time is 50% slower than that in the real world. The attached patch > fixes the issue.Looks like this code will inject fewer HPET interrupts. How does that make time go *faster*? Also, looking at hpet_timer_fn() is vcpu_kick() actually required? The PIC logic invoked from hpet_route_interrupt() should do any required kicking. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Cui, Dexuan
2007-Jan-10 11:53 UTC
RE: [Xen-devel] [Patch] Fix the slow wall clock time issue in x64 SMP Vista
1) I made the change mainly for timer1 of HPET. Actually I don''t know exactly how x64 SMP Vista uses HPET''s timer1 to adjust the wall clock time; but without the patch, HPET''s timer1 can generate and inject interrupts with a frequency of 20KHz or so! and the wall clock time would become slow. 2) Yes. I think the vcpu_kick() in hpet_timer_fn() is actually unnecessary. Thanks for pointing out this! -- Dexuan -----Original Message----- From: Keir Fraser [mailto:keir@xensource.com] Sent: 2007年1月10日 19:37 To: Cui, Dexuan; xen-devel@lists.xensource.com Cc: Keir Fraser Subject: Re: [Xen-devel] [Patch] Fix the slow wall clock time issue in x64 SMP Vista On 10/1/07 11:29, "Cui, Dexuan" <dexuan.cui@intel.com> wrote:> In x64 SMP Vista HVM guest (vcpus=2 in the configuration file), the wall > clock time is 50% slower than that in the real world. The attached patch > fixes the issue.Looks like this code will inject fewer HPET interrupts. How does that make time go *faster*? Also, looking at hpet_timer_fn() is vcpu_kick() actually required? The PIC logic invoked from hpet_route_interrupt() should do any required kicking. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Cui, Dexuan
2007-Jan-10 12:00 UTC
RE: [Xen-devel] [Patch] Fix the slow wall clock time issue in x64 SMPVista
I just removed the vcpu_kick() in hpet_timer_fn(), and booted an SMP FC5 guest: I found a warning message on the console: Warning: many lost ticks. Your time source seems to be instable or some driver is hogging interrupts rip __do_softirq+0x4a/0xd2 Falling back to HPET So I need do more to see whether we can remove the vcpu_kick(). -- Dexuan -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Cui, Dexuan Sent: 2007年1月10日 19:54 To: Keir Fraser; xen-devel@lists.xensource.com Subject: RE: [Xen-devel] [Patch] Fix the slow wall clock time issue in x64 SMPVista 1) I made the change mainly for timer1 of HPET. Actually I don''t know exactly how x64 SMP Vista uses HPET''s timer1 to adjust the wall clock time; but without the patch, HPET''s timer1 can generate and inject interrupts with a frequency of 20KHz or so! and the wall clock time would become slow. 2) Yes. I think the vcpu_kick() in hpet_timer_fn() is actually unnecessary. Thanks for pointing out this! -- Dexuan -----Original Message----- From: Keir Fraser [mailto:keir@xensource.com] Sent: 2007年1月10日 19:37 To: Cui, Dexuan; xen-devel@lists.xensource.com Cc: Keir Fraser Subject: Re: [Xen-devel] [Patch] Fix the slow wall clock time issue in x64 SMP Vista On 10/1/07 11:29, "Cui, Dexuan" <dexuan.cui@intel.com> wrote:> In x64 SMP Vista HVM guest (vcpus=2 in the configuration file), the wall > clock time is 50% slower than that in the real world. The attached patch > fixes the issue.Looks like this code will inject fewer HPET interrupts. How does that make time go *faster*? Also, looking at hpet_timer_fn() is vcpu_kick() actually required? The PIC logic invoked from hpet_route_interrupt() should do any required kicking. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2007-Jan-10 12:02 UTC
Re: [Xen-devel] [Patch] Fix the slow wall clock time issue in x64 SMP Vista
On 10/1/07 11:53, "Cui, Dexuan" <dexuan.cui@intel.com> wrote:> 1) I made the change mainly for timer1 of HPET. Actually I don''t know exactly > how x64 SMP Vista uses HPET''s timer1 to adjust the wall clock time; but > without the patch, HPET''s timer1 can generate and inject interrupts with a > frequency of 20KHz or so! and the wall clock time would become slow.I cannot see the logic behind the change though. If the guest has set up a timeout, and the timeout is in the past, it makes sense to deliver immediately. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Cui, Dexuan
2007-Jan-10 13:09 UTC
RE: [Xen-devel] [Patch] Fix the slow wall clock time issue in x64 SMP Vista
>I cannot see the logic behind the change though. If the guest has set up a >timeout, and the timeout is in the past, it makes sense to deliver >immediately.Some guests may set up on purpose a timeout which is in the "past"? Following are a patch snippet and some output results when I boot x64 SMP Vista (my CPU is 2.133G; the results are printed 1 line per second or so); it seems x64 SMP Vista is doing some kind of adjustment using HPET''s timer1. + scheduled = tn_cmp - cur_tick; + if ( (int64_t)scheduled < 0 ) + { + printk("t%u_cmp= %lu, cur_tick= %lu, scheduled = negative %lu \n", tn, tn_cmp, cur_tick, - scheduled); + passed = cur_tick - tn_cmp; + + /* the number of HPET tick that stands for + * 1/(2^10) second, namely, 0.9765625 milliseconds */ + #define hpet_tiny_time_span (h->tsc_freq >> 10) + if ( passed < hpet_tiny_time_span ) + scheduled = 0; + else (XEN) t1_cmp= 0, cur_tick= 1947268012, scheduled = negative 1947268012 (XEN) t1_cmp= 2147483648, cur_tick= 4258824668, scheduled = negative 2111341020 (XEN) t1_cmp= 0, cur_tick= 1993055268, scheduled = negative 1993055268 (XEN) t1_cmp= 2147483648, cur_tick= 3917908772, scheduled = negative 1770425124 (XEN) t1_cmp= 0, cur_tick= 1858024708, scheduled = negative 1858024708 (XEN) t1_cmp= 2147483648, cur_tick= 4000005532, scheduled = negative 1852521884 (XEN) t1_cmp= 0, cur_tick= 2089118628, scheduled = negative 2089118628 (XEN) t1_cmp= 2147483648, cur_tick= 4012630076, scheduled = negative 1865146428 (XEN) t1_cmp= 0, cur_tick= 2003908668, scheduled = negative 2003908668 (XEN) t1_cmp= 2147483648, cur_tick= 4075156388, scheduled = negative 1927672740 (XEN) t1_cmp= 0, cur_tick= 1819502692, scheduled = negative 1819502692 (XEN) t1_cmp= 0, cur_tick= 3661310188, scheduled = negative 3661310188 (XEN) t1_cmp= 2147483648, cur_tick= 4135112804, scheduled = negative 1987629156 (XEN) t1_cmp= 0, cur_tick= 1642903140, scheduled = negative 1642903140 (XEN) t1_cmp= 0, cur_tick= 3728809316, scheduled = negative 3728809316 (XEN) t1_cmp= 2147483648, cur_tick= 4161725460, scheduled = negative 2014241812 (XEN) t1_cmp= 0, cur_tick= 1722051132, scheduled = negative 1722051132 (XEN) t1_cmp= 2147483648, cur_tick= 3950346516, scheduled = negative 1802862868 (XEN) t1_cmp= 0, cur_tick= 1904943348, scheduled = negative 1904943348 (XEN) t1_cmp= 2147483648, cur_tick= 4128329004, scheduled = negative 1980845356 (XEN) t1_cmp= 0, cur_tick= 2065056964, scheduled = negative 2065056964 (XEN) t1_cmp= 2147483648, cur_tick= 4150274868, scheduled = negative 2002791220 (XEN) t1_cmp= 0, cur_tick= 2139854452, scheduled = negative 2139854452 (XEN) t1_cmp= 2147483648, cur_tick= 4146800116, scheduled = negative 1999316468 (XEN) t1_cmp= 0, cur_tick= 2131365484, scheduled = negative 2131365484 (XEN) t1_cmp= 2147483648, cur_tick= 4270226844, scheduled = negative 2122743196 (XEN) t1_cmp= 0, cur_tick= 2146031844, scheduled = negative 2146031844 (XEN) t1_cmp= 2147483648, cur_tick= 4279941940, scheduled = negative 2132458292 (XEN) t1_cmp= 0, cur_tick= 2119305068, scheduled = negative 2119305068 (XEN) t1_cmp= 2147483648, cur_tick= 4289293244, scheduled = negative 2141809596 (XEN) t1_cmp= 0, cur_tick= 2115127940, scheduled = negative 2115127940 (XEN) t1_cmp= 2147483648, cur_tick= 4272576276, scheduled = negative 2125092628 (XEN) t1_cmp= 0, cur_tick= 2138209348, scheduled = negative 2138209348 (XEN) t1_cmp= 2147483648, cur_tick= 4172725236, scheduled = negative 2025241588 (XEN) t1_cmp= 0, cur_tick= 2124112860, scheduled = negative 2124112860 (XEN) t1_cmp= 2147483648, cur_tick= 4232138948, scheduled = negative 2084655300 btw, I can not reproduce the "Warning: many lost ticks." message now. :( -- Dexuan -----Original Message----- From: Keir Fraser [mailto:keir@xensource.com] Sent: 2007年1月10日 20:02 To: Cui, Dexuan; Keir Fraser; xen-devel@lists.xensource.com Subject: Re: [Xen-devel] [Patch] Fix the slow wall clock time issue in x64 SMP Vista On 10/1/07 11:53, "Cui, Dexuan" <dexuan.cui@intel.com> wrote:> 1) I made the change mainly for timer1 of HPET. Actually I don''t know exactly > how x64 SMP Vista uses HPET''s timer1 to adjust the wall clock time; but > without the patch, HPET''s timer1 can generate and inject interrupts with a > frequency of 20KHz or so! and the wall clock time would become slow.I cannot see the logic behind the change though. If the guest has set up a timeout, and the timeout is in the past, it makes sense to deliver immediately. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2007-Jan-10 13:45 UTC
Re: [Xen-devel] [Patch] Fix the slow wall clock time issue in x64 SMP Vista
On 10/1/07 13:09, "Cui, Dexuan" <dexuan.cui@intel.com> wrote:> Some guests may set up on purpose a timeout which is in the "past"?Does Windows set up the timer to use 32-bit comparator? That would make a difference! -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2007-Jan-10 13:49 UTC
Re: [Xen-devel] [Patch] Fix the slow wall clock time issue in x64 SMPVista
On 10/1/07 12:00, "Cui, Dexuan" <dexuan.cui@intel.com> wrote:> I just removed the vcpu_kick() in hpet_timer_fn(), and booted an SMP FC5 > guest: > I found a warning message on the console: > > Warning: many lost ticks. > Your time source seems to be instable or some driver is hogging interrupts > rip __do_softirq+0x4a/0xd2 > Falling back to HPETGetting this message must just be a weird coincidence. Linux never uses HPET as its source of periodic tick interrupts. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dong, Eddie
2007-Jan-10 14:42 UTC
RE: [Xen-devel] [Patch] Fix the slow wall clock time issue in x64SMP Vista
> I cannot see the logic behind the change though. If the guest has set > up a timeout, and the timeout is in the past, it makes sense to > deliver immediately. >Basically I agree too, but the problem is that how can we determine it is a past time?>From HW point of view, it doesn''t think it is a "past" time, so it willthink it is a time long ahead. HPET spec mentions that SW should set a comparator far enough to avoid HW see a "past" time set. But in our virtualization environment, it could become a "past" time due to overhead of virtualization cost. Thus putting a threshold for this situation may help us. Because HPET is very similar with IA64 ITC timer, in IA64 Linux, the OS will try to read back the counter after comparator setting (ITC or TSC in X86) to make sure it is before the set comparator, otherwise reset with a new comparator. Not sure how Windows HPET handle this, but probably we can assume similar mechanism in OS. thx,eddie _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dong, Eddie
2007-Jan-11 13:29 UTC
RE: [Xen-devel] [Patch] Fix the slow wall clock time issue in x64 SMPVista
> (XEN) t1_cmp= 0, cur_tick= 1947268012, scheduled = negative > 1947268012 (XEN) t1_cmp= 2147483648, cur_tick= 4258824668, scheduled > = negative 2111341020 (XEN) t1_cmp= 0, cur_tick= 1993055268, > scheduled = negative 1993055268 (XEN) t1_cmp= 2147483648, cur_tick> 3917908772, scheduled = negative 1770425124 (XEN) t1_cmp= 0, > cur_tick= 1858024708, scheduled = negative 1858024708 (XEN) t1_cmp> 2147483648, cur_tick= 4000005532, scheduled = negative 1852521884 > (XEN) t1_cmp= 0, cur_tick= 2089118628, scheduled = negative > 2089118628 (XEN) t1_cmp= 2147483648, cur_tick= 4012630076, scheduledKeir: From the above printk data and confimed the HW TSC frequency is 2.133GHZ, my reading to that is 1: Guest HPET frequency=2.133GHZ 2: The guest is setting comparator to 0 or 0x80000000 (2147483648 in BCD). = 1s period. 3: For some reason, current counter seems a little bit ahead than 0x80000000 and 0, thus make a negative value to current counter, i.e. "past" time in previous logic. So I think the patch is mostly needed except some coding style issues. But I didn''t understand why it makes Vista wall clock slow 50% either :-( Thx,eddie _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel