Attached patch adds clocksource=tsc boot option that uses TSC as clocksource. This option should only be used on machines where TSC is known to be synchronized across all processors. A future TODO is to dynamically determine if this is the case. TSC may "beat" with another clocksource, resulting in cross-processor Xen system time skew. This skew can be visible in PV guests and can appear as "time is stopped" in hvm guests. On some systems, this patch can reduce skew by 30x or more. Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com> ==================================Thanks... for the memory I really could use more / My throughput's on the floor The balloon is flat / My swap disk's fat / I've OOM's in store Overcommitted so much (with apologies to the late great Bob Hope) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I meant to add... although this appears to be new functionality
and thus too late for 3.3, I'd like to argue that it is really
a bug fix: Let's say there are good-tsc machines and bad-tsc
machines. Prior to cset 17716 ("hvm: Build guest timers on
monotonic system time") all guest platform timer reads were
built directly on top of physical TSC. This was a big problem
for bad-tsc SMP machines, but fine for good-tsc machines. Cset 17716
made things better for bad-tsc machines, but actually made them
worse for good-tsc SMP machines (because of the inter-processor
skew).
So users with good-tsc machines moving from a pre-3.3 release
to 3.3 may see time "stopped" in 3.3 when there was no problem
with pre-3.3. The clocksource=tsc patch partially
fixes this "regression" for good-tsc machines... at least it
provides a boot option to fix it.
Thanks,
Dan
> -----Original Message-----
> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> Sent: Saturday, July 12, 2008 3:38 PM
> To: Xen-Devel (E-mail); Keir Fraser; dan.magenheimer@oracle.com
> Cc: Dave Winchell
> Subject: [PATCH] clocksource=tsc
>
>
> Attached patch adds clocksource=tsc boot option that
> uses TSC as clocksource. This option should only be
> used on machines where TSC is known to be synchronized
> across all processors. A future TODO is to dynamically
> determine if this is the case.
>
> TSC may "beat" with another clocksource, resulting in
> cross-processor Xen system time skew. This skew can
> be visible in PV guests and can appear as "time is stopped"
> in hvm guests. On some systems, this patch can reduce skew
> by 30x or more.
>
> Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>
>
> ==================================> Thanks... for the memory
> I really could use more / My throughput's on the floor
> The balloon is flat / My swap disk's fat / I've OOM's in store
> Overcommitted so much
> (with apologies to the late great Bob Hope)
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
I''m sympathetic to the patch, but it looks like an ugly hack right now. Most of the clocksource_is_tsc special cases will fall away if you, e.g., have platform_timer_stamp = stime_platform_stamp = local_tsc_stamp stime_local_stamp = 0; and provide a plt->read_counter() function; and accept that for now you will unnecessarily take the platform-timer spinlock; etc. Also it is platform timer 5, not 4! Clean it up and it''ll go in. Since it should clearly only affect behaviour when the new clocksource is explicitly selected, I''ll take this patch further into 3.3 freeze than usual -- it''s more important to me that it''s clean. Thanks, Keir On 12/7/08 22:38, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:> Attached patch adds clocksource=tsc boot option that > uses TSC as clocksource. This option should only be > used on machines where TSC is known to be synchronized > across all processors. A future TODO is to dynamically > determine if this is the case. > > TSC may "beat" with another clocksource, resulting in > cross-processor Xen system time skew. This skew can > be visible in PV guests and can appear as "time is stopped" > in hvm guests. On some systems, this patch can reduce skew > by 30x or more. > > Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com> > > ==================================> Thanks... for the memory > I really could use more / My throughput''s on the floor > The balloon is flat / My swap disk''s fat / I''ve OOM''s in store > Overcommitted so much > (with apologies to the late great Bob Hope)_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> I''m sympathetic to the patch, but it looks like an ugly hack > right now.> Since it should clearly only affect behaviour when the new > clocksource is explicitly selected...Actually the ugliness is because the read_counter function and plt_mask are both u32 and I worried that changing them to a u64 would be too invasive. So here''s two possible replacement patches, one with those switched to u64 and the other with separate 32- and 64-bit read_counter functions. Both are boot-tested with and without clocksource=tsc. Dan> -----Original Message----- > From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] > Sent: Monday, July 14, 2008 3:24 AM > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > Cc: Dave Winchell > Subject: Re: [PATCH] clocksource=tsc > > > I''m sympathetic to the patch, but it looks like an ugly hack > right now. Most > of the clocksource_is_tsc special cases will fall away if > you, e.g., have > platform_timer_stamp = stime_platform_stamp = local_tsc_stamp > stime_local_stamp = 0; and provide a plt->read_counter() function; and > accept that for now you will unnecessarily take the > platform-timer spinlock; > etc. Also it is platform timer 5, not 4! > > Clean it up and it''ll go in. Since it should clearly only > affect behaviour > when the new clocksource is explicitly selected, I''ll take this patch > further into 3.3 freeze than usual -- it''s more important to > me that it''s > clean. > > Thanks, > Keir > > On 12/7/08 22:38, "Dan Magenheimer" > <dan.magenheimer@oracle.com> wrote: > > > Attached patch adds clocksource=tsc boot option that > > uses TSC as clocksource. This option should only be > > used on machines where TSC is known to be synchronized > > across all processors. A future TODO is to dynamically > > determine if this is the case. > > > > TSC may "beat" with another clocksource, resulting in > > cross-processor Xen system time skew. This skew can > > be visible in PV guests and can appear as "time is stopped" > > in hvm guests. On some systems, this patch can reduce skew > > by 30x or more. > > > > Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com> > > > > ==================================> > Thanks... for the memory > > I really could use more / My throughput''s on the floor > > The balloon is flat / My swap disk''s fat / I''ve OOM''s in store > > Overcommitted so much > > (with apologies to the late great Bob Hope) > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>From: Dan Magenheimer >Sent: 2008年7月15日 2:00 > >> I''m sympathetic to the patch, but it looks like an ugly hack >> right now. > >> Since it should clearly only affect behaviour when the new >> clocksource is explicitly selected... > >Actually the ugliness is because the read_counter function >and plt_mask are both u32 and I worried that changing them to >a u64 would be too invasive. > >So here''s two possible replacement patches, one with those >switched to u64 and the other with separate 32- and 64-bit >read_counter functions. Both are boot-tested with and >without clocksource=tsc. > >DanBoot option is good enough for now, but finally I guess you may need some type of dynamic clocksource switch mechanism, just like what Linux does today. User may be not sure of the specific platform. Some features may affect TSC stabilitity, e.g. Px may have freq change and Cx may have TSC stop on some platform. On those path a mark_tsc_unstable step is required to switch current clocksource to other available platfrom timer on the fly. So besides the boot option, Xen itself needs to code such condition checks for TSC stability. :-) Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 14/7/08 18:59, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:>> I''m sympathetic to the patch, but it looks like an ugly hack >> right now. > >> Since it should clearly only affect behaviour when the new >> clocksource is explicitly selected... > > Actually the ugliness is because the read_counter function > and plt_mask are both u32 and I worried that changing them to > a u64 would be too invasive. > > So here''s two possible replacement patches, one with those > switched to u64 and the other with separate 32- and 64-bit > read_counter functions. Both are boot-tested with and > without clocksource=tsc.Changeset 18055 is what I had in mind. I split out the read_counter-is-64-bit portion separately. The changeset needs a bit of testing and it may need a tweak or two. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Changeset 18055 is what I had in mind. I split out the > read_counter-is-64-bit portion separately. The changeset > needs a bit of > testing and it may need a tweak or two.Hmmm... One thing I was trying to do with the special casing in get_s_time() was to avoid using a dynamically changing scaling (t->tsc_scale). If I''m not mistaken, t->tsc_scale is recalculated every EPOCH and thus is a potential source of stime jitter. Seems unnecessary just to make the code look cleaner. One other nit: Especially if the above is changed, do you really prefer a strcmp vs a global variable (or checking for size == 64) for determining if the tsc is the platform timer? Thanks, Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 15/7/08 15:44, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:> Hmmm... One thing I was trying to do with the special > casing in get_s_time() was to avoid using a dynamically > changing scaling (t->tsc_scale). If I''m not mistaken, > t->tsc_scale is recalculated every EPOCH and thus is > a potential source of stime jitter. Seems unnecessary > just to make the code look cleaner.My patch disables the per-epoch calibration. Actually in this mode of operation we hardly need a platform timer *at all*. The idea is that we let the TSCs free-run, because we know they will behave. Returning to 32-bit read_counter(), and having NULL read_counter when clocksource=tsc would be another possibility...> One other nit: Especially if the above is changed, do > you really prefer a strcmp vs a global variable (or > checking for size == 64) for determining if the tsc > is the platform timer?It''s hidden in a macro and it''s called very rarely. So I think it''s fine. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > Hmmm... One thing I was trying to do with the special > > casing in get_s_time() was to avoid using a dynamically > > changing scaling (t->tsc_scale). If I''m not mistaken, > > t->tsc_scale is recalculated every EPOCH and thus is > > a potential source of stime jitter. Seems unnecessary > > just to make the code look cleaner. > > My patch disables the per-epoch calibration.OK, then I think it is the calls to calibrate_tsc_ap() that result in different values of tsc_scale (potentially a different one for every processor depending on the precision of the calibration). I still think it would be clearer and safer to always use a fixed tsc_scale value.> Actually in this mode of operation we hardly need a platform > timer *at all*. > The idea is that we let the TSCs free-run, because we know > they will behave. > Returning to 32-bit read_counter(), and having NULL read_counter when > clocksource=tsc would be another possibility...That''s essentially what the original tscstable.patch did, though I was perhaps much uglier in the miscellaneous parts. Thanks, Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hmmm... 18055 also fails to boot on my machine. Could we perhaps fall back to my original patch and do cleanup later/separately? I also want to try implementing an hpet64-based get_s_time() so will be working more in this code later... but want to get clocksource=tsc working now with minimal code impact given the freeze.> -----Original Message----- > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > Sent: Tuesday, July 15, 2008 9:46 AM > To: ''Keir Fraser''; ''Xen-Devel (E-mail)'' > Cc: ''Dave Winchell'' > Subject: RE: [PATCH] clocksource=tsc > > > Actually in this mode of operation we hardly need a platform > > timer *at all*. > > The idea is that we let the TSCs free-run, because we know > > they will behave. > > Returning to 32-bit read_counter(), and having NULL > read_counter when > > clocksource=tsc would be another possibility... > > That''s essentially what the original tscstable.patch did, though > I was perhaps much uglier in the miscellaneous parts. > > Thanks, > Dan >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > > Returning to 32-bit read_counter(), and having NULL > > read_counter when > > > clocksource=tsc would be another possibility...Well I hacked on 18055 for awhile and just couldn''t get it to boot. I think local_time_calibration() (and thus init_percpu_time()) is necessary for boot, though I''m not really sure why. Possibly the "Weirdness can happen..." comment in that routine? Anyway, this patch (on top of 18055) DOES work, returns to the 32-bit read_counter, and re-enables local_time_calibration(). I''d suggest putting off more major surgery for another day. Thanks, Dan> -----Original Message----- > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > Sent: Tuesday, July 15, 2008 10:04 AM > To: dan.magenheimer@oracle.com; Keir Fraser; Xen-Devel (E-mail) > Cc: Dave Winchell > Subject: RE: [PATCH] clocksource=tsc > > > Hmmm... 18055 also fails to boot on my machine. > > Could we perhaps fall back to my original patch and do > cleanup later/separately? I also want to try implementing > an hpet64-based get_s_time() so will be working more > in this code later... but want to get clocksource=tsc > working now with minimal code impact given the freeze. > > > -----Original Message----- > > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > Sent: Tuesday, July 15, 2008 9:46 AM > > To: ''Keir Fraser''; ''Xen-Devel (E-mail)'' > > Cc: ''Dave Winchell'' > > Subject: RE: [PATCH] clocksource=tsc > > > > > Actually in this mode of operation we hardly need a platform > > > timer *at all*. > > > The idea is that we let the TSCs free-run, because we know > > > they will behave. > > > Returning to 32-bit read_counter(), and having NULL > > read_counter when > > > clocksource=tsc would be another possibility... > > > > That''s essentially what the original tscstable.patch did, though > > I was perhaps much uglier in the miscellaneous parts. > > > > Thanks, > > Dan > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
OK, ignore that. It looks like you have it all fixed in 18060. I tried it and it now boots. Thanks!> -----Original Message----- > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > Sent: Tuesday, July 15, 2008 7:16 PM > To: ''dan.magenheimer@oracle.com''; ''Keir Fraser''; ''Xen-Devel (E-mail)'' > Cc: ''Dave Winchell'' > Subject: RE: [PATCH] clocksource=tsc > > > > > > Returning to 32-bit read_counter(), and having NULL > > > read_counter when > > > > clocksource=tsc would be another possibility... > > Well I hacked on 18055 for awhile and just couldn''t get it > to boot. I think local_time_calibration() (and thus > init_percpu_time()) is necessary for boot, though I''m not really > sure why. Possibly the "Weirdness can happen..." comment in > that routine? > > Anyway, this patch (on top of 18055) DOES work, returns to the > 32-bit read_counter, and re-enables local_time_calibration(). > I''d suggest putting off more major surgery for another day. > > Thanks, > Dan > > > -----Original Message----- > > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > Sent: Tuesday, July 15, 2008 10:04 AM > > To: dan.magenheimer@oracle.com; Keir Fraser; Xen-Devel (E-mail) > > Cc: Dave Winchell > > Subject: RE: [PATCH] clocksource=tsc > > > > > > Hmmm... 18055 also fails to boot on my machine. > > > > Could we perhaps fall back to my original patch and do > > cleanup later/separately? I also want to try implementing > > an hpet64-based get_s_time() so will be working more > > in this code later... but want to get clocksource=tsc > > working now with minimal code impact given the freeze. > > > > > -----Original Message----- > > > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > > Sent: Tuesday, July 15, 2008 9:46 AM > > > To: ''Keir Fraser''; ''Xen-Devel (E-mail)'' > > > Cc: ''Dave Winchell'' > > > Subject: RE: [PATCH] clocksource=tsc > > > > > > > Actually in this mode of operation we hardly need a platform > > > > timer *at all*. > > > > The idea is that we let the TSCs free-run, because we know > > > > they will behave. > > > > Returning to 32-bit read_counter(), and having NULL > > > read_counter when > > > > clocksource=tsc would be another possibility... > > > > > > That''s essentially what the original tscstable.patch did, though > > > I was perhaps much uglier in the miscellaneous parts. > > > > > > Thanks, > > > Dan > > > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Well now I have to take that back. It DOESN''T work yet.
I think I am experiencing "Weirdness can happen..."
when booting with clocksource=tsc... I was away from
the machine overnight but the symptoms I''ve seen before
are that the system becomes less snappy and eventually
grinds to a near-halt.
Oddly, I can login most of the way on the console
and launch new xterm''s in my VNC display, but I never
get a prompt, and I can''t interrupt a process I left
running overnight in another xterm. The time display
in gnome seems to have frozen about an hour after
I booted. Pinging the machine works but ssh''ing to
it doesn''t ("Connection closed")
> -----Original Message-----
> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> Sent: Tuesday, July 15, 2008 10:12 PM
> To: ''dan.magenheimer@oracle.com''; ''Keir
Fraser''; ''Xen-Devel (E-mail)''
> Cc: ''Dave Winchell''
> Subject: RE: [PATCH] clocksource=tsc
>
>
> OK, ignore that. It looks like you have it all fixed
> in 18060. I tried it and it now boots. Thanks!
>
> > -----Original Message-----
> > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> > Sent: Tuesday, July 15, 2008 7:16 PM
> > To: ''dan.magenheimer@oracle.com''; ''Keir
Fraser''; ''Xen-Devel
> (E-mail)''
> > Cc: ''Dave Winchell''
> > Subject: RE: [PATCH] clocksource=tsc
> >
> >
> > > > > Returning to 32-bit read_counter(), and having NULL
> > > > read_counter when
> > > > > clocksource=tsc would be another possibility...
> >
> > Well I hacked on 18055 for awhile and just couldn''t get it
> > to boot. I think local_time_calibration() (and thus
> > init_percpu_time()) is necessary for boot, though I''m not
really
> > sure why. Possibly the "Weirdness can happen..." comment in
> > that routine?
> >
> > Anyway, this patch (on top of 18055) DOES work, returns to the
> > 32-bit read_counter, and re-enables local_time_calibration().
> > I''d suggest putting off more major surgery for another day.
> >
> > Thanks,
> > Dan
> >
> > > -----Original Message-----
> > > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> > > Sent: Tuesday, July 15, 2008 10:04 AM
> > > To: dan.magenheimer@oracle.com; Keir Fraser; Xen-Devel (E-mail)
> > > Cc: Dave Winchell
> > > Subject: RE: [PATCH] clocksource=tsc
> > >
> > >
> > > Hmmm... 18055 also fails to boot on my machine.
> > >
> > > Could we perhaps fall back to my original patch and do
> > > cleanup later/separately? I also want to try implementing
> > > an hpet64-based get_s_time() so will be working more
> > > in this code later... but want to get clocksource=tsc
> > > working now with minimal code impact given the freeze.
> > >
> > > > -----Original Message-----
> > > > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> > > > Sent: Tuesday, July 15, 2008 9:46 AM
> > > > To: ''Keir Fraser''; ''Xen-Devel
(E-mail)''
> > > > Cc: ''Dave Winchell''
> > > > Subject: RE: [PATCH] clocksource=tsc
> > > >
> > > > > Actually in this mode of operation we hardly need a
platform
> > > > > timer *at all*.
> > > > > The idea is that we let the TSCs free-run, because we
know
> > > > > they will behave.
> > > > > Returning to 32-bit read_counter(), and having NULL
> > > > read_counter when
> > > > > clocksource=tsc would be another possibility...
> > > >
> > > > That''s essentially what the original
tscstable.patch did, though
> > > > I was perhaps much uglier in the miscellaneous parts.
> > > >
> > > > Thanks,
> > > > Dan
> > > >
> > >
> > >
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
That''s a weird set of symptoms. Perhaps dom0''s sense of system time is diverging from Xen''s? I don''t see that CPUs can diverge, if their TSCs are in sync, since we shouldn''t be dynamically modifying the per-CPU timestamps or scale factors. -- Keir On 16/7/08 13:43, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:> Well now I have to take that back. It DOESN''T work yet. > I think I am experiencing "Weirdness can happen..." > when booting with clocksource=tsc... I was away from > the machine overnight but the symptoms I''ve seen before > are that the system becomes less snappy and eventually > grinds to a near-halt. > > Oddly, I can login most of the way on the console > and launch new xterm''s in my VNC display, but I never > get a prompt, and I can''t interrupt a process I left > running overnight in another xterm. The time display > in gnome seems to have frozen about an hour after > I booted. Pinging the machine works but ssh''ing to > it doesn''t ("Connection closed") > >> -----Original Message----- >> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >> Sent: Tuesday, July 15, 2008 10:12 PM >> To: ''dan.magenheimer@oracle.com''; ''Keir Fraser''; ''Xen-Devel (E-mail)'' >> Cc: ''Dave Winchell'' >> Subject: RE: [PATCH] clocksource=tsc >> >> >> OK, ignore that. It looks like you have it all fixed >> in 18060. I tried it and it now boots. Thanks! >> >>> -----Original Message----- >>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>> Sent: Tuesday, July 15, 2008 7:16 PM >>> To: ''dan.magenheimer@oracle.com''; ''Keir Fraser''; ''Xen-Devel >> (E-mail)'' >>> Cc: ''Dave Winchell'' >>> Subject: RE: [PATCH] clocksource=tsc >>> >>> >>>>>> Returning to 32-bit read_counter(), and having NULL >>>>> read_counter when >>>>>> clocksource=tsc would be another possibility... >>> >>> Well I hacked on 18055 for awhile and just couldn''t get it >>> to boot. I think local_time_calibration() (and thus >>> init_percpu_time()) is necessary for boot, though I''m not really >>> sure why. Possibly the "Weirdness can happen..." comment in >>> that routine? >>> >>> Anyway, this patch (on top of 18055) DOES work, returns to the >>> 32-bit read_counter, and re-enables local_time_calibration(). >>> I''d suggest putting off more major surgery for another day. >>> >>> Thanks, >>> Dan >>> >>>> -----Original Message----- >>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>> Sent: Tuesday, July 15, 2008 10:04 AM >>>> To: dan.magenheimer@oracle.com; Keir Fraser; Xen-Devel (E-mail) >>>> Cc: Dave Winchell >>>> Subject: RE: [PATCH] clocksource=tsc >>>> >>>> >>>> Hmmm... 18055 also fails to boot on my machine. >>>> >>>> Could we perhaps fall back to my original patch and do >>>> cleanup later/separately? I also want to try implementing >>>> an hpet64-based get_s_time() so will be working more >>>> in this code later... but want to get clocksource=tsc >>>> working now with minimal code impact given the freeze. >>>> >>>>> -----Original Message----- >>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>>> Sent: Tuesday, July 15, 2008 9:46 AM >>>>> To: ''Keir Fraser''; ''Xen-Devel (E-mail)'' >>>>> Cc: ''Dave Winchell'' >>>>> Subject: RE: [PATCH] clocksource=tsc >>>>> >>>>>> Actually in this mode of operation we hardly need a platform >>>>>> timer *at all*. >>>>>> The idea is that we let the TSCs free-run, because we know >>>>>> they will behave. >>>>>> Returning to 32-bit read_counter(), and having NULL >>>>> read_counter when >>>>>> clocksource=tsc would be another possibility... >>>>> >>>>> That''s essentially what the original tscstable.patch did, though >>>>> I was perhaps much uglier in the miscellaneous parts. >>>>> >>>>> Thanks, >>>>> Dan >>>>> >>>> >>>> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I''m thinking the problem may only occur when dom0 is running ntpd. Maybe "setting" the platform time only changes one CPU, or sets the platform time differently than system time?> -----Original Message----- > From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] > Sent: Wednesday, July 16, 2008 6:50 AM > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > Cc: Dave Winchell > Subject: Re: [PATCH] clocksource=tsc > > > That''s a weird set of symptoms. Perhaps dom0''s sense of system time is > diverging from Xen''s? I don''t see that CPUs can diverge, if > their TSCs are > in sync, since we shouldn''t be dynamically modifying the > per-CPU timestamps > or scale factors. > > -- Keir > > On 16/7/08 13:43, "Dan Magenheimer" > <dan.magenheimer@oracle.com> wrote: > > > Well now I have to take that back. It DOESN''T work yet. > > I think I am experiencing "Weirdness can happen..." > > when booting with clocksource=tsc... I was away from > > the machine overnight but the symptoms I''ve seen before > > are that the system becomes less snappy and eventually > > grinds to a near-halt. > > > > Oddly, I can login most of the way on the console > > and launch new xterm''s in my VNC display, but I never > > get a prompt, and I can''t interrupt a process I left > > running overnight in another xterm. The time display > > in gnome seems to have frozen about an hour after > > I booted. Pinging the machine works but ssh''ing to > > it doesn''t ("Connection closed") > > > >> -----Original Message----- > >> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > >> Sent: Tuesday, July 15, 2008 10:12 PM > >> To: ''dan.magenheimer@oracle.com''; ''Keir Fraser''; > ''Xen-Devel (E-mail)'' > >> Cc: ''Dave Winchell'' > >> Subject: RE: [PATCH] clocksource=tsc > >> > >> > >> OK, ignore that. It looks like you have it all fixed > >> in 18060. I tried it and it now boots. Thanks! > >> > >>> -----Original Message----- > >>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > >>> Sent: Tuesday, July 15, 2008 7:16 PM > >>> To: ''dan.magenheimer@oracle.com''; ''Keir Fraser''; ''Xen-Devel > >> (E-mail)'' > >>> Cc: ''Dave Winchell'' > >>> Subject: RE: [PATCH] clocksource=tsc > >>> > >>> > >>>>>> Returning to 32-bit read_counter(), and having NULL > >>>>> read_counter when > >>>>>> clocksource=tsc would be another possibility... > >>> > >>> Well I hacked on 18055 for awhile and just couldn''t get it > >>> to boot. I think local_time_calibration() (and thus > >>> init_percpu_time()) is necessary for boot, though I''m not really > >>> sure why. Possibly the "Weirdness can happen..." comment in > >>> that routine? > >>> > >>> Anyway, this patch (on top of 18055) DOES work, returns to the > >>> 32-bit read_counter, and re-enables local_time_calibration(). > >>> I''d suggest putting off more major surgery for another day. > >>> > >>> Thanks, > >>> Dan > >>> > >>>> -----Original Message----- > >>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > >>>> Sent: Tuesday, July 15, 2008 10:04 AM > >>>> To: dan.magenheimer@oracle.com; Keir Fraser; Xen-Devel (E-mail) > >>>> Cc: Dave Winchell > >>>> Subject: RE: [PATCH] clocksource=tsc > >>>> > >>>> > >>>> Hmmm... 18055 also fails to boot on my machine. > >>>> > >>>> Could we perhaps fall back to my original patch and do > >>>> cleanup later/separately? I also want to try implementing > >>>> an hpet64-based get_s_time() so will be working more > >>>> in this code later... but want to get clocksource=tsc > >>>> working now with minimal code impact given the freeze. > >>>> > >>>>> -----Original Message----- > >>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > >>>>> Sent: Tuesday, July 15, 2008 9:46 AM > >>>>> To: ''Keir Fraser''; ''Xen-Devel (E-mail)'' > >>>>> Cc: ''Dave Winchell'' > >>>>> Subject: RE: [PATCH] clocksource=tsc > >>>>> > >>>>>> Actually in this mode of operation we hardly need a platform > >>>>>> timer *at all*. > >>>>>> The idea is that we let the TSCs free-run, because we know > >>>>>> they will behave. > >>>>>> Returning to 32-bit read_counter(), and having NULL > >>>>> read_counter when > >>>>>> clocksource=tsc would be another possibility... > >>>>> > >>>>> That''s essentially what the original tscstable.patch did, though > >>>>> I was perhaps much uglier in the miscellaneous parts. > >>>>> > >>>>> Thanks, > >>>>> Dan > >>>>> > >>>> > >>>> > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Nope, its not ntpd. Everything seemed fine for at least 15-20 minutes, then I went away for awhile and when I checked back an hour later, same weird symptoms as before. I also tried running with nosmp on the xen boot line (and confirmed that only one processor was started). Same results... after about an hour. So it doesn''t appear to be a result of cross-CPU time issues.> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > I''m thinking the problem may only occur when dom0 is > running ntpd. Maybe "setting" the platform time only > changes one CPU, or sets the platform time differently > than system time? > > > -----Original Message----- > > From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] > > Sent: Wednesday, July 16, 2008 6:50 AM > > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > > Cc: Dave Winchell > > Subject: Re: [PATCH] clocksource=tsc > > > > > > That''s a weird set of symptoms. Perhaps dom0''s sense of > system time is > > diverging from Xen''s? I don''t see that CPUs can diverge, if > > their TSCs are > > in sync, since we shouldn''t be dynamically modifying the > > per-CPU timestamps > > or scale factors. > > > > -- Keir > > > > On 16/7/08 13:43, "Dan Magenheimer" > > <dan.magenheimer@oracle.com> wrote: > > > > > Well now I have to take that back. It DOESN''T work yet. > > > I think I am experiencing "Weirdness can happen..." > > > when booting with clocksource=tsc... I was away from > > > the machine overnight but the symptoms I''ve seen before > > > are that the system becomes less snappy and eventually > > > grinds to a near-halt. > > > > > > Oddly, I can login most of the way on the console > > > and launch new xterm''s in my VNC display, but I never > > > get a prompt, and I can''t interrupt a process I left > > > running overnight in another xterm. The time display > > > in gnome seems to have frozen about an hour after > > > I booted. Pinging the machine works but ssh''ing to > > > it doesn''t ("Connection closed")_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
My best guess is still that Xen''s calculation of time is diverging from dom0 kernel''s. For some reason that is unclear, since of course the calibration info should be statically determined at boot and never change from there on. -- Keir On 16/7/08 16:42, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:> Nope, its not ntpd. Everything seemed fine for at > least 15-20 minutes, then I went away for awhile > and when I checked back an hour later, same weird > symptoms as before. > > I also tried running with nosmp on the xen boot line > (and confirmed that only one processor was started). > Same results... after about an hour. So it doesn''t > appear to be a result of cross-CPU time issues. > >> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >> >> I''m thinking the problem may only occur when dom0 is >> running ntpd. Maybe "setting" the platform time only >> changes one CPU, or sets the platform time differently >> than system time? >> >>> -----Original Message----- >>> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] >>> Sent: Wednesday, July 16, 2008 6:50 AM >>> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) >>> Cc: Dave Winchell >>> Subject: Re: [PATCH] clocksource=tsc >>> >>> >>> That''s a weird set of symptoms. Perhaps dom0''s sense of >> system time is >>> diverging from Xen''s? I don''t see that CPUs can diverge, if >>> their TSCs are >>> in sync, since we shouldn''t be dynamically modifying the >>> per-CPU timestamps >>> or scale factors. >>> >>> -- Keir >>> >>> On 16/7/08 13:43, "Dan Magenheimer" >>> <dan.magenheimer@oracle.com> wrote: >>> >>>> Well now I have to take that back. It DOESN''T work yet. >>>> I think I am experiencing "Weirdness can happen..." >>>> when booting with clocksource=tsc... I was away from >>>> the machine overnight but the symptoms I''ve seen before >>>> are that the system becomes less snappy and eventually >>>> grinds to a near-halt. >>>> >>>> Oddly, I can login most of the way on the console >>>> and launch new xterm''s in my VNC display, but I never >>>> get a prompt, and I can''t interrupt a process I left >>>> running overnight in another xterm. The time display >>>> in gnome seems to have frozen about an hour after >>>> I booted. Pinging the machine works but ssh''ing to >>>> it doesn''t ("Connection closed") > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
After trying lots of things, I fell back to my known working version of time.c and adapted it forward to match tip. The attached patch (on top of 18063) boots clocksource=tsc and doesn''t display the weirdness for 2-1/2 hours of testing (always showed up around 1 hour before). It may not be elegant but it works and tip doesn''t (with clocksource=tsc). Dan> -----Original Message----- > From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] > Sent: Wednesday, July 16, 2008 6:50 AM > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > Cc: Dave Winchell > Subject: Re: [PATCH] clocksource=tsc > > > That''s a weird set of symptoms. Perhaps dom0''s sense of system time is > diverging from Xen''s? I don''t see that CPUs can diverge, if > their TSCs are > in sync, since we shouldn''t be dynamically modifying the > per-CPU timestamps > or scale factors. > > -- Keir > > On 16/7/08 13:43, "Dan Magenheimer" > <dan.magenheimer@oracle.com> wrote: > > > Well now I have to take that back. It DOESN''T work yet. > > I think I am experiencing "Weirdness can happen..." > > when booting with clocksource=tsc... I was away from > > the machine overnight but the symptoms I''ve seen before > > are that the system becomes less snappy and eventually > > grinds to a near-halt. > > > > Oddly, I can login most of the way on the console > > and launch new xterm''s in my VNC display, but I never > > get a prompt, and I can''t interrupt a process I left > > running overnight in another xterm. The time display > > in gnome seems to have frozen about an hour after > > I booted. Pinging the machine works but ssh''ing to > > it doesn''t ("Connection closed") > > > >> -----Original Message----- > >> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > >> Sent: Tuesday, July 15, 2008 10:12 PM > >> To: ''dan.magenheimer@oracle.com''; ''Keir Fraser''; > ''Xen-Devel (E-mail)'' > >> Cc: ''Dave Winchell'' > >> Subject: RE: [PATCH] clocksource=tsc > >> > >> > >> OK, ignore that. It looks like you have it all fixed > >> in 18060. I tried it and it now boots. Thanks! > >> > >>> -----Original Message----- > >>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > >>> Sent: Tuesday, July 15, 2008 7:16 PM > >>> To: ''dan.magenheimer@oracle.com''; ''Keir Fraser''; ''Xen-Devel > >> (E-mail)'' > >>> Cc: ''Dave Winchell'' > >>> Subject: RE: [PATCH] clocksource=tsc > >>> > >>> > >>>>>> Returning to 32-bit read_counter(), and having NULL > >>>>> read_counter when > >>>>>> clocksource=tsc would be another possibility... > >>> > >>> Well I hacked on 18055 for awhile and just couldn''t get it > >>> to boot. I think local_time_calibration() (and thus > >>> init_percpu_time()) is necessary for boot, though I''m not really > >>> sure why. Possibly the "Weirdness can happen..." comment in > >>> that routine? > >>> > >>> Anyway, this patch (on top of 18055) DOES work, returns to the > >>> 32-bit read_counter, and re-enables local_time_calibration(). > >>> I''d suggest putting off more major surgery for another day. > >>> > >>> Thanks, > >>> Dan > >>> > >>>> -----Original Message----- > >>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > >>>> Sent: Tuesday, July 15, 2008 10:04 AM > >>>> To: dan.magenheimer@oracle.com; Keir Fraser; Xen-Devel (E-mail) > >>>> Cc: Dave Winchell > >>>> Subject: RE: [PATCH] clocksource=tsc > >>>> > >>>> > >>>> Hmmm... 18055 also fails to boot on my machine. > >>>> > >>>> Could we perhaps fall back to my original patch and do > >>>> cleanup later/separately? I also want to try implementing > >>>> an hpet64-based get_s_time() so will be working more > >>>> in this code later... but want to get clocksource=tsc > >>>> working now with minimal code impact given the freeze. > >>>> > >>>>> -----Original Message----- > >>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > >>>>> Sent: Tuesday, July 15, 2008 9:46 AM > >>>>> To: ''Keir Fraser''; ''Xen-Devel (E-mail)'' > >>>>> Cc: ''Dave Winchell'' > >>>>> Subject: RE: [PATCH] clocksource=tsc > >>>>> > >>>>>> Actually in this mode of operation we hardly need a platform > >>>>>> timer *at all*. > >>>>>> The idea is that we let the TSCs free-run, because we know > >>>>>> they will behave. > >>>>>> Returning to 32-bit read_counter(), and having NULL > >>>>> read_counter when > >>>>>> clocksource=tsc would be another possibility... > >>>>> > >>>>> That''s essentially what the original tscstable.patch did, though > >>>>> I was perhaps much uglier in the miscellaneous parts. > >>>>> > >>>>> Thanks, > >>>>> Dan > >>>>> > >>>> > >>>> > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Boot option is good enough for now, but finally I guess you may > need some type of dynamic clocksource switch mechanism, > just like what Linux does today. User may be not sure of the > specific platform. Some features may affect TSC stabilitity, e.g. > Px may have freq change and Cx may have TSC stop on some > platform. On those path a mark_tsc_unstable step is required > to switch current clocksource to other available platfrom timer > on the fly. So besides the boot option, Xen itself needs to code > such condition checks for TSC stability. :-)Yes, agreed. I think there are "good tsc" machines where TSC will never skew, "bad tsc" machines where the skew is apparent at boot, and "grey tsc" machines where there is skew but the skew happens to be small at boot but may grow to be bad post-boot possibly due to Px/Cx. In order to handle all of these here's the algorithm that I'm thinking of: 1) Use processor bits (borrowing code from a recent Linux version) to determine whether a system is likely to be "good tsc" or "bad tsc". Set the tsc_invariant global variable accordingly. 2) When synchronize_tsc_bp()/ap() dynamically evaluates skew, change tsc_invariant if appropriate. 3) If tsc_invariant is set when clocksource is being selected, tsc should be the default clocksource, unless overridden by clocksource= on the boot line OR a new boot parameter "notsc". 4) Write a pair of routines equivalent to synchronize_tsc_bp/ap() but which just returns whether or not TSCs are sync'ed. Call this routine whenever a processor exits from Cx/Px and also on a decaying counter, e.g. 1 second after boot, then 2 seconds after that, then 4 seconds after that, etc. If skew is detected, change the clocksource to the next best and printk the change. 5) I don't know if it is currently "safe" to change clocksources after the initial selection in init_platform_timer() so this may take some work. Comments? Thanks, Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
You shouldn''t have to modify get_s_time(). Guests will still be running the old algorithm (sys_stamp + (now_tsc - stamp_tsc) * scale_tsc), so existing get_s_time() ought to work. In fact it must work. -- Keir On 18/7/08 00:05, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:> After trying lots of things, I fell back to my known > working version of time.c and adapted it forward to > match tip. The attached patch (on top of 18063) boots > clocksource=tsc and doesn''t display the weirdness for > 2-1/2 hours of testing (always showed up around 1 hour before). > > It may not be elegant but it works and tip doesn''t (with > clocksource=tsc). > > Dan > >> -----Original Message----- >> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] >> Sent: Wednesday, July 16, 2008 6:50 AM >> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) >> Cc: Dave Winchell >> Subject: Re: [PATCH] clocksource=tsc >> >> >> That''s a weird set of symptoms. Perhaps dom0''s sense of system time is >> diverging from Xen''s? I don''t see that CPUs can diverge, if >> their TSCs are >> in sync, since we shouldn''t be dynamically modifying the >> per-CPU timestamps >> or scale factors. >> >> -- Keir >> >> On 16/7/08 13:43, "Dan Magenheimer" >> <dan.magenheimer@oracle.com> wrote: >> >>> Well now I have to take that back. It DOESN''T work yet. >>> I think I am experiencing "Weirdness can happen..." >>> when booting with clocksource=tsc... I was away from >>> the machine overnight but the symptoms I''ve seen before >>> are that the system becomes less snappy and eventually >>> grinds to a near-halt. >>> >>> Oddly, I can login most of the way on the console >>> and launch new xterm''s in my VNC display, but I never >>> get a prompt, and I can''t interrupt a process I left >>> running overnight in another xterm. The time display >>> in gnome seems to have frozen about an hour after >>> I booted. Pinging the machine works but ssh''ing to >>> it doesn''t ("Connection closed") >>> >>>> -----Original Message----- >>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>> Sent: Tuesday, July 15, 2008 10:12 PM >>>> To: ''dan.magenheimer@oracle.com''; ''Keir Fraser''; >> ''Xen-Devel (E-mail)'' >>>> Cc: ''Dave Winchell'' >>>> Subject: RE: [PATCH] clocksource=tsc >>>> >>>> >>>> OK, ignore that. It looks like you have it all fixed >>>> in 18060. I tried it and it now boots. Thanks! >>>> >>>>> -----Original Message----- >>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>>> Sent: Tuesday, July 15, 2008 7:16 PM >>>>> To: ''dan.magenheimer@oracle.com''; ''Keir Fraser''; ''Xen-Devel >>>> (E-mail)'' >>>>> Cc: ''Dave Winchell'' >>>>> Subject: RE: [PATCH] clocksource=tsc >>>>> >>>>> >>>>>>>> Returning to 32-bit read_counter(), and having NULL >>>>>>> read_counter when >>>>>>>> clocksource=tsc would be another possibility... >>>>> >>>>> Well I hacked on 18055 for awhile and just couldn''t get it >>>>> to boot. I think local_time_calibration() (and thus >>>>> init_percpu_time()) is necessary for boot, though I''m not really >>>>> sure why. Possibly the "Weirdness can happen..." comment in >>>>> that routine? >>>>> >>>>> Anyway, this patch (on top of 18055) DOES work, returns to the >>>>> 32-bit read_counter, and re-enables local_time_calibration(). >>>>> I''d suggest putting off more major surgery for another day. >>>>> >>>>> Thanks, >>>>> Dan >>>>> >>>>>> -----Original Message----- >>>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>>>> Sent: Tuesday, July 15, 2008 10:04 AM >>>>>> To: dan.magenheimer@oracle.com; Keir Fraser; Xen-Devel (E-mail) >>>>>> Cc: Dave Winchell >>>>>> Subject: RE: [PATCH] clocksource=tsc >>>>>> >>>>>> >>>>>> Hmmm... 18055 also fails to boot on my machine. >>>>>> >>>>>> Could we perhaps fall back to my original patch and do >>>>>> cleanup later/separately? I also want to try implementing >>>>>> an hpet64-based get_s_time() so will be working more >>>>>> in this code later... but want to get clocksource=tsc >>>>>> working now with minimal code impact given the freeze. >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>>>>> Sent: Tuesday, July 15, 2008 9:46 AM >>>>>>> To: ''Keir Fraser''; ''Xen-Devel (E-mail)'' >>>>>>> Cc: ''Dave Winchell'' >>>>>>> Subject: RE: [PATCH] clocksource=tsc >>>>>>> >>>>>>>> Actually in this mode of operation we hardly need a platform >>>>>>>> timer *at all*. >>>>>>>> The idea is that we let the TSCs free-run, because we know >>>>>>>> they will behave. >>>>>>>> Returning to 32-bit read_counter(), and having NULL >>>>>>> read_counter when >>>>>>>> clocksource=tsc would be another possibility... >>>>>>> >>>>>>> That''s essentially what the original tscstable.patch did, though >>>>>>> I was perhaps much uglier in the miscellaneous parts. >>>>>>> >>>>>>> Thanks, >>>>>>> Dan >>>>>>> >>>>>> >>>>>> >>> >> >> >>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Okay, I''m pretty sure the reason your patch works is that system time as calculated in the guest is still being continually calibrated by local_time_calibration() in Xen. You do not turn off that calibration function and hence it will continually update the cpu_time structure with new timestamps and scale factors. Is this really intended? The reason mine doesn''t work is that the time record exposed to the guest *never* changes. The calculation in the guest is thus effectively: sys_time = TSC * scale_factor The problem is that this is sensitive to small errors in the scale factor! And unfortunately we compute a new scale factor, to convert to us, as scale_factor_ns/1000. The resulting scale factor is obviously less accurate, and leads to an accumulating absolute error as the TSC value gets large. So, when all is said and done, what are we actually trying to achieve here and how should it really work? As far as I can see we cannot avoid sending an updated time record to guests periodically. But running local_time_calibration() only for guests and not for Xen system time seems pretty weird to me. Perhaps we should push this all off until after 3.3? -- Keir On 18/7/08 08:24, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:> You shouldn''t have to modify get_s_time(). Guests will still be running the > old algorithm (sys_stamp + (now_tsc - stamp_tsc) * scale_tsc), so existing > get_s_time() ought to work. In fact it must work. > > -- Keir > > On 18/7/08 00:05, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote: > >> After trying lots of things, I fell back to my known >> working version of time.c and adapted it forward to >> match tip. The attached patch (on top of 18063) boots >> clocksource=tsc and doesn''t display the weirdness for >> 2-1/2 hours of testing (always showed up around 1 hour before). >> >> It may not be elegant but it works and tip doesn''t (with >> clocksource=tsc). >> >> Dan >> >>> -----Original Message----- >>> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] >>> Sent: Wednesday, July 16, 2008 6:50 AM >>> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) >>> Cc: Dave Winchell >>> Subject: Re: [PATCH] clocksource=tsc >>> >>> >>> That''s a weird set of symptoms. Perhaps dom0''s sense of system time is >>> diverging from Xen''s? I don''t see that CPUs can diverge, if >>> their TSCs are >>> in sync, since we shouldn''t be dynamically modifying the >>> per-CPU timestamps >>> or scale factors. >>> >>> -- Keir >>> >>> On 16/7/08 13:43, "Dan Magenheimer" >>> <dan.magenheimer@oracle.com> wrote: >>> >>>> Well now I have to take that back. It DOESN''T work yet. >>>> I think I am experiencing "Weirdness can happen..." >>>> when booting with clocksource=tsc... I was away from >>>> the machine overnight but the symptoms I''ve seen before >>>> are that the system becomes less snappy and eventually >>>> grinds to a near-halt. >>>> >>>> Oddly, I can login most of the way on the console >>>> and launch new xterm''s in my VNC display, but I never >>>> get a prompt, and I can''t interrupt a process I left >>>> running overnight in another xterm. The time display >>>> in gnome seems to have frozen about an hour after >>>> I booted. Pinging the machine works but ssh''ing to >>>> it doesn''t ("Connection closed") >>>> >>>>> -----Original Message----- >>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>>> Sent: Tuesday, July 15, 2008 10:12 PM >>>>> To: ''dan.magenheimer@oracle.com''; ''Keir Fraser''; >>> ''Xen-Devel (E-mail)'' >>>>> Cc: ''Dave Winchell'' >>>>> Subject: RE: [PATCH] clocksource=tsc >>>>> >>>>> >>>>> OK, ignore that. It looks like you have it all fixed >>>>> in 18060. I tried it and it now boots. Thanks! >>>>> >>>>>> -----Original Message----- >>>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>>>> Sent: Tuesday, July 15, 2008 7:16 PM >>>>>> To: ''dan.magenheimer@oracle.com''; ''Keir Fraser''; ''Xen-Devel >>>>> (E-mail)'' >>>>>> Cc: ''Dave Winchell'' >>>>>> Subject: RE: [PATCH] clocksource=tsc >>>>>> >>>>>> >>>>>>>>> Returning to 32-bit read_counter(), and having NULL >>>>>>>> read_counter when >>>>>>>>> clocksource=tsc would be another possibility... >>>>>> >>>>>> Well I hacked on 18055 for awhile and just couldn''t get it >>>>>> to boot. I think local_time_calibration() (and thus >>>>>> init_percpu_time()) is necessary for boot, though I''m not really >>>>>> sure why. Possibly the "Weirdness can happen..." comment in >>>>>> that routine? >>>>>> >>>>>> Anyway, this patch (on top of 18055) DOES work, returns to the >>>>>> 32-bit read_counter, and re-enables local_time_calibration(). >>>>>> I''d suggest putting off more major surgery for another day. >>>>>> >>>>>> Thanks, >>>>>> Dan >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>>>>> Sent: Tuesday, July 15, 2008 10:04 AM >>>>>>> To: dan.magenheimer@oracle.com; Keir Fraser; Xen-Devel (E-mail) >>>>>>> Cc: Dave Winchell >>>>>>> Subject: RE: [PATCH] clocksource=tsc >>>>>>> >>>>>>> >>>>>>> Hmmm... 18055 also fails to boot on my machine. >>>>>>> >>>>>>> Could we perhaps fall back to my original patch and do >>>>>>> cleanup later/separately? I also want to try implementing >>>>>>> an hpet64-based get_s_time() so will be working more >>>>>>> in this code later... but want to get clocksource=tsc >>>>>>> working now with minimal code impact given the freeze. >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>>>>>> Sent: Tuesday, July 15, 2008 9:46 AM >>>>>>>> To: ''Keir Fraser''; ''Xen-Devel (E-mail)'' >>>>>>>> Cc: ''Dave Winchell'' >>>>>>>> Subject: RE: [PATCH] clocksource=tsc >>>>>>>> >>>>>>>>> Actually in this mode of operation we hardly need a platform >>>>>>>>> timer *at all*. >>>>>>>>> The idea is that we let the TSCs free-run, because we know >>>>>>>>> they will behave. >>>>>>>>> Returning to 32-bit read_counter(), and having NULL >>>>>>>> read_counter when >>>>>>>>> clocksource=tsc would be another possibility... >>>>>>>> >>>>>>>> That''s essentially what the original tscstable.patch did, though >>>>>>>> I was perhaps much uglier in the miscellaneous parts. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Dan >>>>>>>> >>>>>>> >>>>>>> >>>> >>> >>> >>> > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ah, I changed my mind. I now think it''s due to overflow of a 32-bit usecs variable in Linux''s do_gettimeofday(). Cause is still the same though -- the guest assuming that the timestamp record from Xen will never be ''very'' stale (in this case, older than 1h12m). -- Keir On 18/7/08 12:01, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:> Okay, I''m pretty sure the reason your patch works is that system time as > calculated in the guest is still being continually calibrated by > local_time_calibration() in Xen. You do not turn off that calibration > function and hence it will continually update the cpu_time structure with > new timestamps and scale factors. Is this really intended? > > The reason mine doesn''t work is that the time record exposed to the guest > *never* changes. The calculation in the guest is thus effectively: > sys_time = TSC * scale_factor > > The problem is that this is sensitive to small errors in the scale factor! > And unfortunately we compute a new scale factor, to convert to us, as > scale_factor_ns/1000. The resulting scale factor is obviously less accurate, > and leads to an accumulating absolute error as the TSC value gets large. > > So, when all is said and done, what are we actually trying to achieve here > and how should it really work? As far as I can see we cannot avoid sending > an updated time record to guests periodically. But running > local_time_calibration() only for guests and not for Xen system time seems > pretty weird to me. > > Perhaps we should push this all off until after 3.3? > > -- Keir > > On 18/7/08 08:24, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote: > >> You shouldn''t have to modify get_s_time(). Guests will still be running the >> old algorithm (sys_stamp + (now_tsc - stamp_tsc) * scale_tsc), so existing >> get_s_time() ought to work. In fact it must work. >> >> -- Keir >> >> On 18/7/08 00:05, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote: >> >>> After trying lots of things, I fell back to my known >>> working version of time.c and adapted it forward to >>> match tip. The attached patch (on top of 18063) boots >>> clocksource=tsc and doesn''t display the weirdness for >>> 2-1/2 hours of testing (always showed up around 1 hour before). >>> >>> It may not be elegant but it works and tip doesn''t (with >>> clocksource=tsc). >>> >>> Dan >>> >>>> -----Original Message----- >>>> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] >>>> Sent: Wednesday, July 16, 2008 6:50 AM >>>> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) >>>> Cc: Dave Winchell >>>> Subject: Re: [PATCH] clocksource=tsc >>>> >>>> >>>> That''s a weird set of symptoms. Perhaps dom0''s sense of system time is >>>> diverging from Xen''s? I don''t see that CPUs can diverge, if >>>> their TSCs are >>>> in sync, since we shouldn''t be dynamically modifying the >>>> per-CPU timestamps >>>> or scale factors. >>>> >>>> -- Keir >>>> >>>> On 16/7/08 13:43, "Dan Magenheimer" >>>> <dan.magenheimer@oracle.com> wrote: >>>> >>>>> Well now I have to take that back. It DOESN''T work yet. >>>>> I think I am experiencing "Weirdness can happen..." >>>>> when booting with clocksource=tsc... I was away from >>>>> the machine overnight but the symptoms I''ve seen before >>>>> are that the system becomes less snappy and eventually >>>>> grinds to a near-halt. >>>>> >>>>> Oddly, I can login most of the way on the console >>>>> and launch new xterm''s in my VNC display, but I never >>>>> get a prompt, and I can''t interrupt a process I left >>>>> running overnight in another xterm. The time display >>>>> in gnome seems to have frozen about an hour after >>>>> I booted. Pinging the machine works but ssh''ing to >>>>> it doesn''t ("Connection closed") >>>>> >>>>>> -----Original Message----- >>>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>>>> Sent: Tuesday, July 15, 2008 10:12 PM >>>>>> To: ''dan.magenheimer@oracle.com''; ''Keir Fraser''; >>>> ''Xen-Devel (E-mail)'' >>>>>> Cc: ''Dave Winchell'' >>>>>> Subject: RE: [PATCH] clocksource=tsc >>>>>> >>>>>> >>>>>> OK, ignore that. It looks like you have it all fixed >>>>>> in 18060. I tried it and it now boots. Thanks! >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>>>>> Sent: Tuesday, July 15, 2008 7:16 PM >>>>>>> To: ''dan.magenheimer@oracle.com''; ''Keir Fraser''; ''Xen-Devel >>>>>> (E-mail)'' >>>>>>> Cc: ''Dave Winchell'' >>>>>>> Subject: RE: [PATCH] clocksource=tsc >>>>>>> >>>>>>> >>>>>>>>>> Returning to 32-bit read_counter(), and having NULL >>>>>>>>> read_counter when >>>>>>>>>> clocksource=tsc would be another possibility... >>>>>>> >>>>>>> Well I hacked on 18055 for awhile and just couldn''t get it >>>>>>> to boot. I think local_time_calibration() (and thus >>>>>>> init_percpu_time()) is necessary for boot, though I''m not really >>>>>>> sure why. Possibly the "Weirdness can happen..." comment in >>>>>>> that routine? >>>>>>> >>>>>>> Anyway, this patch (on top of 18055) DOES work, returns to the >>>>>>> 32-bit read_counter, and re-enables local_time_calibration(). >>>>>>> I''d suggest putting off more major surgery for another day. >>>>>>> >>>>>>> Thanks, >>>>>>> Dan >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>>>>>> Sent: Tuesday, July 15, 2008 10:04 AM >>>>>>>> To: dan.magenheimer@oracle.com; Keir Fraser; Xen-Devel (E-mail) >>>>>>>> Cc: Dave Winchell >>>>>>>> Subject: RE: [PATCH] clocksource=tsc >>>>>>>> >>>>>>>> >>>>>>>> Hmmm... 18055 also fails to boot on my machine. >>>>>>>> >>>>>>>> Could we perhaps fall back to my original patch and do >>>>>>>> cleanup later/separately? I also want to try implementing >>>>>>>> an hpet64-based get_s_time() so will be working more >>>>>>>> in this code later... but want to get clocksource=tsc >>>>>>>> working now with minimal code impact given the freeze. >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>>>>>>> Sent: Tuesday, July 15, 2008 9:46 AM >>>>>>>>> To: ''Keir Fraser''; ''Xen-Devel (E-mail)'' >>>>>>>>> Cc: ''Dave Winchell'' >>>>>>>>> Subject: RE: [PATCH] clocksource=tsc >>>>>>>>> >>>>>>>>>> Actually in this mode of operation we hardly need a platform >>>>>>>>>> timer *at all*. >>>>>>>>>> The idea is that we let the TSCs free-run, because we know >>>>>>>>>> they will behave. >>>>>>>>>> Returning to 32-bit read_counter(), and having NULL >>>>>>>>> read_counter when >>>>>>>>>> clocksource=tsc would be another possibility... >>>>>>>>> >>>>>>>>> That''s essentially what the original tscstable.patch did, though >>>>>>>>> I was perhaps much uglier in the miscellaneous parts. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Dan >>>>>>>>> >>>>>>>> >>>>>>>> >>>>> >>>> >>>> >>>> >> >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
OK, I see. So: If we have a 64-bit precise, monotonic, relatively constant rate (i.e. "ideal") timesource, Xen system time should use it directly. Call this the "new" algorithm. If we do NOT have such a timesource, Xen system time MUST use the (sys_stamp+(now_tsc-stamp_tsc)*scale_tsc) method... the "old" algorithm. Unfortunately, the old algorithm is hard-baked into PV domains. But, to maximize precision in HV domains, we would like to use the new algorithm whenever there is an ideal timesource. In fact, there''s no reason why Xen itself shouldn''t use an ideal timesource if available. So perhaps the compromise which I''ve (admittedly accidentally) crafted is the right answer for now. And the right answer for later is to update the PV time mechanisms (in a backwards compatible way of course) to determine if an ideal timesource is available and use it if it is.> -----Original Message----- > From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] > Sent: Friday, July 18, 2008 5:10 AM > To: Keir Fraser; dan.magenheimer@oracle.com; Xen-Devel (E-mail) > Cc: Dave Winchell > Subject: Re: [Xen-devel] Re: [PATCH] clocksource=tsc > > > Ah, I changed my mind. I now think it''s due to overflow of a > 32-bit usecs > variable in Linux''s do_gettimeofday(). Cause is still the > same though -- the > guest assuming that the timestamp record from Xen will never > be ''very'' stale > (in this case, older than 1h12m). > > -- Keir > > On 18/7/08 12:01, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote: > > > Okay, I''m pretty sure the reason your patch works is that > system time as > > calculated in the guest is still being continually calibrated by > > local_time_calibration() in Xen. You do not turn off that > calibration > > function and hence it will continually update the cpu_time > structure with > > new timestamps and scale factors. Is this really intended? > > > > The reason mine doesn''t work is that the time record > exposed to the guest > > *never* changes. The calculation in the guest is thus effectively: > > sys_time = TSC * scale_factor > > > > The problem is that this is sensitive to small errors in > the scale factor! > > And unfortunately we compute a new scale factor, to convert > to us, as > > scale_factor_ns/1000. The resulting scale factor is > obviously less accurate, > > and leads to an accumulating absolute error as the TSC > value gets large. > > > > So, when all is said and done, what are we actually trying > to achieve here > > and how should it really work? As far as I can see we > cannot avoid sending > > an updated time record to guests periodically. But running > > local_time_calibration() only for guests and not for Xen > system time seems > > pretty weird to me. > > > > Perhaps we should push this all off until after 3.3? > > > > -- Keir > > > > On 18/7/08 08:24, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote: > > > >> You shouldn''t have to modify get_s_time(). Guests will > still be running the > >> old algorithm (sys_stamp + (now_tsc - stamp_tsc) * > scale_tsc), so existing > >> get_s_time() ought to work. In fact it must work. > >> > >> -- Keir > >> > >> On 18/7/08 00:05, "Dan Magenheimer" > <dan.magenheimer@oracle.com> wrote: > >> > >>> After trying lots of things, I fell back to my known > >>> working version of time.c and adapted it forward to > >>> match tip. The attached patch (on top of 18063) boots > >>> clocksource=tsc and doesn''t display the weirdness for > >>> 2-1/2 hours of testing (always showed up around 1 hour before). > >>> > >>> It may not be elegant but it works and tip doesn''t (with > >>> clocksource=tsc). > >>> > >>> Dan > >>> > >>>> -----Original Message----- > >>>> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] > >>>> Sent: Wednesday, July 16, 2008 6:50 AM > >>>> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > >>>> Cc: Dave Winchell > >>>> Subject: Re: [PATCH] clocksource=tsc > >>>> > >>>> > >>>> That''s a weird set of symptoms. Perhaps dom0''s sense of > system time is > >>>> diverging from Xen''s? I don''t see that CPUs can diverge, if > >>>> their TSCs are > >>>> in sync, since we shouldn''t be dynamically modifying the > >>>> per-CPU timestamps > >>>> or scale factors. > >>>> > >>>> -- Keir > >>>> > >>>> On 16/7/08 13:43, "Dan Magenheimer" > >>>> <dan.magenheimer@oracle.com> wrote: > >>>> > >>>>> Well now I have to take that back. It DOESN''T work yet. > >>>>> I think I am experiencing "Weirdness can happen..." > >>>>> when booting with clocksource=tsc... I was away from > >>>>> the machine overnight but the symptoms I''ve seen before > >>>>> are that the system becomes less snappy and eventually > >>>>> grinds to a near-halt. > >>>>> > >>>>> Oddly, I can login most of the way on the console > >>>>> and launch new xterm''s in my VNC display, but I never > >>>>> get a prompt, and I can''t interrupt a process I left > >>>>> running overnight in another xterm. The time display > >>>>> in gnome seems to have frozen about an hour after > >>>>> I booted. Pinging the machine works but ssh''ing to > >>>>> it doesn''t ("Connection closed") > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > >>>>>> Sent: Tuesday, July 15, 2008 10:12 PM > >>>>>> To: ''dan.magenheimer@oracle.com''; ''Keir Fraser''; > >>>> ''Xen-Devel (E-mail)'' > >>>>>> Cc: ''Dave Winchell'' > >>>>>> Subject: RE: [PATCH] clocksource=tsc > >>>>>> > >>>>>> > >>>>>> OK, ignore that. It looks like you have it all fixed > >>>>>> in 18060. I tried it and it now boots. Thanks! > >>>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > >>>>>>> Sent: Tuesday, July 15, 2008 7:16 PM > >>>>>>> To: ''dan.magenheimer@oracle.com''; ''Keir Fraser''; ''Xen-Devel > >>>>>> (E-mail)'' > >>>>>>> Cc: ''Dave Winchell'' > >>>>>>> Subject: RE: [PATCH] clocksource=tsc > >>>>>>> > >>>>>>> > >>>>>>>>>> Returning to 32-bit read_counter(), and having NULL > >>>>>>>>> read_counter when > >>>>>>>>>> clocksource=tsc would be another possibility... > >>>>>>> > >>>>>>> Well I hacked on 18055 for awhile and just couldn''t get it > >>>>>>> to boot. I think local_time_calibration() (and thus > >>>>>>> init_percpu_time()) is necessary for boot, though I''m > not really > >>>>>>> sure why. Possibly the "Weirdness can happen..." comment in > >>>>>>> that routine? > >>>>>>> > >>>>>>> Anyway, this patch (on top of 18055) DOES work, returns to the > >>>>>>> 32-bit read_counter, and re-enables local_time_calibration(). > >>>>>>> I''d suggest putting off more major surgery for another day. > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Dan > >>>>>>> > >>>>>>>> -----Original Message----- > >>>>>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > >>>>>>>> Sent: Tuesday, July 15, 2008 10:04 AM > >>>>>>>> To: dan.magenheimer@oracle.com; Keir Fraser; > Xen-Devel (E-mail) > >>>>>>>> Cc: Dave Winchell > >>>>>>>> Subject: RE: [PATCH] clocksource=tsc > >>>>>>>> > >>>>>>>> > >>>>>>>> Hmmm... 18055 also fails to boot on my machine. > >>>>>>>> > >>>>>>>> Could we perhaps fall back to my original patch and do > >>>>>>>> cleanup later/separately? I also want to try implementing > >>>>>>>> an hpet64-based get_s_time() so will be working more > >>>>>>>> in this code later... but want to get clocksource=tsc > >>>>>>>> working now with minimal code impact given the freeze. > >>>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > >>>>>>>>> Sent: Tuesday, July 15, 2008 9:46 AM > >>>>>>>>> To: ''Keir Fraser''; ''Xen-Devel (E-mail)'' > >>>>>>>>> Cc: ''Dave Winchell'' > >>>>>>>>> Subject: RE: [PATCH] clocksource=tsc > >>>>>>>>> > >>>>>>>>>> Actually in this mode of operation we hardly need > a platform > >>>>>>>>>> timer *at all*. > >>>>>>>>>> The idea is that we let the TSCs free-run, because we know > >>>>>>>>>> they will behave. > >>>>>>>>>> Returning to 32-bit read_counter(), and having NULL > >>>>>>>>> read_counter when > >>>>>>>>>> clocksource=tsc would be another possibility... > >>>>>>>>> > >>>>>>>>> That''s essentially what the original > tscstable.patch did, though > >>>>>>>>> I was perhaps much uglier in the miscellaneous parts. > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Dan > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>> > >>>> > >>>> > >>>> > >> > >> > >> > >> _______________________________________________ > >> Xen-devel mailing list > >> Xen-devel@lists.xensource.com > >> http://lists.xensource.com/xen-devel > > > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 18/7/08 15:19, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:> So perhaps the compromise which I''ve (admittedly accidentally) > crafted is the right answer for now. And the right answer for > later is to update the PV time mechanisms (in a backwards > compatible way of course) to determine if an ideal timesource > is available and use it if it is.Well, maybe, although I don''t see we have any guarantee that ''platform system time'' and Xen''s get_s_time won''t diverge in absolute terms as time passes. The scale factors each use are calculated somewhat independently. Actually I think it may get it right just now, but it looks rather fragile and it stores up trouble for systems with long uptimes if you get it wrong. There''s no particular reason not to generate fresh time records every few seconds and use those both in Xen and in guests, using the existing compute-system-time algorithm. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > So perhaps the compromise which I''ve (admittedly accidentally) > > crafted is the right answer for now. And the right answer for > > later is to update the PV time mechanisms (in a backwards > > compatible way of course) to determine if an ideal timesource > > is available and use it if it is. > > Well, maybe, although I don''t see we have any guarantee that ''platform > system time'' and Xen''s get_s_time won''t diverge in absolute > terms as time > passes. The scale factors each use are calculated somewhat > independently. > Actually I think it may get it right just now, but it looks > rather fragile > and it stores up trouble for systems with long uptimes if you > get it wrong. > There''s no particular reason not to generate fresh time > records every few > seconds and use those both in Xen and in guests, using the existing > compute-system-time algorithm.It appears that, for clocksource=tsc, as long as both read_platform_stime() and get_s_time() are returning scaled-tsc there can be no divergence. The issue with generating fresh time records every few seconds is that it unnecessarily introduces jitter into an otherwise ideal timesource. Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 18/7/08 15:56, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:> It appears that, for clocksource=tsc, as long as both > read_platform_stime() and get_s_time() are returning > scaled-tsc there can be no divergence. > > The issue with generating fresh time records every > few seconds is that it unnecessarily introduces jitter > into an otherwise ideal timesource.I don''t think this is necessarily true. If we write code to generate accurate time records specifically for clocksource=tsc then we should easily get accuracy down to a couple of parts per billion. This is certainly a more pragmatic solution than extending the guest time interfaces. I am at least coming round to the fact that the changes required in Xen''s time.c are going to have to be a bit more drastic than I first hoped. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> I don''t think this is necessarily true. If we write code to generate > accurate time records specifically for clocksource=tsc then > we should easily > get accuracy down to a couple of parts per billion. This is > certainly a more > pragmatic solution than extending the guest time interfaces. > I am at least > coming round to the fact that the changes required in Xen''s > time.c are going > to have to be a bit more drastic than I first hoped. > > -- KeirI guess you are correct as long as "generate a time record" doesn''t include recomputing the scaling factor every second (which is I think what introduces the jitter... see more on this below). However, I''m not sure why you perceive the aesthetics to be so bad to put "if (ideal_clocksource)" in get_s_time() and a few other places in time.c... or for that matter in the PV guest code. Your existing algorithm is a very cool and elegant way to optimize time handling for all "legacy" time sources. The more I understand about it, the more I am impressed! But it is very difficult to understand and for most "modern" hardware it unnecessarily obfuscates time handling. So IMHO I much prefer the "if (ideal_clocksource)... else ..." approach rather than shoehorning the ideal to look like the legacy. And that applies for the PV guest code as well. On a related note:> The problem is that this is sensitive to small errors in the > scale factor! > And unfortunately we compute a new scale factor, to convert to us, as > scale_factor_ns/1000. The resulting scale factor is obviously > less accurate, > and leads to an accumulating absolute error as the TSC value > gets large.Yes I noticed that the mul_frac+scale_delta code gives different answers than muldiv64, though they have essentially the same API. I didn''t do much testing but they differed after about 5 digits and muldiv64 does a hardware divide so I suspect it is more accurate (though also probably slower). It might be worth looking at this to squeeze out more precision. Thanks for all your time and attention on this. As Xen supports more time-sensitive workloads (such as databases on enterprise systems and games on PCs), I think it will pay off! Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 18/7/08 17:51, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:> I guess you are correct as long as "generate a time record" > doesn''t include recomputing the scaling factor every > second (which is I think what introduces the jitter... see > more on this below). > > However, I''m not sure why you perceive the aesthetics to > be so bad to put "if (ideal_clocksource)" in get_s_time() > and a few other places in time.c... or for that matter in > the PV guest code.Extra special case are ugly. And then if they extend into the guests then you also have to worry about new guests on old Xen and old guests on new Xen (i.e., compatibility both ways). Which is not a good way to go when there is an easy way to hide this new implementation from guests. Indeed we would only update the TSC and system-time stamps -- the scale factor would never change. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel