Xen devel - May 2008 - [PATCH] [RFC] Building guests on monotonic Xen system time

Currently, hvm guest platform timers are built on top of the tsc.
Even though the guest believes it is utilizing a monotonic timesource
(such as pit, hpet, or pmtimer), all of these plumb down to an
rdtsc instruction.

Since on many SMP platforms tsc's in different processors are not
synchronized, VMs re-scheduled from a "fast tsc" processor to
a "slow tsc" processor may experience "time going
backwards".
This is discussed in the following thread:

http://lists.xensource.com/archives/html/xen-devel/2008-04/msg00277.html

The fix proposed by Keir is that "The logic in vpt.c should
be fixed to use Xen's concept of system time and everything,
guest TSC included, should be derived from that."

The attached patch is a first attempt to derive all
guest timers from Xen's system time... and also to ensure
that system time is non-decreasing. I don't believe the
patch is complete... and possibly not even correct.
For example, I'm concerned that Xen's system time uses
different units than a guest tsc and I don't recall if
all guest tsc accesses are trapped.

Dan

==================================Thanks... for the memory
I really could use more / My throughput's on the floor
The balloon is flat / My swap disk's fat / I've OOM's in store
Overcommitted so much
(with apologies to the late great Bob Hope)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

As I look at this more, I'm convincing myself that
it may not make sense to build the guest TSC on top
of Xen system time and only the guest virtual platform
timer(s) should be built on Xen system time (and should
be monotonically non-decreasing).

Why? Scaling and adjusting of xen-time-based-tsc will
be very difficult to coordinate with processor-based-tsc.
We need to always ensure that A < B < C for a guest
executing:

rdtsc(A) /* untrapped */
emulated_rdtsc(B)
rdtsc(C) /* untrapped */

Further, OS's use TSC as a highest-resolution time source
with knowledge that TSCs on different processors may
not be synchronized, whereas they assume that a platform
timer is one-per-system and monotonically increasing.

Keir, if you disagree and see guest-TSC-on-Xen-system-time
as an absolute must, please let me know.

Unfortunately, hvm_get/set_system_time() are #defined to
be hvm_get/set_system_tsc() so the usage is pretty muddled.
I think the attached patch has teased the two apart though.

Note this is still RFC, not a finished patch.
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of Dan
> Magenheimer
> Sent: Friday, May 16, 2008 11:31 AM
> To: Xen-Devel (E-mail)
> Subject: [Xen-devel] [PATCH] [RFC] Building guests on monotonic Xen
> system time
> 
> 
> Currently, hvm guest platform timers are built on top of the tsc.
> Even though the guest believes it is utilizing a monotonic timesource
> (such as pit, hpet, or pmtimer), all of these plumb down to an
> rdtsc instruction.
> 
> Since on many SMP platforms tsc's in different processors are not
> synchronized, VMs re-scheduled from a "fast tsc" processor to
> a "slow tsc" processor may experience "time going
backwards".
> This is discussed in the following thread:
> 
> http://lists.xensource.com/archives/html/xen-devel/2008-04/msg
00277.html

The fix proposed by Keir is that "The logic in vpt.c should
be fixed to use Xen's concept of system time and everything,
guest TSC included, should be derived from that."

The attached patch is a first attempt to derive all
guest timers from Xen's system time... and also to ensure
that system time is non-decreasing. I don't believe the
patch is complete... and possibly not even correct.
For example, I'm concerned that Xen's system time uses
different units than a guest tsc and I don't recall if
all guest tsc accesses are trapped.

Dan

==================================Thanks... for the memory
I really could use more / My throughput's on the floor
The balloon is flat / My swap disk's fat / I've OOM's in store
Overcommitted so much
(with apologies to the late great Bob Hope)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Next version of guest-virtual-platform-time-on-xen-system-time
attached.  This version seems to work but could use more
testing.
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of Dan
> Magenheimer
> Sent: Monday, May 19, 2008 12:27 PM
> To: Xen-Devel (E-mail)
> Subject: RE: [Xen-devel] [PATCH] [RFC] Building guests on 
> monotonic Xen
> system time
> 
> 
> As I look at this more, I'm convincing myself that
> it may not make sense to build the guest TSC on top
> of Xen system time and only the guest virtual platform
> timer(s) should be built on Xen system time (and should
> be monotonically non-decreasing).
> 
> Why? Scaling and adjusting of xen-time-based-tsc will
> be very difficult to coordinate with processor-based-tsc.
> We need to always ensure that A < B < C for a guest
> executing:
> 
> rdtsc(A) /* untrapped */
> emulated_rdtsc(B)
> rdtsc(C) /* untrapped */
> 
> Further, OS's use TSC as a highest-resolution time source
> with knowledge that TSCs on different processors may
> not be synchronized, whereas they assume that a platform
> timer is one-per-system and monotonically increasing.
> 
> Keir, if you disagree and see guest-TSC-on-Xen-system-time
> as an absolute must, please let me know.
> 
> Unfortunately, hvm_get/set_system_time() are #defined to
> be hvm_get/set_system_tsc() so the usage is pretty muddled.
> I think the attached patch has teased the two apart though.
> 
> Note this is still RFC, not a finished patch.
> 
> > -----Original Message-----
> > From: xen-devel-bounces@lists.xensource.com
> > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of Dan
> > Magenheimer
> > Sent: Friday, May 16, 2008 11:31 AM
> > To: Xen-Devel (E-mail)
> > Subject: [Xen-devel] [PATCH] [RFC] Building guests on monotonic Xen
> > system time
> > 
> > 
> > Currently, hvm guest platform timers are built on top of the tsc.
> > Even though the guest believes it is utilizing a monotonic 
> timesource
> > (such as pit, hpet, or pmtimer), all of these plumb down to an
> > rdtsc instruction.
> > 
> > Since on many SMP platforms tsc's in different processors are not
> > synchronized, VMs re-scheduled from a "fast tsc" processor
to
> > a "slow tsc" processor may experience "time going
backwards".
> > This is discussed in the following thread:
> > 
> > http://lists.xensource.com/archives/html/xen-devel/2008-04/msg
> 00277.html
> 
> The fix proposed by Keir is that "The logic in vpt.c should
> be fixed to use Xen's concept of system time and everything,
> guest TSC included, should be derived from that."
> 
> The attached patch is a first attempt to derive all
> guest timers from Xen's system time... and also to ensure
> that system time is non-decreasing. I don't believe the
> patch is complete... and possibly not even correct.
> For example, I'm concerned that Xen's system time uses
> different units than a guest tsc and I don't recall if
> all guest tsc accesses are trapped.
> 
> Dan
> 
> ==================================> Thanks... for the memory
> I really could use more / My throughput's on the floor
> The balloon is flat / My swap disk's fat / I've OOM's in store
> Overcommitted so much
> (with apologies to the late great Bob Hope)
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 19/5/08 19:27, "Dan Magenheimer" <dan.magenheimer@oracle.com>
wrote:
> Why? Scaling and adjusting of xen-time-based-tsc will
> be very difficult to coordinate with processor-based-tsc.
> We need to always ensure that A < B < C for a guest
> executing:
> 
> rdtsc(A) /* untrapped */
> emulated_rdtsc(B)
> rdtsc(C) /* untrapped */
>
> Further, OS''s use TSC as a highest-resolution time source
> with knowledge that TSCs on different processors may
> not be synchronized, whereas they assume that a platform
> timer is one-per-system and monotonically increasing.
> 
> Keir, if you disagree and see guest-TSC-on-Xen-system-time
> as an absolute must, please let me know.
I am inclined to say we should have a guest-TSC-on-system-time mode where
*all* RDTSC executions get trapped. This would at least be useful as a
baseline for tracking down guest time problems, and also provide a
guaranteed stable TSC timesource for those who care about that more than
pure performance.

*However* I would agree that, with TSC virtualisation as it currently is,
there actually isn''t really a way to build guest TSC on Xen system time
without periodically warping the TSC back or forth. The guest TSC runs at
the host TSC rate and that is that!

I think my original point was that at least we should not build all our
other virtual time sources on this dodgy ''guest TSC''. :-)

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

OK, I think this version is ready for prime-time.

Keir, please check-in to unstable.

[PATCH] Build hvm guest platform timers on monotonic Xen system time

This patch moves hvm platform timers from underlying physical CPU TSC
to Xen system time and ensures monotonicity.  TSC on many systems may
skew between processors, thus hvm platform timer reads were not
monotonic which led to "Time going backwards" problems.

Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>
> -----Original Message-----
> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> Sent: Tuesday, May 20, 2008 1:37 AM
> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> Subject: Re: [Xen-devel] [PATCH] [RFC] Building guests on 
> monotonic Xen
> system time
> 
> 
> On 19/5/08 19:27, "Dan Magenheimer" 
> <dan.magenheimer@oracle.com> wrote:
> 
> > Why? Scaling and adjusting of xen-time-based-tsc will
> > be very difficult to coordinate with processor-based-tsc.
> > We need to always ensure that A < B < C for a guest
> > executing:
> >
> > rdtsc(A) /* untrapped */
> > emulated_rdtsc(B)
> > rdtsc(C) /* untrapped */
> >
> > Further, OS''s use TSC as a highest-resolution time source
> > with knowledge that TSCs on different processors may
> > not be synchronized, whereas they assume that a platform
> > timer is one-per-system and monotonically increasing.
> >
> > Keir, if you disagree and see guest-TSC-on-Xen-system-time
> > as an absolute must, please let me know.
> 
> I am inclined to say we should have a 
> guest-TSC-on-system-time mode where
> *all* RDTSC executions get trapped. This would at least be useful as a
> baseline for tracking down guest time problems, and also provide a
> guaranteed stable TSC timesource for those who care about 
> that more than
> pure performance.
> 
> *However* I would agree that, with TSC virtualisation as it 
> currently is,
> there actually isn''t really a way to build guest TSC on Xen 
> system time
> without periodically warping the TSC back or forth. The guest 
> TSC runs at
> the host TSC rate and that is that!
> 
> I think my original point was that at least we should not 
> build all our
> other virtual time sources on this dodgy ''guest TSC''. :-)
> 
>  -- Keir
> 
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Attached is a modified version (really just renames). However the new
hvm_set_guest_time() interface doesn''t work for vpt.c timer modes which
allow progress of time to temporarily deviate across VCPUs. I guess you
don''t use those modes so might not notice this issue. I think hvm guest
time
handling may need to be integrated into vpt.c so that correct handling can
be applied for the various different types of timer mode. Having a blanket
per-domain stime offset and enforcing monotonicity regardless of timer mode
doesn''t fly.

 -- Keir

On 21/5/08 20:01, "Dan Magenheimer" <dan.magenheimer@oracle.com>
wrote:
> OK, I think this version is ready for prime-time.
> 
> Keir, please check-in to unstable.
> 
> [PATCH] Build hvm guest platform timers on monotonic Xen system time
> 
> This patch moves hvm platform timers from underlying physical CPU TSC
> to Xen system time and ensures monotonicity.  TSC on many systems may
> skew between processors, thus hvm platform timer reads were not
> monotonic which led to "Time going backwards" problems.
> 
> Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>
> 
>> -----Original Message-----
>> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
>> Sent: Tuesday, May 20, 2008 1:37 AM
>> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>> Subject: Re: [Xen-devel] [PATCH] [RFC] Building guests on
>> monotonic Xen
>> system time
>> 
>> 
>> On 19/5/08 19:27, "Dan Magenheimer"
>> <dan.magenheimer@oracle.com> wrote:
>> 
>>> Why? Scaling and adjusting of xen-time-based-tsc will
>>> be very difficult to coordinate with processor-based-tsc.
>>> We need to always ensure that A < B < C for a guest
>>> executing:
>>> 
>>> rdtsc(A) /* untrapped */
>>> emulated_rdtsc(B)
>>> rdtsc(C) /* untrapped */
>>> 
>>> Further, OS''s use TSC as a highest-resolution time source
>>> with knowledge that TSCs on different processors may
>>> not be synchronized, whereas they assume that a platform
>>> timer is one-per-system and monotonically increasing.
>>> 
>>> Keir, if you disagree and see guest-TSC-on-Xen-system-time
>>> as an absolute must, please let me know.
>> 
>> I am inclined to say we should have a
>> guest-TSC-on-system-time mode where
>> *all* RDTSC executions get trapped. This would at least be useful as a
>> baseline for tracking down guest time problems, and also provide a
>> guaranteed stable TSC timesource for those who care about
>> that more than
>> pure performance.
>> 
>> *However* I would agree that, with TSC virtualisation as it
>> currently is,
>> there actually isn''t really a way to build guest TSC on Xen
>> system time
>> without periodically warping the TSC back or forth. The guest
>> TSC runs at
>> the host TSC rate and that is that!
>> 
>> I think my original point was that at least we should not
>> build all our
>> other virtual time sources on this dodgy ''guest TSC''.
:-)
>> 
>>  -- Keir
>> 
>> 
>> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks for the review and the modifications!

It does allow time to temporarily deviate across VCPUs, but
doesn''t allow them to "apparently" deviate; that is a VCPU
can never observe or set a time which is older than the time
that another VCPU previously observed or set.

Unless you can guarantee this, there''s no way to avoid "Time
going backwards" errors, is there?

Admittedly, I don''t fully understand the inner workings of
all the timer modes, but if this doesn''t fly for one or more
of the modes, doesn''t this mean it is impossible for those
timer modes to guarantee monotonicity -- at least without
also requiring gang scheduling?

Thanks,
Dan

P.S. I had left the naming as hvm_get_guest_time_mono()
because I wasn''t sure there would never be a version
needed that was non-mono, so thought it would be better
to be explicit in the name.  (I was especially sensitized
to this after having just completed teasing apart the
hvm_get_guest_tsc() from hvm_get_guest_time() ;-)
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of Keir Fraser
> Sent: Thursday, May 22, 2008 2:46 AM
> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> Subject: Re: [Xen-devel] [PATCH] [RFC] Building guests on 
> monotonic Xen
> system time
> 
> 
> Attached is a modified version (really just renames). However the new
> hvm_set_guest_time() interface doesn''t work for vpt.c timer 
> modes which
> allow progress of time to temporarily deviate across VCPUs. I 
> guess you
> don''t use those modes so might not notice this issue. I think 
> hvm guest time
> handling may need to be integrated into vpt.c so that correct 
> handling can
> be applied for the various different types of timer mode. 
> Having a blanket
> per-domain stime offset and enforcing monotonicity regardless 
> of timer mode
> doesn''t fly.
> 
>  -- Keir
> 
> On 21/5/08 20:01, "Dan Magenheimer" 
> <dan.magenheimer@oracle.com> wrote:
> 
> > OK, I think this version is ready for prime-time.
> >
> > Keir, please check-in to unstable.
> >
> > [PATCH] Build hvm guest platform timers on monotonic Xen system time
> >
> > This patch moves hvm platform timers from underlying 
> physical CPU TSC
> > to Xen system time and ensures monotonicity.  TSC on many 
> systems may
> > skew between processors, thus hvm platform timer reads were not
> > monotonic which led to "Time going backwards" problems.
> >
> > Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>
> >
> >> -----Original Message-----
> >> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> >> Sent: Tuesday, May 20, 2008 1:37 AM
> >> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> >> Subject: Re: [Xen-devel] [PATCH] [RFC] Building guests on
> >> monotonic Xen
> >> system time
> >>
> >>
> >> On 19/5/08 19:27, "Dan Magenheimer"
> >> <dan.magenheimer@oracle.com> wrote:
> >>
> >>> Why? Scaling and adjusting of xen-time-based-tsc will
> >>> be very difficult to coordinate with processor-based-tsc.
> >>> We need to always ensure that A < B < C for a guest
> >>> executing:
> >>>
> >>> rdtsc(A) /* untrapped */
> >>> emulated_rdtsc(B)
> >>> rdtsc(C) /* untrapped */
> >>>
> >>> Further, OS''s use TSC as a highest-resolution time
source
> >>> with knowledge that TSCs on different processors may
> >>> not be synchronized, whereas they assume that a platform
> >>> timer is one-per-system and monotonically increasing.
> >>>
> >>> Keir, if you disagree and see guest-TSC-on-Xen-system-time
> >>> as an absolute must, please let me know.
> >>
> >> I am inclined to say we should have a
> >> guest-TSC-on-system-time mode where
> >> *all* RDTSC executions get trapped. This would at least be 
> useful as a
> >> baseline for tracking down guest time problems, and also provide a
> >> guaranteed stable TSC timesource for those who care about
> >> that more than
> >> pure performance.
> >>
> >> *However* I would agree that, with TSC virtualisation as it
> >> currently is,
> >> there actually isn''t really a way to build guest TSC on
Xen
> >> system time
> >> without periodically warping the TSC back or forth. The guest
> >> TSC runs at
> >> the host TSC rate and that is that!
> >>
> >> I think my original point was that at least we should not
> >> build all our
> >> other virtual time sources on this dodgy ''guest
TSC''. :-)
> >>
> >>  -- Keir
> >>
> >>
> >>
> 
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Well, really it''s baked into the design of some of the timer modes that
time
can differ across VCPUs: it''s their reason for existence. We could say
we
don''t support them any more -- I''m not sure anyone enables
them for
productised releases of Xen -- but we should at least do that explicitly
rather than subtly breaking them.

What I''ll do is append doing some more work to your patch to my work
queue.
I would say I''m moderately confident that your patch actually will work
fine
for the timer modes that you guys are using in your packaging of Xen, but I
think some more architectural cleanup in this area would be good for
xen-unstable. Whether that is remove some of the older timer modes now, or
clean up hvm_[sg]et_guest_time() to work nicely with those modes, is open to
debate.

 -- Keir

On 22/5/08 17:05, "Dan Magenheimer" <dan.magenheimer@oracle.com>
wrote:
> Thanks for the review and the modifications!
> 
> It does allow time to temporarily deviate across VCPUs, but
> doesn''t allow them to "apparently" deviate; that is a
VCPU
> can never observe or set a time which is older than the time
> that another VCPU previously observed or set.
> 
> Unless you can guarantee this, there''s no way to avoid "Time
> going backwards" errors, is there?
> 
> Admittedly, I don''t fully understand the inner workings of
> all the timer modes, but if this doesn''t fly for one or more
> of the modes, doesn''t this mean it is impossible for those
> timer modes to guarantee monotonicity -- at least without
> also requiring gang scheduling?
> 
> Thanks,
> Dan
> 
> P.S. I had left the naming as hvm_get_guest_time_mono()
> because I wasn''t sure there would never be a version
> needed that was non-mono, so thought it would be better
> to be explicit in the name.  (I was especially sensitized
> to this after having just completed teasing apart the
> hvm_get_guest_tsc() from hvm_get_guest_time() ;-)
> 
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com
>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of Keir Fraser
>> Sent: Thursday, May 22, 2008 2:46 AM
>> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>> Subject: Re: [Xen-devel] [PATCH] [RFC] Building guests on
>> monotonic Xen
>> system time
>> 
>> 
>> Attached is a modified version (really just renames). However the new
>> hvm_set_guest_time() interface doesn''t work for vpt.c timer
>> modes which
>> allow progress of time to temporarily deviate across VCPUs. I
>> guess you
>> don''t use those modes so might not notice this issue. I think
>> hvm guest time
>> handling may need to be integrated into vpt.c so that correct
>> handling can
>> be applied for the various different types of timer mode.
>> Having a blanket
>> per-domain stime offset and enforcing monotonicity regardless
>> of timer mode
>> doesn''t fly.
>> 
>>  -- Keir
>> 
>> On 21/5/08 20:01, "Dan Magenheimer"
>> <dan.magenheimer@oracle.com> wrote:
>> 
>>> OK, I think this version is ready for prime-time.
>>> 
>>> Keir, please check-in to unstable.
>>> 
>>> [PATCH] Build hvm guest platform timers on monotonic Xen system
time
>>> 
>>> This patch moves hvm platform timers from underlying
>> physical CPU TSC
>>> to Xen system time and ensures monotonicity.  TSC on many
>> systems may
>>> skew between processors, thus hvm platform timer reads were not
>>> monotonic which led to "Time going backwards" problems.
>>> 
>>> Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>
>>> 
>>>> -----Original Message-----
>>>> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
>>>> Sent: Tuesday, May 20, 2008 1:37 AM
>>>> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>>>> Subject: Re: [Xen-devel] [PATCH] [RFC] Building guests on
>>>> monotonic Xen
>>>> system time
>>>> 
>>>> 
>>>> On 19/5/08 19:27, "Dan Magenheimer"
>>>> <dan.magenheimer@oracle.com> wrote:
>>>> 
>>>>> Why? Scaling and adjusting of xen-time-based-tsc will
>>>>> be very difficult to coordinate with processor-based-tsc.
>>>>> We need to always ensure that A < B < C for a guest
>>>>> executing:
>>>>> 
>>>>> rdtsc(A) /* untrapped */
>>>>> emulated_rdtsc(B)
>>>>> rdtsc(C) /* untrapped */
>>>>> 
>>>>> Further, OS''s use TSC as a highest-resolution time
source
>>>>> with knowledge that TSCs on different processors may
>>>>> not be synchronized, whereas they assume that a platform
>>>>> timer is one-per-system and monotonically increasing.
>>>>> 
>>>>> Keir, if you disagree and see guest-TSC-on-Xen-system-time
>>>>> as an absolute must, please let me know.
>>>> 
>>>> I am inclined to say we should have a
>>>> guest-TSC-on-system-time mode where
>>>> *all* RDTSC executions get trapped. This would at least be
>> useful as a
>>>> baseline for tracking down guest time problems, and also
provide a
>>>> guaranteed stable TSC timesource for those who care about
>>>> that more than
>>>> pure performance.
>>>> 
>>>> *However* I would agree that, with TSC virtualisation as it
>>>> currently is,
>>>> there actually isn''t really a way to build guest TSC
on Xen
>>>> system time
>>>> without periodically warping the TSC back or forth. The guest
>>>> TSC runs at
>>>> the host TSC rate and that is that!
>>>> 
>>>> I think my original point was that at least we should not
>>>> build all our
>>>> other virtual time sources on this dodgy ''guest
TSC''. :-)
>>>> 
>>>>  -- Keir
>>>> 
>>>> 
>>>> 
>> 
>> 
>> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Whether that is remove some of the older timer modes now,
Looking at the various timer modes, it appears that:

0==delay_for_missed_ticks is required for non-resilient Linuxes
   (e.g. 32-bit RHEL4-based systems)
1==no_delay_for_missed_ticks is required for Windows(??)
2==no_missed_ticks_pending is required for resilient Linuxes
   (e.g. 64-bit RHEL5-based systems)
3==one_missed_tick_pending DON''T KNOW WHAT IS USED FOR

So timer_mode=3=one_missed_tick_pending might possibly
be retired, but the others couldn''t.

Here''s another option:  I think delay_for_missed_ticks is
the only mode where time must be set backwards.  All of the
other modes track wallclock.  It looks possible that we
could enforce monotonicity for the other three modes and
not enforce it for delay_for_missed_ticks (which is,
unfortunately, the default).

If this is acceptable to you (and you don''t disagree with
my "only mode...set backwards" conjecture), I''ll try to
code a patch for this.

Thanks,
Dan

P.S. I think the answer is no, but is there any way to specify
gang-scheduling for a domain?  If so, we could say if your
OS is non-resilient to preemption and your application requires
time monotonicity, you must specify gang scheduling.

> -----Original Message-----
> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> Sent: Thursday, May 22, 2008 10:12 AM
> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> Cc: Dave Winchell
> Subject: Re: [Xen-devel] [PATCH] [RFC] Building guests on 
> monotonic Xen
> system time
> 
> 
> Well, really it''s baked into the design of some of the timer 
> modes that time
> can differ across VCPUs: it''s their reason for existence. We 
> could say we
> don''t support them any more -- I''m not sure anyone
enables them for
> productised releases of Xen -- but we should at least do that 
> explicitly
> rather than subtly breaking them.
> 
> What I''ll do is append doing some more work to your patch to 
> my work queue.
> I would say I''m moderately confident that your patch actually 
> will work fine
> for the timer modes that you guys are using in your packaging 
> of Xen, but I
> think some more architectural cleanup in this area would be good for
> xen-unstable. Whether that is remove some of the older timer 
> modes now, or
> clean up hvm_[sg]et_guest_time() to work nicely with those 
> modes, is open to
> debate.
> 
>  -- Keir
> 
> On 22/5/08 17:05, "Dan Magenheimer" 
> <dan.magenheimer@oracle.com> wrote:
> 
> > Thanks for the review and the modifications!
> >
> > It does allow time to temporarily deviate across VCPUs, but
> > doesn''t allow them to "apparently" deviate; that is
a VCPU
> > can never observe or set a time which is older than the time
> > that another VCPU previously observed or set.
> >
> > Unless you can guarantee this, there''s no way to avoid
"Time
> > going backwards" errors, is there?
> >
> > Admittedly, I don''t fully understand the inner workings of
> > all the timer modes, but if this doesn''t fly for one or more
> > of the modes, doesn''t this mean it is impossible for those
> > timer modes to guarantee monotonicity -- at least without
> > also requiring gang scheduling?
> >
> > Thanks,
> > Dan
> >
> > P.S. I had left the naming as hvm_get_guest_time_mono()
> > because I wasn''t sure there would never be a version
> > needed that was non-mono, so thought it would be better
> > to be explicit in the name.  (I was especially sensitized
> > to this after having just completed teasing apart the
> > hvm_get_guest_tsc() from hvm_get_guest_time() ;-)
> >
> >> -----Original Message-----
> >> From: xen-devel-bounces@lists.xensource.com
> >> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of 
> Keir Fraser
> >> Sent: Thursday, May 22, 2008 2:46 AM
> >> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> >> Subject: Re: [Xen-devel] [PATCH] [RFC] Building guests on
> >> monotonic Xen
> >> system time
> >>
> >>
> >> Attached is a modified version (really just renames). 
> However the new
> >> hvm_set_guest_time() interface doesn''t work for vpt.c
timer
> >> modes which
> >> allow progress of time to temporarily deviate across VCPUs. I
> >> guess you
> >> don''t use those modes so might not notice this issue. I
think
> >> hvm guest time
> >> handling may need to be integrated into vpt.c so that correct
> >> handling can
> >> be applied for the various different types of timer mode.
> >> Having a blanket
> >> per-domain stime offset and enforcing monotonicity regardless
> >> of timer mode
> >> doesn''t fly.
> >>
> >>  -- Keir
> >>
> >> On 21/5/08 20:01, "Dan Magenheimer"
> >> <dan.magenheimer@oracle.com> wrote:
> >>
> >>> OK, I think this version is ready for prime-time.
> >>>
> >>> Keir, please check-in to unstable.
> >>>
> >>> [PATCH] Build hvm guest platform timers on monotonic Xen 
> system time
> >>>
> >>> This patch moves hvm platform timers from underlying
> >> physical CPU TSC
> >>> to Xen system time and ensures monotonicity.  TSC on many
> >> systems may
> >>> skew between processors, thus hvm platform timer reads were
not
> >>> monotonic which led to "Time going backwards"
problems.
> >>>
> >>> Signed-off-by: Dan Magenheimer
<dan.magenheimer@oracle.com>
> >>>
> >>>> -----Original Message-----
> >>>> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> >>>> Sent: Tuesday, May 20, 2008 1:37 AM
> >>>> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> >>>> Subject: Re: [Xen-devel] [PATCH] [RFC] Building guests on
> >>>> monotonic Xen
> >>>> system time
> >>>>
> >>>>
> >>>> On 19/5/08 19:27, "Dan Magenheimer"
> >>>> <dan.magenheimer@oracle.com> wrote:
> >>>>
> >>>>> Why? Scaling and adjusting of xen-time-based-tsc will
> >>>>> be very difficult to coordinate with
processor-based-tsc.
> >>>>> We need to always ensure that A < B < C for a
guest
> >>>>> executing:
> >>>>>
> >>>>> rdtsc(A) /* untrapped */
> >>>>> emulated_rdtsc(B)
> >>>>> rdtsc(C) /* untrapped */
> >>>>>
> >>>>> Further, OS''s use TSC as a highest-resolution
time source
> >>>>> with knowledge that TSCs on different processors may
> >>>>> not be synchronized, whereas they assume that a
platform
> >>>>> timer is one-per-system and monotonically increasing.
> >>>>>
> >>>>> Keir, if you disagree and see
guest-TSC-on-Xen-system-time
> >>>>> as an absolute must, please let me know.
> >>>>
> >>>> I am inclined to say we should have a
> >>>> guest-TSC-on-system-time mode where
> >>>> *all* RDTSC executions get trapped. This would at least be
> >> useful as a
> >>>> baseline for tracking down guest time problems, and also 
> provide a
> >>>> guaranteed stable TSC timesource for those who care about
> >>>> that more than
> >>>> pure performance.
> >>>>
> >>>> *However* I would agree that, with TSC virtualisation as
it
> >>>> currently is,
> >>>> there actually isn''t really a way to build guest
TSC on Xen
> >>>> system time
> >>>> without periodically warping the TSC back or forth. The
guest
> >>>> TSC runs at
> >>>> the host TSC rate and that is that!
> >>>>
> >>>> I think my original point was that at least we should not
> >>>> build all our
> >>>> other virtual time sources on this dodgy ''guest
TSC''. :-)
> >>>>
> >>>>  -- Keir
> >>>>
> >>>>
> >>>>
> >>
> >>
> >>
> >
> 
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 23/5/08 23:44, "Dan Magenheimer" <dan.magenheimer@oracle.com>
wrote:
> Here''s another option:  I think delay_for_missed_ticks is
> the only mode where time must be set backwards.  All of the
> other modes track wallclock.  It looks possible that we
> could enforce monotonicity for the other three modes and
> not enforce it for delay_for_missed_ticks (which is,
> unfortunately, the default).
> 
> If this is acceptable to you (and you don''t disagree with
> my "only mode...set backwards" conjecture), I''ll try to
> code a patch for this.
I agree with your analysis. I couldn''t remember if any modes other than
0
would require non-monotonic time across VCPUs, but it does appear that this
property is exclusive to mode 0.
> P.S. I think the answer is no, but is there any way to specify
> gang-scheduling for a domain?  If so, we could say if your
> OS is non-resilient to preemption and your application requires
> time monotonicity, you must specify gang scheduling.
There isn''t currently a way to request gang scheduling. I think that is
a
second step anyway. It sounds like you could simply turn off your
monotonicity checks for timer mode 0 and I''ll be happy.

I have a pretty clear vision of what I want now, so I will make the
necessary adjustments to the patch. Then we can go from there if you think
further modifications are required.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> I have a pretty clear vision of what I want now, so I will make the
> necessary adjustments to the patch. Then we can go from there 
I noticed you fixed a related bug (cset 17760), but haven''t
seen anything that changes timer_mode==0 back to non-monotonic.
Is that still on your list, or do you want me to submit a
patch for that?

Related, I''m wondering if we should change the defaults for
timer_mode and hpet now.  I''ll start a new thread on that
to avoid topic drift.

Dan  
> -----Original Message-----
> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> Sent: Saturday, May 24, 2008 1:34 AM
> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> Cc: Dave Winchell
> Subject: Re: [Xen-devel] [PATCH] [RFC] Building guests on 
> monotonic Xen
> system time
> 
> 
> On 23/5/08 23:44, "Dan Magenheimer" 
> <dan.magenheimer@oracle.com> wrote:
> 
> > Here''s another option:  I think delay_for_missed_ticks is
> > the only mode where time must be set backwards.  All of the
> > other modes track wallclock.  It looks possible that we
> > could enforce monotonicity for the other three modes and
> > not enforce it for delay_for_missed_ticks (which is,
> > unfortunately, the default).
> >
> > If this is acceptable to you (and you don''t disagree with
> > my "only mode...set backwards" conjecture), I''ll
try to
> > code a patch for this.
> 
> I agree with your analysis. I couldn''t remember if any modes 
> other than 0
> would require non-monotonic time across VCPUs, but it does 
> appear that this
> property is exclusive to mode 0.
> 
> > P.S. I think the answer is no, but is there any way to specify
> > gang-scheduling for a domain?  If so, we could say if your
> > OS is non-resilient to preemption and your application requires
> > time monotonicity, you must specify gang scheduling.
> 
> There isn''t currently a way to request gang scheduling. I 
> think that is a
> second step anyway. It sounds like you could simply turn off your
> monotonicity checks for timer mode 0 and I''ll be happy.
> 
> I have a pretty clear vision of what I want now, so I will make the
> necessary adjustments to the patch. Then we can go from there 
> if you think
> further modifications are required.
> 
>  -- Keir
> 
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 3/6/08 20:47, "Dan Magenheimer" <dan.magenheimer@oracle.com>
wrote:
>> I have a pretty clear vision of what I want now, so I will make the
>> necessary adjustments to the patch. Then we can go from there
> 
> I noticed you fixed a related bug (cset 17760), but haven''t
> seen anything that changes timer_mode==0 back to non-monotonic.
> Is that still on your list, or do you want me to submit a
> patch for that?
I think it is correctly handled by my patch. The per-vcpu time offset is
only adjusted when timer_mode==0 and that is added into the result returned
by hvm_get_guest_time() outside of the domain lock or monotinicity checks.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel