Xen devel - Jul 2008 - [PATCH?] monotonically increasing Xen system time

(This is probably post-3.3, but feedback would be appreciated
as I would hope it could go into 3.3.1 if it doesn't make 3.3.)

I've finally surrendered to the fact that intra-CPU stime skew
can't be reduced down to the point where it can be ignored,
at least on non-tsc-invariant boxes.  It always seems to
max out at least at several microseconds, and in some cases
in tens of microseconds.  This is probably a result of
crystal oscillation drift, and perhaps the "beating" of
the platform timer crystal vs the tsc crystal.

So the attached patch adds a get_s_time_mono() call that
always returns a monotonically INcreasing (not just
non-decreasing) stime.  A stime_minstep is computed that
guarantees that mono_stime can't increase faster than
stime, even if all processors are pounding on stime
in a loop.  The result is the resolution for mono_stime.
(On my dual-core box, it's 24ns... your mileage may vary.)

I want to use this in hvm_get_guest_time() (and thus for
softtsc) but it may also be appropriate for at least some
of the many uses of NOW() in Xen.  If so, it might make
sense that this should be the default get_s_time() and the
current get_s_time() should be renamed get_local_s_time().
In any case, there are most likely other uses for it
in Xen so I didn't want to build it just into
hvm_get_guest_time().

(Note that init_xen_time() was moved down in __start_xen()
because num_online_cpus() gave the wrong answer at its
current position.)

Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>

==================================Thanks... for the memory
I really could use more / My throughput's on the floor
The balloon is flat / My swap disk's fat / I've OOM's in store
Overcommitted so much
(with apologies to the late great Bob Hope)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Xen itself doesn''t in most cases care about a bit of skew across CPUs.
So I
think the cuurent get_s_time() is fine, and then build monotonicity on top
where we want it (notably hvm_get_guest_time() as has already been done).

 -- Keir

On 28/7/08 18:04, "Dan Magenheimer" <dan.magenheimer@oracle.com>
wrote:
> (This is probably post-3.3, but feedback would be appreciated
> as I would hope it could go into 3.3.1 if it doesn''t make 3.3.)
> 
> I''ve finally surrendered to the fact that intra-CPU stime skew
> can''t be reduced down to the point where it can be ignored,
> at least on non-tsc-invariant boxes.  It always seems to
> max out at least at several microseconds, and in some cases
> in tens of microseconds.  This is probably a result of
> crystal oscillation drift, and perhaps the "beating" of
> the platform timer crystal vs the tsc crystal.
> 
> So the attached patch adds a get_s_time_mono() call that
> always returns a monotonically INcreasing (not just
> non-decreasing) stime.  A stime_minstep is computed that
> guarantees that mono_stime can''t increase faster than
> stime, even if all processors are pounding on stime
> in a loop.  The result is the resolution for mono_stime.
> (On my dual-core box, it''s 24ns... your mileage may vary.)
> 
> I want to use this in hvm_get_guest_time() (and thus for
> softtsc) but it may also be appropriate for at least some
> of the many uses of NOW() in Xen.  If so, it might make
> sense that this should be the default get_s_time() and the
> current get_s_time() should be renamed get_local_s_time().
> In any case, there are most likely other uses for it
> in Xen so I didn''t want to build it just into
> hvm_get_guest_time().
> 
> (Note that init_xen_time() was moved down in __start_xen()
> because num_online_cpus() gave the wrong answer at its
> current position.)
> 
> Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>
> 
> ==================================> Thanks... for the memory
> I really could use more / My throughput''s on the floor
> The balloon is flat / My swap disk''s fat / I''ve
OOM''s in store
> Overcommitted so much
> (with apologies to the late great Bob Hope)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Since my primary purpose is to guarantee monotonically
INcreasing time to hvm_get_guest_time(), I am happy
to resubmit the patch with only that fix provided
(as the current solution only provides monotonically
NON-DEcreasing time).

However, having spent far too much time looking at
inter-CPU skew in the last few weeks, I have to argue.
You say Xen doesn''t care in *most* cases.  I suspect
that it does care in *some* cases and that there are
one or more subtle race conditions that would only
get tripped if skew is "very bad".  Since it gets bad
randomly and with very low frequency, and since
on some machines "bad" may be "very bad", and since
such races will be nearly impossible to isolate and
debug... and since the increasing adoption of
high-precision timers is setting higher expectations
of resolution, I thought it might be wise to
provide a solution that avoids the possibility
of a skew-related bug.  And then examine each
use of NOW() to see whether it is really using it
as a timestamp or as a "period from now on the
current processor".

Thanks,
Dan
> -----Original Message-----
> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> Sent: Monday, July 28, 2008 1:10 PM
> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> Subject: Re: [PATCH?] monotonically increasing Xen system time
> 
> 
> Xen itself doesn''t in most cases care about a bit of skew 
> across CPUs. So I
> think the cuurent get_s_time() is fine, and then build 
> monotonicity on top
> where we want it (notably hvm_get_guest_time() as has already 
> been done).
> 
>  -- Keir
> 
> On 28/7/08 18:04, "Dan Magenheimer" 
> <dan.magenheimer@oracle.com> wrote:
> 
> > (This is probably post-3.3, but feedback would be appreciated
> > as I would hope it could go into 3.3.1 if it doesn''t make
3.3.)
> >
> > I''ve finally surrendered to the fact that intra-CPU stime
skew
> > can''t be reduced down to the point where it can be ignored,
> > at least on non-tsc-invariant boxes.  It always seems to
> > max out at least at several microseconds, and in some cases
> > in tens of microseconds.  This is probably a result of
> > crystal oscillation drift, and perhaps the "beating" of
> > the platform timer crystal vs the tsc crystal.
> >
> > So the attached patch adds a get_s_time_mono() call that
> > always returns a monotonically INcreasing (not just
> > non-decreasing) stime.  A stime_minstep is computed that
> > guarantees that mono_stime can''t increase faster than
> > stime, even if all processors are pounding on stime
> > in a loop.  The result is the resolution for mono_stime.
> > (On my dual-core box, it''s 24ns... your mileage may vary.)
> >
> > I want to use this in hvm_get_guest_time() (and thus for
> > softtsc) but it may also be appropriate for at least some
> > of the many uses of NOW() in Xen.  If so, it might make
> > sense that this should be the default get_s_time() and the
> > current get_s_time() should be renamed get_local_s_time().
> > In any case, there are most likely other uses for it
> > in Xen so I didn''t want to build it just into
> > hvm_get_guest_time().
> >
> > (Note that init_xen_time() was moved down in __start_xen()
> > because num_online_cpus() gave the wrong answer at its
> > current position.)
> >
> > Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>
> >
> > ==================================> > Thanks... for the memory
> > I really could use more / My throughput''s on the floor
> > The balloon is flat / My swap disk''s fat / I''ve
OOM''s in store
> > Overcommitted so much
> > (with apologies to the late great Bob Hope)
> 
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Xen itself doesn''t in most cases care about a bit of skew 
Well, FYI, I did a survey of all the uses of NOW() in
Xen and couldn''t find ANY that would be sensitive to
even tens of microseconds of skew across CPUs.

So, one point for Keir... as usual ;-)
> -----Original Message-----
> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> Sent: Monday, July 28, 2008 1:10 PM
> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> Subject: Re: [PATCH?] monotonically increasing Xen system time
> 
> 
> Xen itself doesn''t in most cases care about a bit of skew 
> across CPUs. So I
> think the cuurent get_s_time() is fine, and then build 
> monotonicity on top
> where we want it (notably hvm_get_guest_time() as has already 
> been done).
> 
>  -- Keir
> 
> On 28/7/08 18:04, "Dan Magenheimer" 
> <dan.magenheimer@oracle.com> wrote:
> 
> > (This is probably post-3.3, but feedback would be appreciated
> > as I would hope it could go into 3.3.1 if it doesn''t make
3.3.)
> >
> > I''ve finally surrendered to the fact that intra-CPU stime
skew
> > can''t be reduced down to the point where it can be ignored,
> > at least on non-tsc-invariant boxes.  It always seems to
> > max out at least at several microseconds, and in some cases
> > in tens of microseconds.  This is probably a result of
> > crystal oscillation drift, and perhaps the "beating" of
> > the platform timer crystal vs the tsc crystal.
> >
> > So the attached patch adds a get_s_time_mono() call that
> > always returns a monotonically INcreasing (not just
> > non-decreasing) stime.  A stime_minstep is computed that
> > guarantees that mono_stime can''t increase faster than
> > stime, even if all processors are pounding on stime
> > in a loop.  The result is the resolution for mono_stime.
> > (On my dual-core box, it''s 24ns... your mileage may vary.)
> >
> > I want to use this in hvm_get_guest_time() (and thus for
> > softtsc) but it may also be appropriate for at least some
> > of the many uses of NOW() in Xen.  If so, it might make
> > sense that this should be the default get_s_time() and the
> > current get_s_time() should be renamed get_local_s_time().
> > In any case, there are most likely other uses for it
> > in Xen so I didn''t want to build it just into
> > hvm_get_guest_time().
> >
> > (Note that init_xen_time() was moved down in __start_xen()
> > because num_online_cpus() gave the wrong answer at its
> > current position.)
> >
> > Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>
> >
> > ==================================> > Thanks... for the memory
> > I really could use more / My throughput''s on the floor
> > The balloon is flat / My swap disk''s fat / I''ve
OOM''s in store
> > Overcommitted so much
> > (with apologies to the late great Bob Hope)
> 
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel