The other discussion of RADclock reminded me: IIRC the pvclock algorithm is still incompatible with vsyscall/vdso (fast system calls) and there was no obvious and upstreamable solution to resolve this. This means that any userland call to the various gettimeofday routines will always do a true system call on both (a) a PV domain or (b) any PV on HVM domain with Stefanos'' pvclock patch. Since true syscalls are very expensive on a 64-bit PV domain, I''m wondering if pvclock is still the right default choice for upstream (at least for 64-bit). _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Oct-16 00:28 UTC
[Xen-devel] Re: pvclock (PV and HVM) and vsyscall
On 10/15/2010 08:48 AM, Dan Magenheimer wrote:> The other discussion of RADclock reminded me: > > IIRC the pvclock algorithm is still incompatible with > vsyscall/vdso (fast system calls) and there was no obvious > and upstreamable solution to resolve this. > > This means that any userland call to the various gettimeofday > routines will always do a true system call on both (a) a PV > domain or (b) any PV on HVM domain with Stefanos'' pvclock patch. > > Since true syscalls are very expensive on a 64-bit > PV domain, I''m wondering if pvclock is still the right > default choice for upstream (at least for 64-bit).What other options are there? If the tsc is globally stable, then using pvclock in userspace will work fine; if it isn''t, you''ll need to do the syscall anyway. There''s no basic problem with the vsyscall pvclock patch so long as we can know under what circumstances it is safe to enable. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> From: Jeremy Fitzhardinge [mailto:jeremy@goop.org] > Sent: Friday, October 15, 2010 6:28 PM > To: Dan Magenheimer > Cc: Tim Deegan; Stefano Stabellini; xen-devel@lists.xensource.com > Subject: Re: pvclock (PV and HVM) and vsyscall > > On 10/15/2010 08:48 AM, Dan Magenheimer wrote: > > The other discussion of RADclock reminded me: > > > > IIRC the pvclock algorithm is still incompatible with > > vsyscall/vdso (fast system calls) and there was no obvious > > and upstreamable solution to resolve this. > > > > This means that any userland call to the various gettimeofday > > routines will always do a true system call on both (a) a PV > > domain or (b) any PV on HVM domain with Stefanos'' pvclock patch. > > > > Since true syscalls are very expensive on a 64-bit > > PV domain, I''m wondering if pvclock is still the right > > default choice for upstream (at least for 64-bit). > > What other options are there? If the tsc is globally stable, then > using > pvclock in userspace will work fine; if it isn''t, you''ll need to do the > syscall anyway. > > There''s no basic problem with the vsyscall pvclock patch so long as we > can know under what circumstances it is safe to enable.I think (but am not positive) that the circumstances under which vsyscall pvclock can be enabled are exactly the same as those for which tsc is globally stable. And when tsc is globally stable, upstream guest kernels can use tsc instead of pvclock. And when tsc is NOT globally stable, as of Xen 4.0, rdtsc emulation is faster than non-vsyscall pvclock, at least in many/most environments. To quote the RADclock paper: "[the Xen Clocksource counter] works well as intended, however note that it is a complex solution created to solve a problem which will soon disappear as reliable TSC counters again become ubiquitous." (And, yes, we both know that there will always be exceptions to "reliable" TSC, but that is in part why rdtsc emulation is the default, and Xen and RADclock and pvclock will all have the same problems as a native kernel when these exceptions occur.) I''m afraid my available time is too limited right now to fight this battle, but I think we should be very cautious about building even more infrastructure on top of pvclock. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Oct-18 16:34 UTC
[Xen-devel] Re: pvclock (PV and HVM) and vsyscall
On 10/17/2010 06:29 PM, Dan Magenheimer wrote:>> From: Jeremy Fitzhardinge [mailto:jeremy@goop.org] >> Sent: Friday, October 15, 2010 6:28 PM >> To: Dan Magenheimer >> Cc: Tim Deegan; Stefano Stabellini; xen-devel@lists.xensource.com >> Subject: Re: pvclock (PV and HVM) and vsyscall >> >> On 10/15/2010 08:48 AM, Dan Magenheimer wrote: >>> The other discussion of RADclock reminded me: >>> >>> IIRC the pvclock algorithm is still incompatible with >>> vsyscall/vdso (fast system calls) and there was no obvious >>> and upstreamable solution to resolve this. >>> >>> This means that any userland call to the various gettimeofday >>> routines will always do a true system call on both (a) a PV >>> domain or (b) any PV on HVM domain with Stefanos'' pvclock patch. >>> >>> Since true syscalls are very expensive on a 64-bit >>> PV domain, I''m wondering if pvclock is still the right >>> default choice for upstream (at least for 64-bit). >> What other options are there? If the tsc is globally stable, then >> using >> pvclock in userspace will work fine; if it isn''t, you''ll need to do the >> syscall anyway. >> >> There''s no basic problem with the vsyscall pvclock patch so long as we >> can know under what circumstances it is safe to enable. > I think (but am not positive) that the circumstances under which > vsyscall pvclock can be enabled are exactly the same as those > for which tsc is globally stable. And when tsc is globally stable, > upstream guest kernels can use tsc instead of pvclock.Not wanting to go around in circles, but usermode can''t reliably know whether the tsc is stable enough to be usable directly. Its most reliable course is to always use the proper APIs and rely on the kernel to implement the most efficient mechanism for those APIs. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> >> On 10/15/2010 08:48 AM, Dan Magenheimer wrote: > >>> The other discussion of RADclock reminded me: > >>> > >>> IIRC the pvclock algorithm is still incompatible with > >>> vsyscall/vdso (fast system calls) and there was no obvious > >>> and upstreamable solution to resolve this. > >>> > >>> This means that any userland call to the various gettimeofday > >>> routines will always do a true system call on both (a) a PV > >>> domain or (b) any PV on HVM domain with Stefanos'' pvclock patch. > >>> > >>> Since true syscalls are very expensive on a 64-bit > >>> PV domain, I''m wondering if pvclock is still the right > >>> default choice for upstream (at least for 64-bit). > >> What other options are there? If the tsc is globally stable, then > >> using > >> pvclock in userspace will work fine; if it isn''t, you''ll need to do > the > >> syscall anyway. > >> > >> There''s no basic problem with the vsyscall pvclock patch so long as > we > >> can know under what circumstances it is safe to enable. > > I think (but am not positive) that the circumstances under which > > vsyscall pvclock can be enabled are exactly the same as those > > for which tsc is globally stable. And when tsc is globally stable, > > upstream guest kernels can use tsc instead of pvclock. > > Not wanting to go around in circles, but usermode can''t reliably know > whether the tsc is stable enough to be usable directly. Its most > reliable course is to always use the proper APIs and rely on the kernel > to implement the most efficient mechanism for those APIs.Understood. In case of misunderstanding, I *am* talking about usermode using the proper APIs and relying on the kernel to implement the most efficient mechanism. I''m saying that, at least in many cases (and certainly in the case of 64-bit PV guests), the most efficient mechanism will be tsc instead of pvclock, but I think every pvops kernel running on Xen (as well as Stefano''s PV-on-HVM patch) assumes pvclock is always best. It is that fiction (pvclock is always best) that I am concerned about building further infrastructure on top of. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel