Andy Lutomirski
2018-Oct-04 14:00 UTC
[patch 00/11] x86/vdso: Cleanups, simmplifications and CLOCK_TAI support
> On Oct 4, 2018, at 1:11 AM, Peter Zijlstra <peterz at infradead.org> wrote: > >> On Thu, Oct 04, 2018 at 09:54:45AM +0200, Vitaly Kuznetsov wrote: >> I was hoping to hear this from you :-) If I am to suggest how we can >> move forward I'd propose: >> - Check if pure TSC can be used on SkyLake+ systems (where TSC scaling >> is supported). >> - Check if non-masterclock mode is still needed. E.g. HyperV's TSC page >> clocksource is a single page for the whole VM, not a per-cpu thing. Can >> we think that all the buggy hardware is already gone? > > No, and it is not the hardware you have to worry about (mostly), it is > the frigging PoS firmware people put on it. > > Ever since Nehalem TSC is stable (unless you get to >4 socket systems, > after which it still can be, but bets are off). But even relatively > recent systems fail the TSC sync test because firmware messes it up by > writing to either MSR_TSC or MSR_TSC_ADJUST. > > But the thing is, if the TSC is not synced, you cannot use it for > timekeeping, full stop. So having a single page is fine, it either > contains a mult/shift that is valid, or it indicates TSC is messed up > and you fall back to something else. > > There is no inbetween there. > > For sched_clock we can still use the global page, because the rate will > still be the same for each cpu, it's just offset between CPUs and the > code compensates for that.But if we?re in a KVM guest, then the clock will jump around on the same *vCPU* when the vCPU migrates. But I don?t see how kvmclock helps here, since I don?t think it?s used for sched_clock.
Peter Zijlstra
2018-Oct-04 19:31 UTC
[patch 00/11] x86/vdso: Cleanups, simmplifications and CLOCK_TAI support
On Thu, Oct 04, 2018 at 07:00:45AM -0700, Andy Lutomirski wrote:> > On Oct 4, 2018, at 1:11 AM, Peter Zijlstra <peterz at infradead.org> wrote: > > > >> On Thu, Oct 04, 2018 at 09:54:45AM +0200, Vitaly Kuznetsov wrote: > >> I was hoping to hear this from you :-) If I am to suggest how we can > >> move forward I'd propose: > >> - Check if pure TSC can be used on SkyLake+ systems (where TSC scaling > >> is supported). > >> - Check if non-masterclock mode is still needed. E.g. HyperV's TSC page > >> clocksource is a single page for the whole VM, not a per-cpu thing. Can > >> we think that all the buggy hardware is already gone? > > > > No, and it is not the hardware you have to worry about (mostly), it is > > the frigging PoS firmware people put on it. > > > > Ever since Nehalem TSC is stable (unless you get to >4 socket systems, > > after which it still can be, but bets are off). But even relatively > > recent systems fail the TSC sync test because firmware messes it up by > > writing to either MSR_TSC or MSR_TSC_ADJUST. > > > > But the thing is, if the TSC is not synced, you cannot use it for > > timekeeping, full stop. So having a single page is fine, it either > > contains a mult/shift that is valid, or it indicates TSC is messed up > > and you fall back to something else. > > > > There is no inbetween there. > > > > For sched_clock we can still use the global page, because the rate will > > still be the same for each cpu, it's just offset between CPUs and the > > code compensates for that. > > But if we?re in a KVM guest, then the clock will jump around on the > same *vCPU* when the vCPU migrates.Urgh yes..> But I don?t see how kvmclock helps here, since I don?t think it?s used > for sched_clock.I get hits on kvm_sched_clock, but haven't looked further. Anyway, Most of the argument still holds, either TSC is synced or it is not and it _really_ should not be used. Not much middle ground there.
Andy Lutomirski
2018-Oct-04 20:05 UTC
[patch 00/11] x86/vdso: Cleanups, simmplifications and CLOCK_TAI support
> On Oct 4, 2018, at 12:31 PM, Peter Zijlstra <peterz at infradead.org> wrote: > > On Thu, Oct 04, 2018 at 07:00:45AM -0700, Andy Lutomirski wrote: >>> On Oct 4, 2018, at 1:11 AM, Peter Zijlstra <peterz at infradead.org> wrote: >>> >>>> On Thu, Oct 04, 2018 at 09:54:45AM +0200, Vitaly Kuznetsov wrote: >>>> I was hoping to hear this from you :-) If I am to suggest how we can >>>> move forward I'd propose: >>>> - Check if pure TSC can be used on SkyLake+ systems (where TSC scaling >>>> is supported). >>>> - Check if non-masterclock mode is still needed. E.g. HyperV's TSC page >>>> clocksource is a single page for the whole VM, not a per-cpu thing. Can >>>> we think that all the buggy hardware is already gone? >>> >>> No, and it is not the hardware you have to worry about (mostly), it is >>> the frigging PoS firmware people put on it. >>> >>> Ever since Nehalem TSC is stable (unless you get to >4 socket systems, >>> after which it still can be, but bets are off). But even relatively >>> recent systems fail the TSC sync test because firmware messes it up by >>> writing to either MSR_TSC or MSR_TSC_ADJUST. >>> >>> But the thing is, if the TSC is not synced, you cannot use it for >>> timekeeping, full stop. So having a single page is fine, it either >>> contains a mult/shift that is valid, or it indicates TSC is messed up >>> and you fall back to something else. >>> >>> There is no inbetween there. >>> >>> For sched_clock we can still use the global page, because the rate will >>> still be the same for each cpu, it's just offset between CPUs and the >>> code compensates for that. >> >> But if we?re in a KVM guest, then the clock will jump around on the >> same *vCPU* when the vCPU migrates. > > Urgh yes.. > >> But I don?t see how kvmclock helps here, since I don?t think it?s used >> for sched_clock. > > I get hits on kvm_sched_clock, but haven't looked further.Ok, so KVM is using the per-vCPU pvclock data for sched_clock. Which hopefully does something intelligent. Regardless of any TSC syncing issues, a paravirt clock should presumably be used for sched_clock to account for time that the vCPU was stopped. It would be fantastic if this stuff were documented much better, both in terms of the data structures and the Linux code.
Maybe Matching Threads
- [patch 00/11] x86/vdso: Cleanups, simmplifications and CLOCK_TAI support
- [patch 00/11] x86/vdso: Cleanups, simmplifications and CLOCK_TAI support
- [patch 00/11] x86/vdso: Cleanups, simmplifications and CLOCK_TAI support
- [patch 00/11] x86/vdso: Cleanups, simmplifications and CLOCK_TAI support
- [patch 00/11] x86/vdso: Cleanups, simmplifications and CLOCK_TAI support