It seems that VCPU system time isn''t monotonic (using 3.0.4). It seems
it might be correlated to when a VCPU is switched across real CPUs but I
haven''t conclusively proved that. But e.g.:
{
old = {
time = {
version = 0x4ec
pad0 = 0xe8e0
tsc_timestamp = 0x22cc8398b7194
system_time = 0xe8e0345d8805
tsc_to_system_mul = 0xd62c0083
tsc_shift = ''\377''
pad1 = [ ''\002'', ''\027'',
''\365'' ]
}
result = 0xe8e0484568fa
tsc = 0x22cc86921ab00
cpu = 0
}
new = {
time = {
version = 0x4ee
pad0 = 0
tsc_timestamp = 0x22cc7db96cd29
system_time = 0xe8e00d1031f3
tsc_to_system_mul = 0xd62ae844
tsc_shift = ''\377''
pad1 = [ ''\357'', ''\002'',
''\365'' ]
}
result = 0xe8e048456012
tsc = 0x22cc869225443
cpu = 0
}
delta = 0xfffffffffffff718
}
>From what I can work out, time is supposed to be monotonic but I admit I
can''t really understand the time code yet at least. I couldn''t
find any
documentation on what to expect from system time. Any suggestions?
This seems to happen across all the hardware we''ve tried but this
particular case is a Sun V20Z with two CPUs:
x86 (AuthenticAMD family 15 model 5 step 10 clock 2392 MHz)
AMD Opteron(tm) Processor 250
cheers,
john
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
On 26/3/07 19:23, "John Levon" <levon@movementarian.org> wrote:>> From what I can work out, time is supposed to be monotonic but I admit I > can''t really understand the time code yet at least. I couldn''t find any > documentation on what to expect from system time. Any suggestions? > > This seems to happen across all the hardware we''ve tried but this > particular case is a Sun V20Z with two CPUs: > > x86 (AuthenticAMD family 15 model 5 step 10 clock 2392 MHz) > AMD Opteron(tm) Processor 250Small backwards time deltas are possible from the current time code. You''ll have to filter them out yourself if you can''t deal with them. We could add extra code in Xen to stop this happening for any individual VCPU. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> It seems that VCPU system time isn''t monotonic (using 3.0.4). It seems > it might be correlated to when a VCPU is switched across real CPUs butI> haven''t conclusively proved that. But e.g.:What output do you get when you hit ''t'' a few times on the xen serial console? There''s no guarantee that the system time calculated will be perfectly monotonic, but it should be very close. If the guest needs it to be monotonic, the time reading code should simply clamp the value that''s read to ensure it always goes up. A little jitter at the microsecond granularity should be just fine. On your system it appears to be a couple of microseconds out, which is on the high side of what we''ve observed. Normally you only see that kind of mismatch on systems with TSCs running off different crystals. Ian> { > old = { > time = { > version = 0x4ec > pad0 = 0xe8e0 > tsc_timestamp = 0x22cc8398b7194 > system_time = 0xe8e0345d8805 > tsc_to_system_mul = 0xd62c0083 > tsc_shift = ''\377'' > pad1 = [ ''\002'', ''\027'', ''\365'' ] > } > result = 0xe8e0484568fa > tsc = 0x22cc86921ab00 > cpu = 0 > } > new = { > time = { > version = 0x4ee > pad0 = 0 > tsc_timestamp = 0x22cc7db96cd29 > system_time = 0xe8e00d1031f3 > tsc_to_system_mul = 0xd62ae844 > tsc_shift = ''\377'' > pad1 = [ ''\357'', ''\002'', ''\365'' ] > } > result = 0xe8e048456012 > tsc = 0x22cc869225443 > cpu = 0 > } > delta = 0xfffffffffffff718 > } > > >From what I can work out, time is supposed to be monotonic but Iadmit> I > can''t really understand the time code yet at least. I couldn''t findany> documentation on what to expect from system time. Any suggestions? > > This seems to happen across all the hardware we''ve tried but this > particular case is a Sun V20Z with two CPUs: > > x86 (AuthenticAMD family 15 model 5 step 10 clock 2392 MHz) > AMD Opteron(tm) Processor 250 > > cheers, > john > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 26/3/07 19:50, "Ian Pratt" <Ian.Pratt@cl.cam.ac.uk> wrote:> On your system it appears to be a couple of microseconds out, which is > on the high side of what we''ve observed. Normally you only see that kind > of mismatch on systems with TSCs running off different crystals.More likely a jittery chipset timer -- we''ve observed less-than-ideal stability from some chipset timers, which can throw us off a bit when independently sync''ing the TSCs (which each CPU does for its TSC independently every couple of seconds). -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, Mar 26, 2007 at 07:47:49PM +0100, Keir Fraser wrote:> >> From what I can work out, time is supposed to be monotonic but I admit I > > can''t really understand the time code yet at least. I couldn''t find any > > documentation on what to expect from system time. Any suggestions? > > > > This seems to happen across all the hardware we''ve tried but this > > particular case is a Sun V20Z with two CPUs: > > > > x86 (AuthenticAMD family 15 model 5 step 10 clock 2392 MHz) > > AMD Opteron(tm) Processor 250 > > Small backwards time deltas are possible from the current time code. You''ll > have to filter them out yourself if you can''t deal with them. We could add > extra code in Xen to stop this happening for any individual VCPUSome instrumentation indicated that we had cross-VCPU jitter of significant deltas, ~18us at worst. Though the instrumentation wasn''t completely reliable so that might not be accurate. On real hardware we deal with any jitter via comparing against a regular clock on CPU0. This obviously won''t work for Xen since we have no control over where VCPU0 actually lives. Presumably the Linux code (monotonic_clock() especially) must have something to handle it, but I can''t see where... regards john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, Mar 26, 2007 at 07:59:27PM +0100, Keir Fraser wrote:> On 26/3/07 19:50, "Ian Pratt" <Ian.Pratt@cl.cam.ac.uk> wrote: > > > On your system it appears to be a couple of microseconds out, which is > > on the high side of what we''ve observed. Normally you only see that kind > > of mismatch on systems with TSCs running off different crystals. > > More likely a jittery chipset timer -- we''ve observed less-than-ideal > stability from some chipset timers, which can throw us off a bit when > independently sync''ing the TSCs (which each CPU does for its TSC > independently every couple of seconds).And what about cross-VCPU monotonicity? It''s called very frequently and having to fake monotonicity via a single variable across all CPUs would not be pleasant, unless I can think of something smarter... regards john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > More likely a jittery chipset timer -- we''ve observedless-than-ideal> > stability from some chipset timers, which can throw us off a bitwhen> > independently sync''ing the TSCs (which each CPU does for its TSC > > independently every couple of seconds). > > And what about cross-VCPU monotonicity? It''s called very frequentlyand> having to fake monotonicity via a single variable across all CPUswould> not be pleasant, unless I can think of something smarter...One LOCK''ed cmpxchg is still a lot quicker than reading the pm_timer or hpet... Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 26/3/07 21:14, "John Levon" <levon@movementarian.org> wrote:>> More likely a jittery chipset timer -- we''ve observed less-than-ideal >> stability from some chipset timers, which can throw us off a bit when >> independently sync''ing the TSCs (which each CPU does for its TSC >> independently every couple of seconds). > > And what about cross-VCPU monotonicity? It''s called very frequently and > having to fake monotonicity via a single variable across all CPUs would > not be pleasant, unless I can think of something smarter...Yet it is what you need to do if you want to absolutely guarantee monotonic time across all VCPUS. I don''t think there''s any way around that, but as Ian says it''s still going to be cheaper than reading a chipset timer. I''d be inclined to settle for per-vcpu and per-OS-task monotonicity. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 26/3/07 21:04, "John Levon" <levon@movementarian.org> wrote:>> Small backwards time deltas are possible from the current time code. You''ll >> have to filter them out yourself if you can''t deal with them. We could add >> extra code in Xen to stop this happening for any individual VCPU > > Some instrumentation indicated that we had cross-VCPU jitter of > significant deltas, ~18us at worst. Though the instrumentation wasn''t > completely reliable so that might not be accurate.I should add that time synchronisation is currently broken in xen-unstable, and has been for about two weeks. I just checked in a patch (based on one from Jan Beulich) to fix this (changeset 14573:ba9d3fd4ee4b). -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, Mar 26, 2007 at 09:04:47PM +0100, John Levon wrote:> > >> From what I can work out, time is supposed to be monotonic but I admit I > > > can''t really understand the time code yet at least. I couldn''t find any > > > documentation on what to expect from system time. Any suggestions? > > > > > > This seems to happen across all the hardware we''ve tried but this > > > particular case is a Sun V20Z with two CPUs: > > > > > > x86 (AuthenticAMD family 15 model 5 step 10 clock 2392 MHz) > > > AMD Opteron(tm) Processor 250 > > > > Small backwards time deltas are possible from the current time code. You''ll > > have to filter them out yourself if you can''t deal with them. We could add > > extra code in Xen to stop this happening for any individual VCPU > > Some instrumentation indicated that we had cross-VCPU jitter of > significant deltas, ~18us at worst. Though the instrumentation wasn''t > completely reliable so that might not be accurate.I''ve done some more testing, the problem appears to be significantly worse than you''re claiming. I store the previously read value in the thread structure and breakpoint whenever it fails: [2]> fffffffece9e6040::print kthread_t t_dhrtime t_dhrtime_tsc t_dhrtime_cpu t_dhrtime_vers t_dhrtime = 0x3de7923989c t_dhrtime_tsc = 0x9414981ce0e t_dhrtime_cpu = 0x2 t_dhrtime_vers = 0x2616 And the new values: t_dhrtime = 0x3de7923088b t_dhrtime_tsc = 0x9414982169b t_dhrtime_cpu = 0x2 t_dhrtime_vers = 0x2618 [2]> 0x3de7923989c - 0x000003de7923088b=D 36881 That''s 36us. It doesn''t even seem that we migrated across physical CPUs given the version field. So something seems significantly awry to me? (I must admit I still don''t quite understand why monotonic_clock() is called monotonic_clock() ...) regards john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > Some instrumentation indicated that we had cross-VCPU jitter of > > significant deltas, ~18us at worst. Though the > instrumentation wasn''t > > completely reliable so that might not be accurate. > > I''ve done some more testing, the problem appears to be > significantly worse than you''re claiming. I store the > previously read value in the thread structure and breakpoint > whenever it fails:Try booting with one physical (maxcpus=1 on the xen command line) CPU just to verify this isn''t a multi-CPU issue. If you''re still seeing jitter, please post the boot logs. We''ve seen this on white-box AMD systems which were over heating and the CPU was bouncing in and out of thermal throttling, but never on a tier-1 vendor system. You could try switching between the PIT and HPET as the calibration source. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Apr 03, 2007 at 03:36:48PM +0100, Ian Pratt wrote:> Try booting with one physical (maxcpus=1 on the xen command line) CPU > just to verify this isn''t a multi-CPU issue.Can reproduce: [1]> 0xfffffffed5cc0120::print kthread_t t_dhrtime t_dhrtime_cpu t_dhrtime_tsc t_dhrtime_vers t_dhrtime = 0x509957b7c2 t_dhrtime_cpu = 0x1 t_dhrtime_tsc = 0xf6f1529888 t_dhrtime_vers = 0x132 [1]> 0x509957b7c2 - 0x000000509957039e=D 46116 See boot log below.> You could try switching between the PIT and HPET as the calibration > source.Doesn''t look like there''s a flag to force PIT usage. I''ll modify the source and try it. john http://www.cl.cam.ac.uk/netos/xen University of Cambridge Computer Laboratory Xen version 3.0.4-1-sun (johnlev@mpklab.sfbay.sun.com) (gcc version 3.4.3 (csl-sol210-3_4-20050802)) Mon Apr 2 16:14:36 PDT 2007 Latest ChangeSet: Mon Apr 02 16:11:18 2007 -0700 13188:3a69441c8c65 (XEN) Command line: /boot/amd64/xen.gz console=com1 com1=9600,8n1 maxcpus=1 (XEN) Physical RAM map: (XEN) 0000000000000000 - 000000000009a000 (usable) (XEN) 000000000009a000 - 00000000000a0000 (reserved) (XEN) 00000000000d0000 - 0000000000100000 (reserved) (XEN) 0000000000100000 - 00000000fbf70000 (usable) (XEN) 00000000fbf70000 - 00000000fbf77000 (ACPI data) (XEN) 00000000fbf77000 - 00000000fbf80000 (ACPI NVS) (XEN) 00000000fbf80000 - 00000000fc000000 (reserved) (XEN) 00000000fec00000 - 00000000fec00400 (reserved) (XEN) 00000000fee00000 - 00000000fee01000 (reserved) (XEN) 00000000fff80000 - 0000000100000000 (reserved) (XEN) System RAM: 4031MB (4127784kB) (XEN) ACPI: RSDP (v002 PTLTD ) @ 0x00000000000f7dc0 (XEN) ACPI: XSDT (v001 PTLTD XSDT 0x06040000 LTP 0x00000000) @ 0x00000000fbf74bd4 (XEN) ACPI: FADT (v003 SUN V20z 0x06040000 PTEC 0x000f4240) @ 0x00000000fbf76c0c (XEN) ACPI: HPET (v001 Sun V20z 0x06040000 PTEC 0x00000000) @ 0x00000000fbf76d00 (XEN) ACPI: MADT (v001 PTLTD APIC 0x06040000 LTP 0x00000000) @ 0x00000000fbf76d38 (XEN) ACPI: SPCR (v001 PTLTD $UCRTBL$ 0x06040000 PTL 0x00000001) @ 0x00000000fbf76dae (XEN) ACPI: SSDT (v001 SUN V20z 0x06040000 LTP 0x00000001) @ 0x00000000fbf76dfe (XEN) ACPI: SSDT (v001 SUN V20z 0x06040000 LTP 0x00000001) @ 0x00000000fbf76e9b (XEN) ACPI: SRAT (v001 SUN V20z 0x06040000 SUN 0x00000001) @ 0x00000000fbf76f38 (XEN) ACPI: DSDT (v001 Sun V20z 0x06040000 MSFT 0x0100000e) @ 0x0000000000000000 (XEN) NUMA turned off (XEN) Faking a node at 0000000000000000-00000000fbf70000 (XEN) Domain heap initialised: DMA width 30 bits (XEN) Xen heap: 13MB (14044kB) (XEN) found SMP MP-table at 000f7df0 (XEN) DMI present. (XEN) Using APIC driver default (XEN) ACPI: Local APIC address 0xfee00000 (XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) (XEN) Processor #0 15:5 APIC version 16 (XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) (XEN) Processor #1 15:5 APIC version 16 (XEN) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) (XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0]) (XEN) IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, GSI 0-23 (XEN) ACPI: IOAPIC (id[0x03] address[0xfc800000] gsi_base[24]) (XEN) IOAPIC[1]: apic_id 3, version 17, address 0xfc800000, GSI 24-27 (XEN) ACPI: IOAPIC (id[0x04] address[0xfc801000] gsi_base[28]) (XEN) IOAPIC[2]: apic_id 4, version 17, address 0xfc801000, GSI 28-31 (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge) (XEN) ACPI: IRQ0 used by override. (XEN) ACPI: IRQ2 used by override. (XEN) Enabling APIC mode: Flat. Using 3 I/O APICs (XEN) ACPI: HPET id: 0x102282a0 base: 0xfed00000 (XEN) Using ACPI (MADT) for SMP configuration information (XEN) Using scheduler: SMP Credit Scheduler (credit) (XEN) Initializing CPU#0 (XEN) Detected 2392.196 MHz processor. (XEN) CPU0: AMD Flush Filter disabled (XEN) CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) (XEN) CPU: L2 Cache: 1024K (64 bytes/line) (XEN) Intel machine check architecture supported. (XEN) Intel machine check reporting enabled on CPU#0. (XEN) CPU0: AMD Opteron(tm) Processor 250 stepping 0a (XEN) Mapping cpu 0 to node 255 (XEN) Total of 1 processors activated. (XEN) ENABLING IO-APIC IRQs (XEN) -> Using new ACK method (XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=0 pin2=0 (XEN) Platform timer is 14.318MHz HPET (XEN) Brought up 1 CPUs (XEN) Machine check exception polling timer started. (XEN) *** LOADING DOMAIN 0 *** (XEN) PHYSICAL MEMORY ARRANGEMENT: (XEN) Dom0 alloc.: 000000000c000000->0000000010000000 (966649 pages to be allocated) (XEN) VIRTUAL MEMORY ARRANGEMENT: (XEN) Loaded kernel: 0000000040000000->00000000408263a8 (XEN) Init. ramdisk: 0000000040827000->0000000043273800 (XEN) Phys-Mach map: 0000000043274000->00000000439f3fc8 (XEN) Start info: 00000000439f4000->00000000439f449c (XEN) Page tables: 00000000439f5000->0000000043a16000 (XEN) Boot stack: 0000000043a16000->0000000043a17000 (XEN) TOTAL: 0000000040000000->0000000043c00000 (XEN) ENTRY ADDRESS: 0000000040800000 (XEN) Dom0 has maximum 1 VCPUs (XEN) Initrd len 0x2a4c800, start at 0x40827000 (XEN) Scrubbing Free RAM: .done. (XEN) Xen trace buffers: disabled (XEN) Std. Loglevel: All (XEN) Guest Loglevel: All (XEN) *** Serial input -> DOM0 (type ''CTRL-a'' three times to switch input to Xen). Loading kmdb... SunOS Release 5.11 Version onnv-johnlev 64-bit _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > Try booting with one physical (maxcpus=1 on the xen command > line) CPU > > just to verify this isn''t a multi-CPU issue. > > Can reproduce: > > [1]> 0xfffffffed5cc0120::print kthread_t t_dhrtime > t_dhrtime_cpu t_dhrtime_tsc t_dhrtime_vers t_dhrtime = > 0x509957b7c2 t_dhrtime_cpu = 0x1 t_dhrtime_tsc = 0xf6f1529888 > t_dhrtime_vers = 0x132 > > [1]> 0x509957b7c2 - 0x000000509957039e=D > 46116Please can you try hitting ''t'' a few times on the Xen debug console (again, maxcpus=1) If you''re seeing time jump back on a single CPU, it must mean that the TSC ran quicker during the current period than what it was calibrated to be running at in the last period. It''s worth adding some debuging printk''s to the calibration code in Xen to see what it thinks is going on from one calibration period to the next. Thanks, Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel