Does anyone on this list have an idea of expected accuracy of chipset time sources (e.g., PIT/HPET)? I''ve been looking at relative frequency stability of the CPU-local TSC and platform HPET on a few different systems, and see weird instabilities that I cannot explain. On a couple of systems based on Intel E7520 (Lindenhurst) chipset I see the relative frequency is stable to within a couple of parts per million (which is within measurement error). Either the two are clocked from the same crystal, or two crystals with low frequency drift/wander (but not unexpectedly low -- I don''t think crystals should drift over short time periods). In contrast, two different systems based on AMD 8111 chipset perform very badly. The relative frequency of TSC vs HPET is stable only to within +/- 30ppm! This means that, during one second, timers based on these two sources can diverge by around 30us, which is a *lot*. Clearly the TSC and HPET are driven off different time sources, but I almost cannot believe that both are driven off crystals -- one of the two must have a 1Hz frequency drift/wander of 30ppm which is totally outside the specs I''d expect of a crystal source. Any ideas what could be going on?! This is something of a pain for the new time code, which syncs the local TSCs to the chipset timer.... -- Keir (I am talking frequency stability here, by the way, not frequency tolerance. I know a crystal can be 100s of ppm out of whack from what is stamped on the can, but I think over short time periods it should not drift noticeably from its natural frequency). _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I can only tell you we''ve seen weird numbers from the opteron tsc, to the point that we don''t entirely trust it. Not sure why. ron _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> In contrast, two different systems based on AMD 8111 chipset > perform very badly. The relative frequency of TSC vs HPET is > stable only to within +/- 30ppm! This means that, during one > second, timers based on these two sources can diverge by > around 30us, which is a *lot*. Clearly the TSC and HPET are > driven off different time sources, but I almost cannot > believe that both are driven off crystals -- one of the two > must have a 1Hz frequency drift/wander of 30ppm which is > totally outside the specs I''d expect of a crystal source. > > Any ideas what could be going on?! This is something of a > pain for the new time code, which syncs the local TSCs to the > chipset timer....These systems haven''t got a ''spread spectrum''[*] PCI bus clock per chance? It might be worth seeing if there''s a BIOS option for it. Ian [*] This is a horrible hack used by some system vendors to cheat EM emmisions regulations. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 2005-07-29 at 18:26 +0100, Keir Fraser wrote:> Does anyone on this list have an idea of expected accuracy of chipset > time sources (e.g., PIT/HPET)? I''ve been looking at relative frequency > stability of the CPU-local TSC and platform HPET on a few different > systems, and see weird instabilities that I cannot explain.The TSC is not a reliable time source. There is no guarantee of synchronization between multiple CPUs. Also note that the TSC will stop in ACPI C3 mode. Probably not an issue for Xen today, but could be in the future, especially as servers start doing more power management. The HPET is preferable for a time source on systems that this is available on.> > On a couple of systems based on Intel E7520 (Lindenhurst) chipset I see > the relative frequency is stable to within a couple of parts per > million (which is within measurement error). Either the two are clocked > from the same crystal, or two crystals with low frequency drift/wander > (but not unexpectedly low -- I don''t think crystals should drift over > short time periods). > > In contrast, two different systems based on AMD 8111 chipset perform > very badly. The relative frequency of TSC vs HPET is stable only to > within +/- 30ppm! This means that, during one second, timers based on > these two sources can diverge by around 30us, which is a *lot*. Clearly > the TSC and HPET are driven off different time sources, but I almost > cannot believe that both are driven off crystals -- one of the two must > have a 1Hz frequency drift/wander of 30ppm which is totally outside the > specs I''d expect of a crystal source.These types of observations are not inconsistent with what others have seen. Very platform dependent.> > Any ideas what could be going on?! This is something of a pain for the > new time code, which syncs the local TSCs to the chipset timer....This is a reality of the various hardware time sources. There are many problems associated with this. You might look at the OLS paper on a proposed rewrite for the Linux time subsystem to get a feel for how the kernel people are trying to deal with some of these issues. Michael> > -- Keir > > (I am talking frequency stability here, by the way, not frequency > tolerance. I know a crystal can be 100s of ppm out of whack from what > is stamped on the can, but I think over short time periods it should > not drift noticeably from its natural frequency). >-- Michael Hohnbaum 503 578 5486 hohnbaum@us.ibm.com t/l 775 5486 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 29 Jul 2005, at 19:17, Michael Hohnbaum wrote:> The TSC is not a reliable time source. There is no guarantee of > synchronization between multiple CPUs. Also note that the TSC will > stop in ACPI C3 mode. Probably not an issue for Xen today, but could > be in the future, especially as servers start doing more power > management. The HPET is preferable for a time source on systems > that this is available on.We''re not relying on TSC synchronisation between CPUs. They can run at different frequencies, stop in deep sleep, and so on. But we *do* want the frequency of each one to be as stable as possible. I''d expect to get <1ppm stability from a crystal source at constant temperature, no problem. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> We''re not relying on TSC synchronisation between CPUs. They > can run at different frequencies, stop in deep sleep, and so > on. But we *do* want the frequency of each one to be as > stable as possible. I''d expect to get <1ppm stability from a > crystal source at constant temperature, no problem.I''d wager the TSC is stable and it''s the HPET that''s wandering... We''ll probably just have to add more damping on the re-calibration. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 29 Jul 2005, at 18:53, Ian Pratt wrote:>> Any ideas what could be going on?! This is something of a >> pain for the new time code, which syncs the local TSCs to the >> chipset timer.... > > These systems haven''t got a ''spread spectrum''[*] PCI bus clock per > chance? It might be worth seeing if there''s a BIOS option for it.I did check one of our machines, and spread spectrum was definitely disabled. In any case, I''d expect that to jitter the clock at some high frequency, but not add/remove ticks in a way that would be discernable at 1Hz. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 2005-07-29 at 19:29 +0100, Keir Fraser wrote:> On 29 Jul 2005, at 19:17, Michael Hohnbaum wrote: > > > The TSC is not a reliable time source. There is no guarantee of > > synchronization between multiple CPUs. Also note that the TSC will > > stop in ACPI C3 mode. Probably not an issue for Xen today, but could > > be in the future, especially as servers start doing more power > > management. The HPET is preferable for a time source on systems > > that this is available on. > > We''re not relying on TSC synchronisation between CPUs. They can run at > different frequencies, stop in deep sleep, and so on. But we *do* want > the frequency of each one to be as stable as possible. I''d expect to > get <1ppm stability from a crystal source at constant temperature, no > problem.Not being a hardware expert, terminology and details might be a bit off, but my understanding is that yes, this level of instability from the hardware time source (not necessarily a crystal) is not uncommon. HPET time source tends to be more reliable (stable) than the TSC. There tend to be many factors that affect the accuracy of the TSC, most of which I do not grok. Much of the complexity in the Linux time subsystem is there to deal with the inadequacy of hardware time sources. Michael> > -- Keir > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >-- Michael Hohnbaum 503 578 5486 hohnbaum@us.ibm.com t/l 775 5486 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 29 Jul 2005, at 19:29, Ian Pratt wrote:>> We''re not relying on TSC synchronisation between CPUs. They >> can run at different frequencies, stop in deep sleep, and so >> on. But we *do* want the frequency of each one to be as >> stable as possible. I''d expect to get <1ppm stability from a >> crystal source at constant temperature, no problem. > > I''d wager the TSC is stable and it''s the HPET that''s wandering... > > We''ll probably just have to add more damping on the re-calibration.I haven''t done any further tests to see how low the frequency of wandering goes. Perhaps we''ll find the platform timer is more stable over e.g., 60s periods. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Jul 29, 2005 at 11:51:19AM -0600, Ronald G. Minnich wrote:> I can only tell you we''ve seen weird numbers from the opteron tsc, to the > point that we don''t entirely trust it. Not sure why.Opterons can execute RDTSC out of order. -ben -- "Time is what keeps everything from happening all at once." -- John Wheeler _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
You have to consider that the HPET and TSC are both driven by a separate PLL that is based of some sort of crystal/oscillator. These are often not as precise as we''d want them to be. If you do a google with "crystal oscillator ppm" (or some such), you will find that most of these are within the 50-100 ppm ranges [these are the guaranteed values, which probably means that the typical is around 25-50 ppm, much along the lines of your findings]. If you want more precise crystals than that, you''ll be looking at expensive stuff, and when you want less than around 10 ppm, you''re probably looking at temperature controlled ones (aka oven oscillators). They are both expensive and bulky, and very unlikely to be found on the motherboard. A 2 ppm, 2MHz clock generator that I know of from a GSM base station is about 80 x 50 x 50 mm. Not sure how much it cost, but I''m sure it''s much more than the clocks that are on even the most expensive motherboards. It''s possible (or even LIKELY) that the differences you see between the AMD and Intel platforms stem from something along the lines of using different or the same oscillator for different parts of the system. One way to reduce the clock-drift between different parts (assuming this is something desirable, and that is a debate of it''s own) would be to use the input to the CPU PLL as an input to the (PLL that drives) HPET. I know for sure that the AMD chipset clock for HPET is NOT derived from the same as the CPU clock source. I don''t know how Intel has solved their clocking... And a note on RDTSC: There''s a RDTSCP which is a "serializing" instruction. It also clobbers, I think, ECX, with the current CPU number. Standard RDTSC is not serializing, so it can be performed out of order by several instructions. I doubt that this will cause much of the error discussed, but it may contribute in a positive or negative way... -- Mats> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of > Keir Fraser > Sent: 29 July 2005 19:54 > To: Ian Pratt > Cc: xen-devel List; Michael Hohnbaum > Subject: Re: [Xen-devel] HPET/PIT timer accuracy > > > On 29 Jul 2005, at 19:29, Ian Pratt wrote: > > >> We''re not relying on TSC synchronisation between CPUs. > They can run > >> at different frequencies, stop in deep sleep, and so on. > But we *do* > >> want the frequency of each one to be as stable as possible. I''d > >> expect to get <1ppm stability from a crystal source at constant > >> temperature, no problem. > > > > I''d wager the TSC is stable and it''s the HPET that''s wandering... > > > > We''ll probably just have to add more damping on the re-calibration. > > I haven''t done any further tests to see how low the frequency > of wandering goes. Perhaps we''ll find the platform timer is > more stable over e.g., 60s periods. > > -- Keir > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 1 Aug 2005, at 13:17, Petersson, Mats wrote:> You have to consider that the HPET and TSC are both driven by a > separate > PLL that is based of some sort of crystal/oscillator. These are often > not as precise as we''d want them to be. If you do a google with > "crystal > oscillator ppm" (or some such), you will find that most of these are > within the 50-100 ppm ranges [these are the guaranteed values, which > probably means that the typical is around 25-50 ppm, much along the > lines of your findings]. If you want more precise crystals than that, > you''ll be looking at expensive stuff, and when you want less than > around > 10 ppm, you''re probably looking at temperature controlled ones (aka > oven > oscillators). They are both expensive and bulky, and very unlikely to > be > found on the motherboard.Those variations of up to 100ppm are primarily composed of two factors which I can definitely discount from my own tests: drift due to temperature change, and frequency tolerance. Where a separate frequency tolerance figure is not given, that is often folded into the overall stability figure: I know that can be 100ppm for a chipset crystal. And that''ll be why a 2ppm clock generator is so expensive and bulky: tolerance is hard to achieve, and drift is hard to compensate for over a wide temperature range (that circuitry is where the bulk comes from). *However* I am talking about frequency jitter at 1Hz: any constant deviation from the frequency stamped on the can, due to ''tolerances'', can obviously be discounted. Also, at 1Hz temperature changes will be negligible, and the jitter continues at the same magnitude even when the machine has warmed up and reached a steady temperature. A crystal simply shouldn''t jitter at that frequency, all external factors held constant. It''s resonating at its natural frequency (or some harmonic) after all. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> You have to consider that the HPET and TSC are both driven by > a separate PLL that is based of some sort of > crystal/oscillator. These are often not as precise as we''d > want them to be. If you do a google with "crystal oscillator > ppm" (or some such), you will find that most of these are > within the 50-100 ppm ranges [these are the guaranteed > values, which probably means that the typical is around 25-50 > ppm, much along the lines of your findings].The figures you are quoting are for frequency *accuracy*, not *stability*. Crystals typically have good stability, varying mostly as a result of temperature changes. The only time I''ve ever seen such high instability on a crystal was when the load capacitors were a couple of orders of magnitude out and it was a miracle the circuit was oscillating at all... Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
The RDTSC instruction is out-of-order on Intel CPU''s as well. You can place serializing instructions such as CPUID before and/or after RDTSC to get better locality. Unfortunately, CPUID expends many more cycles than the RDTSC instruction on its own. -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Benjamin LaHaise Sent: Friday, July 29, 2005 6:33 PM To: Ronald G. Minnich Cc: Ian Pratt; xen-devel List Subject: Re: [Xen-devel] HPET/PIT timer accuracy On Fri, Jul 29, 2005 at 11:51:19AM -0600, Ronald G. Minnich wrote:> I can only tell you we''ve seen weird numbers from the opteron tsc, to > the point that we don''t entirely trust it. Not sure why.Opterons can execute RDTSC out of order. -ben -- "Time is what keeps everything from happening all at once." -- John Wheeler _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel