Langsdorf, Mark
2007-Mar-21 21:15 UTC
[Xen-devel] x86 Physical CPUs at different frequencies - timer error messages
I''m experimenting with getting PowerNow! running on SMP Opteron systems. One problem I''m having is that Xen assumes that all physical processors are incrementing the TSCs at the same rate, and complains in the timer ISR that time is going backward if this isn''t the case: "Timer ISR/1: Time went backwards: delta=BIGNUM1 delta_cpu=BIGNUM2 shadow=BIGNUM3 off=BIGNUM4 processed=BIGNUM5 cpu_processed=BIGNUM6 0: BIGNUM5 1: BIGNUM6" I''m not surprised this is happening, since the TSC values on processor 0 and processor 1 increment at different rates and tend to have wildly differing values. Normally AMD strongly discourages using TSC for time-keeping with PowerNow! enabled for exactly this reason. However, the worst effects of time skew that we see on native systems is not happening on my dom0 with PowerNow!. How much should I worry about this and what can be done to mitigate it? -Mark Langsdorf AMD, Inc. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Pratt
2007-Mar-21 22:32 UTC
RE: [Xen-devel] x86 Physical CPUs at different frequencies - timer error messages
> I''m experimenting with getting PowerNow! running on > SMP Opteron systems. > > One problem I''m having is that Xen assumes that all > physical processors are incrementing the TSCs at > the same rate, and complains in the timer ISR that > time is going backward if this isn''t the case:Xen *is* able to cope with TSCs running at different frequencies. It calibrates each CPU every few seconds on the assumption that the frequencies are basically stable but different. To integrate with PowerNow, after you''ve changed the frequency you need to run the code to generate a new time record and propagate it to the VCPU of the guest currently using the CPU. For the frequency field, you are best off scaling the frequency measured in the last calibration period rather than assuming the new target frequency is precise. Ian> "Timer ISR/1: Time went backwards: delta=BIGNUM1 > delta_cpu=BIGNUM2 shadow=BIGNUM3 off=BIGNUM4 > processed=BIGNUM5 cpu_processed=BIGNUM6 > 0: BIGNUM5 > 1: BIGNUM6" > > I''m not surprised this is happening, since the TSC > values on processor 0 and processor 1 increment at > different rates and tend to have wildly differing > values. Normally AMD strongly discourages using > TSC for time-keeping with PowerNow! enabled for > exactly this reason. > > However, the worst effects of time skew that we > see on native systems is not happening on my dom0 > with PowerNow!. How much should I worry about this > and what can be done to mitigate it? > > -Mark Langsdorf > AMD, Inc. > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
hi I have attach a disk onto the vm and now I try to understand the code relative to blk-attach it seems like that device_creat()->createdevice()->addStoreEntries() ,add store entry into xenstore , then backend and frontend will watch and do several related work to real add-device in backend and frontend ,is it right? and xenstore have watches to check the related entry ,is it right? I am confused ,which code or function to the real adding work, I guess it is the watches do it ,but I cannot find that fucntions could you help me Thanks in advance _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
hi I try to understand the code relative to dynamically add block device in a running VM it seems like that device_creat()->createdevice()->addStoreEntries() ,add store entry into xenstore , then backend and frontend will watch and do several related work to real add-device in backend and frontend ,is it right? and xenstore have watches to check the related entry ,is it right? I am confused ,which code or function to the real adding work, I guess it is the watches do it ,but I cannot find that fucntions could you help me Thanks in advance _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
hi I try to understand the code relative to blk-attach it seems like that device_creat()->createdevice()->addStoreEntries() ,add store entry into xenstore then backend and frontend will watch and do several related work to real add-device in backend and frontend ,is it right? and xenstore have watches to check the related entry ,is it right? I am confused ,which code or function to the real adding work I guess there is somekind watches which do it ,but I cannot find that fucntions I am confused about it could you help me Thanks in advance _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Langsdorf, Mark
2007-Jul-16 19:43 UTC
RE: [Xen-devel] x86 Physical CPUs at different frequencies - timer error messages
> > I''m experimenting with getting PowerNow! running on > > SMP Opteron systems. > > > > One problem I''m having is that Xen assumes that all > > physical processors are incrementing the TSCs at > > the same rate, and complains in the timer ISR that > > time is going backward if this isn''t the case: > > Xen *is* able to cope with TSCs running at different frequencies. It > calibrates each CPU every few seconds on the assumption that the > frequencies are basically stable but different. > > To integrate with PowerNow, after you''ve changed the > frequency you need to run the code to generate a new > time record and propagate it to the VCPU of the guest > currently using the CPU. For the frequency field, you > are best off scaling the frequency measured in the last > calibration period rather than assuming the new target > frequency is precise.Could you give some rough hints on how to do that? Nothing in time-xen.c or the published hypercalls looks more than vaguely related. All the functions and hypercalls deal with absolute offsets. The only reference to a frequency field is in vcpu_time_info, but no functions take that as an argument. vcpu_info doesn''t have anything either. What functions calls/hypercalls should I be dealing with? -Mark Langsdorf Operating System Research Center AMD _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Mark Langsdorf
2007-Jul-26 21:00 UTC
Re: [Xen-devel] x86 Physical CPUs at different frequencies - timer error messages
> Xen *is* able to cope with TSCs running at different frequencies. It > calibrates each CPU every few seconds on the assumption that the > frequencies are basically stable but different. > > To integrate with PowerNow, after you''ve changed the > frequency you need to run the code to generate a new > time record and propagate it to the VCPU of the guest > currently using the CPU. For the frequency field, you > are best off scaling the frequency measured in the last > calibration period rather than assuming the new target > frequency is precise.Is something like this what you meant? I''m still getting a half dozen Time error messages every time I increase frequency. diff -r a70de77dd8d3 arch/i386/kernel/time-xen.c --- a/arch/i386/kernel/time-xen.c Tue Jul 10 10:18:24 2007 +0100 +++ b/arch/i386/kernel/time-xen.c Thu Jul 26 15:21:38 2007 -0500 @@ -50,6 +50,7 @@ #include <linux/percpu.h> #include <linux/kernel_stat.h> #include <linux/posix-timers.h> +#include <linux/cpufreq.h> #include <asm/io.h> #include <asm/smp.h> @@ -1118,6 +1119,95 @@ void local_teardown_timer(unsigned int c BUG_ON(cpu == 0); unbind_from_irqhandler(per_cpu(timer_irq, cpu), NULL); } +#endif + +#if CONFIG_CPU_FREQ +/* + * cpufreq scaling handling + */ +static int time_cpufreq_notifier(struct notifier_block *nb, unsigned long val, + void *data) +{ + struct cpufreq_freqs *freq = data; + struct vcpu_time_info *info = &vcpu_info(freq->cpu)->time; + u64 new_mult; + char new_shift; + + if (cpu_has(&cpu_data[freq->cpu], X86_FEATURE_CONSTANT_TSC)) + return 0; + + if (val == CPUFREQ_PRECHANGE) + return 0; + + /* re order time per Ian''s instructions */ + { + u64 __cpu_khz = info->tsc_to_system_mul; + do_div(__cpu_khz, freq->new / 1000); + new_mult = __cpu_khz * freq->old / 1000; + new_shift = info->tsc_shift; + while (new_mult > (1LL <<32)) { + new_shift += 1; + new_mult = new_mult >> 1; + } + while (new_mult < (1LL << 31)) { + new_shift -= 1; + new_mult = new_mult << 1; + } + } + do { + rmb(); + info->tsc_to_system_mul = new_mult; + info->tsc_shift = new_shift; + rmb(); + } while (info->version & 1); + get_time_values_from_xen(freq->cpu); + + return 0; +} + +static struct notifier_block time_cpufreq_notifier_block = { + .notifier_call = time_cpufreq_notifier +}; + +static int __init cpufreq_time_setup(void) +{ + if (!cpufreq_register_notifier(&time_cpufreq_notifier_block, + CPUFREQ_TRANSITION_NOTIFIER)) { + printk(KERN_ERR "failed to set up cpufreq notifier\n"); + return -ENODEV; + } + return 0; +} + +core_initcall(cpufreq_time_setup); #endif /* _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2007-Jul-27 07:08 UTC
Re: [Xen-devel] x86 Physical CPUs at different frequencies - timer error messages
On 26/7/07 22:00, "Mark Langsdorf" <mark.langsdorf@amd.com> wrote:> Is something like this what you meant? I''m still > getting a half dozen Time error messages every time > I increase frequency.No. The code belongs in Xen itself. Otherwise no other domains will pick up the new scale factor, and even dom0 itself may lose it as soon as it re-picks up the scale information from Xen. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Langsdorf, Mark
2007-Jul-27 14:30 UTC
RE: [Xen-devel] x86 Physical CPUs at different frequencies - timer error messages
> > Is something like this what you meant? I''m still > > getting a half dozen Time error messages every time > > I increase frequency. > > No. The code belongs in Xen itself. Otherwise no other > domains will pick up the new scale factor, and even > dom0 itself may lose it as soon as it > re-picks up the scale information from Xen.Oops. I take it there isn''t currently a hypercall to support changing it in Xen, though? Shouldn''t be too hard to add, but I want to confirm. -Mark Langsdorf Operating System Research Center AMD _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2007-Jul-27 14:32 UTC
Re: [Xen-devel] x86 Physical CPUs at different frequencies - timer error messages
On 27/7/07 15:30, "Langsdorf, Mark" <mark.langsdorf@amd.com> wrote:>> No. The code belongs in Xen itself. Otherwise no other >> domains will pick up the new scale factor, and even >> dom0 itself may lose it as soon as it >> re-picks up the scale information from Xen. > > Oops. > > I take it there isn''t currently a hypercall to > support changing it in Xen, though? Shouldn''t > be too hard to add, but I want to confirm.Correct. Makes sense to add a platform_op for this. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel