Xen devel - May 2011 - [PATCH] SVM: enables TSC scaling ratio support for SVM

Future AMD CPUs support TSC scaling. It allows guests to have a 
different TSC frequency from host system using this formula: guest_tsc = 
host_tsc * tsc_ratio + vmcb_offset. The tsc_ratio is a 64bit MSR 
contains a fixed-point number in 8.32 format (8 bits for integer part 
and 32bits for fractional part). For instance 0x00000003_80000000 means 
tsc_ratio=3.5.

This patch enables TSC scaling ratio for SVM. With it, guest VMs don''t 
need take #VMEXIT to calculate a translated TSC value when it is running 
under TSC emulation mode. This can SUBSTANTIALLY reduce the rdtsc overhead.

Signed-off-by: Wei Huang <wei.huang2@amd.com>

  arch/x86/hvm/svm/svm.c        |   32 +++++++++++++++++++++++++++++++-
  arch/x86/hvm/svm/vmcb.c       |    4 +++-
  include/asm-x86/hvm/svm/svm.h |    8 ++++++++
  include/asm-x86/msr-index.h   |    3 +++
  4 files changed, 45 insertions(+), 2 deletions(-)




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> From: Wei Huang [mailto:wei.huang2@amd.com]
> Sent: Friday, May 27, 2011 10:35 AM
> To: xen-devel@lists.xensource.com; Keir Fraser
> Subject: [Xen-devel] [PATCH] SVM: enables TSC scaling ratio support for
> SVM
> 
> Future AMD CPUs support TSC scaling. It allows guests to have a
> different TSC frequency from host system using this formula: guest_tsc
> > host_tsc * tsc_ratio + vmcb_offset. The tsc_ratio is a 64bit MSR
> contains a fixed-point number in 8.32 format (8 bits for integer part
> and 32bits for fractional part). For instance 0x00000003_80000000 means
> tsc_ratio=3.5.
> 
> This patch enables TSC scaling ratio for SVM. With it, guest VMs
don''t
> need take #VMEXIT to calculate a translated TSC value when it is
> running
> under TSC emulation mode. This can SUBSTANTIALLY reduce the rdtsc
> overhead.
> 
> Signed-off-by: Wei Huang <wei.huang2@amd.com>
Has this patch been tested across save/restore and migration,
especially between machines with and without the feature and
especially across many migrations where each physical
machine has a different tsc_ratio?

ISTR that this feature does not really do a generic adjustment
so may mis-scale time that has been accrued on one or more
previous physical machines.  In other words, I think the problem
is that it does

(host_tsc * tsc_ratio) + offset

and not

((host_tsc + offset1) * tsc_ratio) + offset2

This can be fixed if you trust cpu_khz to be precise on all
machines, but I don''t think it is sufficiently precise to
guarantee that time never goes backwards in a guest (though
that may be fixable too).  If time DOES go backwards and
the guest detects it, it may switch to a much slower
timer mechanism which could be worse than trapping
rdtsc.

All of this is from vague recollection... if it is all fully
tested across all cases (and sufficient testing proves that
time never goes backwards), consider this just noting a concern.

Also, is this feature visible from an HVM guest kernel?  Is
it visible from a PV guest cpuid (e.g. OS or app)?

Thanks,
Dan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dan,

I tested it by migrating between systems with TSC and without TSC, 
mainly using a Linux guest VM.

I might have missed your point. But what is offset1 in your formula? My 
understanding is that hvm_get_guest_tsc() returns a TSC value guests are 
supposed to use. So when guest_tsc is set, we can re-calculate 
offset=hvm_get_guest_tsc() - host_tsc * ratio. This is how current Xen 
is implemented. Also, as long as host_tsc doesn''t go backwards (i.e. 
TSC_RELIABLE), host_tsc * ratio+offset should be reliable. Are you 
concerned that cpu_khz might be (slightly) different on different cores?

This feature is only available under SVM mode. HVM guests won''t be able
to see it in CPUID. PV guests shouldn''t be able to change the value of 
this MSR.

Thanks,
-Wei

On 05/27/2011 03:41 PM, Dan Magenheimer wrote:
>> From: Wei Huang [mailto:wei.huang2@amd.com]
>> Sent: Friday, May 27, 2011 10:35 AM
>> To: xen-devel@lists.xensource.com; Keir Fraser
>> Subject: [Xen-devel] [PATCH] SVM: enables TSC scaling ratio support for
>> SVM
>>
>> Future AMD CPUs support TSC scaling. It allows guests to have a
>> different TSC frequency from host system using this formula: guest_tsc
>> >> host_tsc * tsc_ratio + vmcb_offset. The tsc_ratio is a 64bit
MSR
>> contains a fixed-point number in 8.32 format (8 bits for integer part
>> and 32bits for fractional part). For instance 0x00000003_80000000 means
>> tsc_ratio=3.5.
>>
>> This patch enables TSC scaling ratio for SVM. With it, guest VMs
don''t
>> need take #VMEXIT to calculate a translated TSC value when it is
>> running
>> under TSC emulation mode. This can SUBSTANTIALLY reduce the rdtsc
>> overhead.
>>
>> Signed-off-by: Wei Huang<wei.huang2@amd.com>
> Has this patch been tested across save/restore and migration,
> especially between machines with and without the feature and
> especially across many migrations where each physical
> machine has a different tsc_ratio?
>
> ISTR that this feature does not really do a generic adjustment
> so may mis-scale time that has been accrued on one or more
> previous physical machines.  In other words, I think the problem
> is that it does
>
> (host_tsc * tsc_ratio) + offset
>
> and not
>
> ((host_tsc + offset1) * tsc_ratio) + offset2
>
> This can be fixed if you trust cpu_khz to be precise on all
> machines, but I don''t think it is sufficiently precise to
> guarantee that time never goes backwards in a guest (though
> that may be fixable too).  If time DOES go backwards and
> the guest detects it, it may switch to a much slower
> timer mechanism which could be worse than trapping
> rdtsc.
>
> All of this is from vague recollection... if it is all fully
> tested across all cases (and sufficient testing proves that
> time never goes backwards), consider this just noting a concern.
>
> Also, is this feature visible from an HVM guest kernel?  Is
> it visible from a PV guest cpuid (e.g. OS or app)?
>
> Thanks,
> Dan
>
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> From: Wei Huang [mailto:wei.huang2@amd.com]
> Sent: Friday, May 27, 2011 3:59 PM
> To: Dan Magenheimer
> Cc: xen-devel@lists.xensource.com; Keir Fraser
> Subject: Re: [Xen-devel] [PATCH] SVM: enables TSC scaling ratio support
> for SVM
> 
> Hi Dan,
> 
> I tested it by migrating between systems with TSC and without TSC,
> mainly using a Linux guest VM.
> 
> I might have missed your point. But what is offset1 in your formula? My
> understanding is that hvm_get_guest_tsc() returns a TSC value guests
> are
> supposed to use. So when guest_tsc is set, we can re-calculate
> offset=hvm_get_guest_tsc() - host_tsc * ratio. This is how current Xen
> is implemented. Also, as long as host_tsc doesn''t go backwards
(i.e.
> TSC_RELIABLE), host_tsc * ratio+offset should be reliable. Are you
> concerned that cpu_khz might be (slightly) different on different
> cores?
offset1 is the last "virtualized" TSC value on the previous physical
machine.  offset2 is the value of TSC when the guest starts.
(Does "virtualized" TSC always start at 0 on an HVM guest?)
But my formula is just an attempt to restate the concern and
I don''t remember it exactly.

cpu_khz is missing 10 bits of precision (because it is khz, not hz)
and multiplying by cpu_khz propagates and magnifies that loss of
precision.

I *think* it may be OK as long as your "offset" is always greater
than or equal to the last "virtualized" TSC value read by the
guest... at least TSC will never be observed going backwards
in the guest.

But I still have concern about the loss of precision after
multiple migrations. (It may be possible to prove mathematically
that it is OK... I''m just not a mathematician.)
> This feature is only available under SVM mode. HVM guests won''t be
able
> to see it in CPUID. PV guests shouldn''t be able to change the
value of
> this MSR.
OK, thanks for the clarification.

Thanks,
Dan
> On 05/27/2011 03:41 PM, Dan Magenheimer wrote:
> >> From: Wei Huang [mailto:wei.huang2@amd.com]
> >> Sent: Friday, May 27, 2011 10:35 AM
> >> To: xen-devel@lists.xensource.com; Keir Fraser
> >> Subject: [Xen-devel] [PATCH] SVM: enables TSC scaling ratio
support
> for
> >> SVM
> >>
> >> Future AMD CPUs support TSC scaling. It allows guests to have a
> >> different TSC frequency from host system using this formula:
> guest_tsc
> >> > >> host_tsc * tsc_ratio + vmcb_offset. The tsc_ratio is
a 64bit MSR
> >> contains a fixed-point number in 8.32 format (8 bits for integer
> part
> >> and 32bits for fractional part). For instance 0x00000003_80000000
> means
> >> tsc_ratio=3.5.
> >>
> >> This patch enables TSC scaling ratio for SVM. With it, guest VMs
> don''t
> >> need take #VMEXIT to calculate a translated TSC value when it is
> >> running
> >> under TSC emulation mode. This can SUBSTANTIALLY reduce the rdtsc
> >> overhead.
> >>
> >> Signed-off-by: Wei Huang<wei.huang2@amd.com>
> > Has this patch been tested across save/restore and migration,
> > especially between machines with and without the feature and
> > especially across many migrations where each physical
> > machine has a different tsc_ratio?
> >
> > ISTR that this feature does not really do a generic adjustment
> > so may mis-scale time that has been accrued on one or more
> > previous physical machines.  In other words, I think the problem
> > is that it does
> >
> > (host_tsc * tsc_ratio) + offset
> >
> > and not
> >
> > ((host_tsc + offset1) * tsc_ratio) + offset2
> >
> > This can be fixed if you trust cpu_khz to be precise on all
> > machines, but I don''t think it is sufficiently precise to
> > guarantee that time never goes backwards in a guest (though
> > that may be fixable too).  If time DOES go backwards and
> > the guest detects it, it may switch to a much slower
> > timer mechanism which could be worse than trapping
> > rdtsc.
> >
> > All of this is from vague recollection... if it is all fully
> > tested across all cases (and sufficient testing proves that
> > time never goes backwards), consider this just noting a concern.
> >
> > Also, is this feature visible from an HVM guest kernel?  Is
> > it visible from a PV guest cpuid (e.g. OS or app)?
> >
> > Thanks,
> > Dan
> >
> >
> 
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel