Biemueller, Sebastian
2007-May-15 10:00 UTC
[Xen-devel] [PATCH][HVM][SVM] Reintroduce ASIDs.
This patch reintroduces ASID management for AMD-V. This work was presented on
the last XenSummit [1].
Short overview:
ASIDs partition the physical TLB for SVM. In the current implementation ASIDs
are used to reduce the number of TLB flushes. Each time the guest''s
virtual address space changes (e.g. due to an INVLPG, MOV-TO-{CR3, CR4}
operation), instead of flushing the TLB, a new ASID is assigned. This reduces
the number of TLB flushes to at most 1/#ASIDs (currently 1/64). The biggest
advantage is that hot parts of the hypervisor''s code and data remain in
the TLB.
Sketch of the Implementation:
ASIDs are a CPU-local resource. As preemption of ASIDs is not possible, ASIDs
are assigned in a round-robin scheme. To minimize the overhead of ASID
invalidation, at the time of a TLB flush, ASIDs are tagged with a 64-bit
generation. Generation overflows are designed not to happen and thus not
optimized. (They appear at most every 2^80 cycles). Here ASIDs are simply
disabled to retain correctness [2].
b/xen/arch/x86/hvm/svm/asid.c | 181 ++++++++++++++++++++++++++++++++
b/xen/include/asm-x86/hvm/svm/asid.h | 162 +++++++++++++++++++++++++++++++
xen/arch/x86/cpu/amd.c | 4
xen/arch/x86/hvm/svm/Makefile | 1
xen/arch/x86/hvm/svm/svm.c | 41 ++++++-
xen/arch/x86/hvm/svm/vmcb.c | 8 -
xen/arch/x86/hvm/svm/x86_32/exits.S | 7 +
xen/arch/x86/hvm/svm/x86_64/exits.S | 7 +
xen/include/asm-x86/hvm/svm/vmcb.h | 5
9 files changed, 401 insertions(+), 15 deletions(-
Regards,
Sebastian
References:
[1] Presentation given at the XenSummit Spring, 2007
http://www.xensource.com/files/xensummit_4/2007XenSummit-AMD-ASIDS-Biemueller.pdf
[2] The 2^80 cycles result from: 1. And VM-Entry/-Exit cycle takes at least 1800
cycles (approximated by 2^10) 2. We have 64 ASIDs (2^6) 3. 2^64 generations.
--
AMD Inc.
Operating System Research Center
Legal Information:
AMD Saxony Limited Liability Company & Co. KG
Sitz (Geschäftsanschrift):
Wilschdorfer Landstr. 101, 01109 Dresden, Deutschland
Registergericht Dresden: HRA 4896
vertretungsberechtigter Komplementär:
AMD Saxony LLC (Sitz Wilmington, Delaware, USA)
Geschäftsführer der AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Hi Sebastian, Nice work, but please resync to staging tip of xen-unstable. http://xenbits.xensource.com/staging/xen-unstable.hg. We haven''t pushed to the main tree for a while now due to various issues, so there''ll be some merging to do. Thanks, Keir On 15/5/07 11:00, "Biemueller, Sebastian" <Sebastian.Biemueller@amd.com> wrote:> This patch reintroduces ASID management for AMD-V. This work was presented on > the last XenSummit [1]. > > Short overview: > > ASIDs partition the physical TLB for SVM. In the current implementation ASIDs > are used to reduce the number of TLB flushes. Each time the guest''s virtual > address space changes (e.g. due to an INVLPG, MOV-TO-{CR3, CR4} operation), > instead of flushing the TLB, a new ASID is assigned. This reduces the number > of TLB flushes to at most 1/#ASIDs (currently 1/64). The biggest advantage is > that hot parts of the hypervisor''s code and data remain in the TLB. > > > Sketch of the Implementation: > > ASIDs are a CPU-local resource. As preemption of ASIDs is not possible, ASIDs > are assigned in a round-robin scheme. To minimize the overhead of ASID > invalidation, at the time of a TLB flush, ASIDs are tagged with a 64-bit > generation. Generation overflows are designed not to happen and thus not > optimized. (They appear at most every 2^80 cycles). Here ASIDs are simply > disabled to retain correctness [2]. > > > b/xen/arch/x86/hvm/svm/asid.c | 181 ++++++++++++++++++++++++++++++++ > b/xen/include/asm-x86/hvm/svm/asid.h | 162 +++++++++++++++++++++++++++++++ > xen/arch/x86/cpu/amd.c | 4 > xen/arch/x86/hvm/svm/Makefile | 1 > xen/arch/x86/hvm/svm/svm.c | 41 ++++++- > xen/arch/x86/hvm/svm/vmcb.c | 8 - > xen/arch/x86/hvm/svm/x86_32/exits.S | 7 + > xen/arch/x86/hvm/svm/x86_64/exits.S | 7 + > xen/include/asm-x86/hvm/svm/vmcb.h | 5 > 9 files changed, 401 insertions(+), 15 deletions(- > > > > Regards, > > Sebastian > > > > References: > > [1] Presentation given at the XenSummit Spring, 2007 > http://www.xensource.com/files/xensummit_4/2007XenSummit-AMD-ASIDS-Biemueller. > pdf > > [2] The 2^80 cycles result from: 1. And VM-Entry/-Exit cycle takes at least > 1800 cycles (approximated by 2^10) 2. We have 64 ASIDs (2^6) 3. 2^64 > generations. > > > -- > AMD Inc. > Operating System Research Center > > Legal Information: > AMD Saxony Limited Liability Company & Co. KG > Sitz (Geschäftsanschrift): > Wilschdorfer Landstr. 101, 01109 Dresden, Deutschland > Registergericht Dresden: HRA 4896 > vertretungsberechtigter Komplementär: > AMD Saxony LLC (Sitz Wilmington, Delaware, USA) > Geschäftsführer der AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> This patch reintroduces ASID management for AMD-V. This work was > presented on the last XenSummit [1].Thanks for submitting this. It would be good if you could post a benchmark or two (e.g. a kernel compile and a few lmbench numbers) as I''d like to know what gain we''re getting -- especially as the previous version ended up doing some long list traversals that hurt performance on systems where use of ASID was prevented by errata. Thanks, Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 15/5/07 11:00, "Biemueller, Sebastian" <Sebastian.Biemueller@amd.com> wrote:> This patch reintroduces ASID management for AMD-V.A few more comments: 1. Rather than adding an extra hook at every use of paging_update_cr3(), paging_update_paging_modes(), paging_invlpg(), move those original calls into your hook functions. Then the original invocation is simply replaced with a call to your function, which can hide the extra work more neatly. 2. Adding an extra function call on every vmentry is unfortunate. Why not guarantee that any running SVM VCPU has a valid ASID at all times, even while running inside Xen? This would mean that you would greedily allocate an ASID as soon as you context-switch to a SVM VCPU, and also when the old ASID becomes invalid. Then you could do your tests and VMCB-twiddling on those less common paths, and avoid extra cost on the much more common vmentry path. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel