Xen devel - Jun 2005 - HW Virtualization Abstraction Layer Work Underway

Tom Woller and I from AMD have been working with Ian, Keir and IBM on
scoping out a plan that will provide a common interface to the user for
both the Intel VT/VMX and the AMD Pacifica/SVM platforms.

A key concept of the  proposal is the creation of a HW Virtualization
Layer (called HVAL), which we describe below. We are still working out
some of the arcane details offline and are discussing it with Sunil from
Intel for a joint proposal.  

While we are offline sorting out some details, we wanted to
share/review/discuss with the community our thinking behind HVAL.

Goals of HVAL: 
 - Abstract the areas of the code that deal with low level platform
level specifics of hardware based virtualization
 - Providing a single domain builder interface creating a simple
abstraction layer within the hypervisor for both VMX and SVM
 - Create an interface that is suitable for both architectures
 - Minimize the number components that need to be modified... keep as
much of existing VMX code 

The modifications will consist of:
(1) support infrastructure modifications (libxc/python/config files)
(2) creating generic HVAL code base from the VMX existing base
(3) creating a simple function table for the generic/common Hypervisor
HVAL functions
(4) modifying the current VMX (and future SVM) to use the new HVAL
interface
(5) modifying current non-VMX files that access VMX specific functions
(6) modify such that all VM logic is compiled by default (obsoletes
CONFIG_VMX)

Some other points worth mentioning is the HVAL will be limited to the
x86 tree and will apply to both 32 and 64 bit.

Attached is a file the describes the implementation of HVAL and a
breakdown of the files that will be touched.  You''ll see there are some
parts that are left as ''MORE TO COME SOON'' as AMD and Intel
work on
these sections together.

Tom Woller is building the patch for this.

Elsie Wahlig
AMD


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Wahlig, Elsie wrote:
> Tom Woller and I from AMD have been working with Ian, Keir and IBM on
> scoping out a plan that will provide a common interface to the user for
> both the Intel VT/VMX and the AMD Pacifica/SVM platforms.
Nice to see some coordination underway here.. thanks for the initiative.

 >..will be limited to the x86 tree..

Why not have the POWER5 and BPA (nee STI-Cell) folks review it too.
I know the desire is to get something out ASAP for x86, but it would
be nice if Xen HVAL could be easily extended to new platforms.


Thanks,
Eric


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Wahlig, Elsie wrote:
> Some other points worth mentioning is the HVAL will be limited to the
> x86 tree and will apply to both 32 and 64 bit.
Are there ever going to be x86-32-only processors that support VT-x or
Pacifica? I thought that only EM64T and AMD64 processors were going
to have this support.

-- 
David Hopwood <david.nospam.hopwood@blueyonder.co.uk>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Wahlig, Elsie wrote:
> Tom Woller and I from AMD have been working with Ian, Keir and IBM on
> scoping out a plan that will provide a common interface to the user
> for both the Intel VT/VMX and the AMD Pacifica/SVM platforms.
> 
> A key concept of the  proposal is the creation of a HW Virtualization
> Layer (called HVAL), which we describe below. We are still working out
> some of the arcane details offline and are discussing it with Sunil
> from Intel for a joint proposal.
> 
We are evaluating this proposal and will work on enhancing the VT-x code
to provide the right level of abstraction to support other VT-x like
technologies.

[Personal opinion] Since the handling of the CPU architectural state (as
handled by vmx.c and vmx_vmcs.c today, for example) is specific to each
virtualization technology and can be optimized more if the code is aware
of the technolgy, for a common interface I think we should focus on the
virtual platform area (i.e. configration definition & domain builder,
device models, I/O request notifcation, I/O VMEXIT handler, instruction
decoder for MMIO, etc.), rather than dealing with broader or vague "HW
Virtualization." That architecture in Xen is independent of VT-x or VT-x
does not define such area, thus it can/should be open to other H/W
assist virtualization technologies. In fact it was designed to support
VT-i (H/W assist virtualization on Itanium) as well. 
> Elsie Wahlig
> AMD
Jun
---
Intel Open Source Technology Center

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 6 Jun 2005, at 09:15, Nakajima, Jun wrote:
> Since the handling of the CPU architectural state (as
> handled by vmx.c and vmx_vmcs.c today, for example) is specific to each
> virtualization technology and can be optimized more if the code is 
> aware
> of the technolgy, for a common interface I think we should focus on the
> virtual platform area (i.e. configration definition & domain builder,
> device models, I/O request notifcation, I/O VMEXIT handler, instruction
> decoder for MMIO, etc.), rather than dealing with broader or vague "HW
> Virtualization."
We need to consider the low-level interfaces too, because we do not 
want a separate set of hooks into the generic Xen code for each 
different vendor mechanism. We will of course want to adapt this layer 
to ensure it doesn''t hide any value-add extensions, but the principle 
of hiding as much non-generic detail as possible behind a common 
interface still remains.

Also, many opportunities for special hw assistance occur early during 
trap into Xen (e.g., why did we trap out of the guest context?). 
Regardless of any common interface, the vendor-specific hwassist code 
has full control at that point, and can decide what it handles itself 
and how it interacts with common Xen code.

I dislike the name HVAL though -- it''s not very informative. Something 
like hwassist, vmassist, hw_vm, or many others would all be preferable 
imo.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Pacifica is designed to work on AMD64 and x86-32 architectures.  The
AMD64 processors can run the x86-32 bit OS''s natively.  Today, this is
a
feature we''re including in our AMD64 processors. 

Elsie    

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of David
Hopwood
Sent: Sunday, June 05, 2005 11:56 AM
To: xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] HW Virtualization Abstraction Layer Work
Underway

Wahlig, Elsie wrote:
> Some other points worth mentioning is the HVAL will be limited to the
> x86 tree and will apply to both 32 and 64 bit.
Are there ever going to be x86-32-only processors that support VT-x or
Pacifica? I thought that only EM64T and AMD64 processors were going to
have this support.

--
David Hopwood <david.nospam.hopwood@blueyonder.co.uk>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> On 6 Jun 2005, at 09:15, Nakajima, Jun wrote:
> 
>> Since the handling of the CPU architectural state (as
>> handled by vmx.c and vmx_vmcs.c today, for example) is specific to
>> each virtualization technology and can be optimized more if the code
>> is aware of the technolgy, for a common interface I think we should
>> focus on the virtual platform area (i.e. configration definition &
>> domain builder, device models, I/O request notifcation, I/O VMEXIT
>> handler, instruction decoder for MMIO, etc.), rather than dealing
>> with broader or vague "HW Virtualization."
> 
> We need to consider the low-level interfaces too, because we do not
> want a separate set of hooks into the generic Xen code for each
> different vendor mechanism. We will of course want to adapt this layer
> to ensure it doesn''t hide any value-add extensions, but the
principle
> of hiding as much non-generic detail as possible behind a common
> interface still remains.
I agree. Today the hooks (e.g. when creating, terminating, or switching
domains) are required to do different things for full virtualization
(rather than para-virutalization), and should be exposed as clean
well-defined interface.
> 
> Also, many opportunities for special hw assistance occur early during
> trap into Xen (e.g., why did we trap out of the guest context?).
> Regardless of any common interface, the vendor-specific hwassist code
> has full control at that point, and can decide what it handles itself
> and how it interacts with common Xen code.
That''s right. One thing we should do is to have a common I/O VMEXIT and
MMIO handler. The first-level trap handlers (like vmx_asm_vmexit_handler
or vmx_vmexit_handler) are very specific to each H/W assist
architecture, but we should be able to define common handlers for these
(by slightly modifying the VMX code). 
> 
> I dislike the name HVAL though -- it''s not very informative.
Something
> like hwassist, vmassist, hw_vm, or many others would all be preferable
> imo.
> 
>   -- Keir
I don''t like the name HVAL, either, because of the same reason. What we
are doing is to have _additional_ or dedicated interfaces exposed in Xen
to provide full virtualization in support of H/W assist virtualization. 

BTW, I don''t think the following is the right abstaraction because the
original arch_vmx_struct was intended to maintain the architectural
state, and it can be different on other H/W assist virtualization. In
fact, we need to add more things to support 64-bit guests. The struct
virtual_platform_def (and flags) should moved out of the architectual
state, and probably arch_svm_struct needs to defined sperately.  

struct arch_hval_struct {
        union {
            struct vmcs_struct *vmcs; //vmx
            struct vmcb_struct *vmcb; //svm
        }
        unsigned long  flags;  /* VMCS/VMCB flags */
        unsigned long  cpu_cr2;
        unsigned long  cpu_cr3;
        unsigned long  cpu_state;
        struct virtual_platform_def hval_platform;
}        

Jun
---
Intel Open Source Technology Center

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Jun 06, 2005 at 10:02:10AM +0100, Keir Fraser
wrote:
> 
> We need to consider the low-level interfaces too, because we do not 
> want a separate set of hooks into the generic Xen code for each 
> different vendor mechanism. We will of course want to adapt this layer 
> to ensure it doesn''t hide any value-add extensions, but the
principle
> of hiding as much non-generic detail as possible behind a common 
> interface still remains.
> 
> Also, many opportunities for special hw assistance occur early during 
> trap into Xen (e.g., why did we trap out of the guest context?). 
> Regardless of any common interface, the vendor-specific hwassist code 
> has full control at that point, and can decide what it handles itself 
> and how it interacts with common Xen code.
> 
> I dislike the name HVAL though -- it''s not very informative.
Something
> like hwassist, vmassist, hw_vm, or many others would all be preferable 
> imo.
Since vmassist is used by other Xen code, we should probably go with 
hwassist.

In the attached diagram, I''ve attempted to draw a picture of where we
are today (no eggs or rotten tomatoes please - I don''t draw boxes often
:)

I think the following points are clear to me:

- The interface between xen/x86 and the box labelled "VMX
infrastructure"
  needs to be generic (let''s call the interface hwassist and the box
  hwassist infrastructure). This interface already seems to be well defined
  and it might just be a renaming that''s required.

- The interface between the cpu and the low level code is going to be
  vendor specific (the box labelled vmexit/vmresume)

- The box labelled "vmx platform" represents the PC platform being
emulated
  (partly in xen, rest in ioemu). This is abstracted already by:

  struct virtual_platform_def in vmx_platform.h

  I think it makes sense to share this code.

- We need to define an interface between the vmexit box and the virtual 
  platform box. This interface should not be very low level (something like 
  __vmread(regX) sounds too low level to me). store_cpu_user_regs(), 
  check_guest_faults(), inject_exception() etc sound like a better abstraction.

- There may be more sharing opportunities which are not too disruptive
  elsewhere, but until we have a second implementation, we may not know 
  what they are. So we''d like to defer any interface changes until we
have
  a second implementation.

	-Arun


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Nakajima, Jun wrote: 
> 
> That''s right. One thing we should do is to have a common I/O 
> VMEXIT and MMIO handler. The first-level trap handlers (like 
> vmx_asm_vmexit_handler or vmx_vmexit_handler) are very 
> specific to each H/W assist architecture, but we should be 
> able to define common handlers for these (by slightly 
> modifying the VMX code). 
The MMIO handler code (handle_mmio entry point in vmx_platform.c) has
been refactored as generic (now hval_platform.c).  The I/O VMEXIT code
handler in vmx_io.c has also been refactored and is now hval_io.c (new
hval_io_assist/hval_intr_assist/hval_do_resume entry points).  The
current vmx exit handler can''t be made more generic for SVM.   This
will
be in the patch that is coming.  
> 
> > 
> > I dislike the name HVAL though -- it''s not very 
> informative. Something 
> > like hwassist, vmassist, hw_vm, or many others would all be 
> preferable 
> > imo.
> > 
> >   -- Keir
> 
> I don''t like the name HVAL, either, because of the same 
> reason. What we are doing is to have _additional_ or 
> dedicated interfaces exposed in Xen to provide full 
> virtualization in support of H/W assist virtualization. 
> 
HWASSIST was our original name for it, but was deemed to be too long.  
The names with the vm letters hit many false positives when grepping
code, 
so we avoided all names with ''vm''.   Since this was defining
an
interface, 
HVAL was one we went with.  Now if the name matters, we can make it
HWASSIST.  
> BTW, I don''t think the following is the right abstaraction 
> because the original arch_vmx_struct was intended to maintain 
> the architectural state, and it can be different on other H/W 
> assist virtualization. In fact, we need to add more things to 
> support 64-bit guests. The struct virtual_platform_def (and 
> flags) should moved out of the architectual state, and 
> probably arch_svm_struct needs to defined sperately.  
> 
> struct arch_hval_struct {
>         union {
>             struct vmcs_struct *vmcs; //vmx
>             struct vmcb_struct *vmcb; //svm
>         }
>         unsigned long  flags;  /* VMCS/VMCB flags */
>         unsigned long  cpu_cr2;
>         unsigned long  cpu_cr3;
>         unsigned long  cpu_state;
>         struct virtual_platform_def hval_platform;
> }        
I don''t see why we would need both a arch_vmx_struct and an
arch_svm_struct at one time. 
That physical platform wont ever be built.  The goal is to abstract out
the hardware 
and need only maintain one of the control blocks during it''s runtime.
Does your arch
Need a different vmcs for 64-bits?  

Elsie Wahlig


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Wahlig, Elsie wrote:
>>> I dislike the name HVAL though -- it''s not very
informative.
>>> Something like hwassist, vmassist, hw_vm, or many others would all
>>> be preferable imo. 
>>> 
>>>   -- Keir
>> 
>> I don''t like the name HVAL, either, because of the same
>> reason. What we are doing is to have _additional_ or
>> dedicated interfaces exposed in Xen to provide full
>> virtualization in support of H/W assist virtualization.
>> 
> 
> HWASSIST was our original name for it, but was deemed to be too long.
> The names with the vm letters hit many false positives when grepping
> code,
> so we avoided all names with ''vm''.   Since this was
defining an
> interface,
> HVAL was one we went with.  Now if the name matters, we can make it
> HWASSIST.
I agree that it''s too long, especially when used in the code. I prefer
a
3 or 4-letter acronyum. I don''t think that real programmers depend on
"grep" when looking at the code. I think using ''vm''
is okay in the sence
that it''s the common one used by ''vmx'' and
''svm'' ;-) hw_vm - we want to
avoid ''_'' because we frequently use ''_'' in
the code. How about hw_vm ->
xvm, i.e. (Xen) Extended Virtual Machine Layer? The idea is that we
extend Xen to support full virtualization. It''s a bit confusing with
vmx, but I think we are okay with that. I tentatively use
''xvm'' below to
see how it looks like.
> 
>> BTW, I don''t think the following is the right abstaraction
>> because the original arch_vmx_struct was intended to maintain
>> the architectural state, and it can be different on other H/W
>> assist virtualization. In fact, we need to add more things to
>> support 64-bit guests. The struct virtual_platform_def (and
>> flags) should moved out of the architectual state, and
>> probably arch_svm_struct needs to defined sperately.
>> 
>> struct arch_hval_struct {
>>         union {
>>             struct vmcs_struct *vmcs; //vmx
>>             struct vmcb_struct *vmcb; //svm
>>         }
>>         unsigned long  flags;  /* VMCS/VMCB flags */
>>         unsigned long  cpu_cr2;
>>         unsigned long  cpu_cr3;
>>         unsigned long  cpu_state;
>>         struct virtual_platform_def hval_platform;
>> }
> 
> I don''t see why we would need both a arch_vmx_struct and an
> arch_svm_struct at one time.
They are exclusive at runtime, and I''m suggesting like

struct arch_vcpu
{
     ...
     struct arch_vmx_struct arch_vmx;  =>(replace with below)
=> 

      union { /* architecture state */
            struct_vmx_struct arch_vmx;
            struct_svm_struct arch_svm;
      }
      struct xvm_struct *xvm; /* xen extened virtual machine */


struct xvm_struct {
      struct virutal_platform_def      xvm_platform;
      struct xvm_ops                   *ops;

};

This way we can cleanly sperate the (vendor-specific) CPU architecture
state, virtual platform state, and the xvm operatons.
 
struct xvm_ops {
      void (*alloc)(struct exec_domain *d);
      void (*free)(struct exec_domain *d);
      void (*load_cpu_user_regs)(struct cpu_user_regs *regs);
      void (*store_cpu_user_regs)(struct cpu_user_regs *regs);
      void (*check_guest_faults)(struct cpu_user_regs *regs, long cs,
long eip, long type);
      void (*inject_exception)(struct cpu_user_regs *regs, long
exception);
      void (*update_eip)(struct cpu_user_regs *regs, long new_eip);     
};
> That physical platform wont ever be built.  The goal is to abstract
> out the hardware
> and need only maintain one of the control blocks during it''s
runtime.
> Does your arch
> Need a different vmcs for 64-bits?
No. For example, handling MSRs, such as MSR_LSTAR, MSR_STAR, MSR_CSTAR,
MSR_SYSCALL_MASK, MSR_EFER can be different. Some of them are
saved/restored at VMEXIT time, but some are not, or even configurable.
Handling of the control registers such as CR0, CR4, CR8 is another
example. It can be different depending on the virtualization technology.
Today we don''t have sufficient data which CPU state shoud be exposed as
the generic state. If we need to expose some CPU state in the CPU (like
long mode), we should add one to the xvm_ops, not to the data structure.
> 
> Elsie Wahlig
Jun
---
Intel Open Source Technology Center

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Nakajima, Jun wrote:
> 
> I agree that it''s too long, especially when used in the code. I
> prefer a 3 or 4-letter acronyum. I don''t think that real
programmers
> depend on "grep" when looking at the code. I think using
''vm'' is okay
> in the sence that it''s the common one used by
''vmx'' and ''svm'' ;-)
> hw_vm - we want to avoid ''_'' because we frequently use
''_'' in the
> code. How about hw_vm -> xvm, i.e. (Xen) Extended Virtual Machine
> Layer? The idea is that we extend Xen to support full virtualization.
> It''s a bit confusing with vmx, but I think we are okay with that.
I
> tentatively use ''xvm'' below to see how it looks like.
> 
Good news is that ''xvm'' appears in the sources precisely zero
times.

BTW, I recommend cscope when reviewing the code. In the xen tree, one
can easily enable fast symbol lookup, by:
$ make cscope

Jun
---
Intel Open Source Technology Center

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Nakajima, Jun wrote:
# Good news is that ''xvm'' appears in the sources precisely
zero times.

Let me put in my $0.02, I don''t like xvm, or hwassist, hwasst. xvm
looks
like I mistyped vmx and hwassist/hwasst are not descriptive enough and
too long. 

What about hve or hvx for Hardware Virtualization Extensions? Elsie seems
to prefer something with hardware in it to convey that this is silicon
thing.

	Leendert


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

leendert@watson.ibm.com wrote:
> Nakajima, Jun wrote:
> # Good news is that ''xvm'' appears in the sources
precisely zero times.
> 
> Let me put in my $0.02, I don''t like xvm, or hwassist, hwasst. xvm
> looks like I mistyped vmx and hwassist/hwasst are not descriptive
> enough and too long.
Leendert, you are right. After actually using more ''xvm'' in
the code (to
show examples), it started flickering between ''vmx'' and
''xvm''; it''s not
good for the eyes.
> 
> What about hve or hvx for Hardware Virtualization Extensions? Elsie
> seems to prefer something with hardware in it to convey that this is
> silicon thing.
> 
> 	Leendert
I like Hardware Virtualization Extensions. It''s simple and captures its
own purpose well. 

''hve'' seems better after Web search, "Did you mean:
have". No hits in
the xen source code except (common/sched_sedf.c):
*     -addition: experiments hve shown that this may have a HUGE impact
on

I think that''s "Did you mean: have" :-)

We''ll post our modification proposal to Elsis''s doc today.

Jun
---
Intel Open Source Technology Center

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel